Package 'stddiff.spark'

Compute Standardized Differences for Binary Variables (Spark)

Description

Calculates standardized differences for binary variables using a Spark DataFrame. Equivalent to stddiff::stddiff.binary but operates on Spark data.

Usage

stddiff.binary(data, gcol, vcol, verbose = FALSE)

Arguments

data	A Spark DataFrame (tbl_spark) containing the variables.
gcol	Integer; column index of the binary grouping variable (e.g., treatment vs control).
vcol	Integer vector; column indices of the binary variables to analyze.
verbose	Logical; if TRUE, prints progress messages. Default is FALSE.

Details

Variables are encoded using lexicographic ordering since Spark does not have factor types. The first level alphabetically becomes 0, the second becomes 1.

The standardized difference is computed as:

$d = \frac{|p_t - p_c|}{\sqrt{(p_t(1-p_t) + p_c(1-p_c))/2}}$

Value

A numeric matrix with one row per variable and columns:

• p.c: Proportion in control group (first level alphabetically)

• p.t: Proportion in treatment group (second level alphabetically)

• missing.c: Number of missing values in control group

• missing.t: Number of missing values in treatment group

• stddiff: Standardized difference

• stddiff.l: Lower bound of 95% confidence interval

• stddiff.u: Upper bound of 95% confidence interval

See Also

stddiff.category, stddiff.numeric

Examples

sc <- sparklyr::spark_connect(master = "local")

spark_df <- sparklyr::copy_to(sc, mtcars)

result <- stddiff.binary(
  data = spark_df,
  gcol = 9,   # column index of grouping variable
  vcol = c(8) # columns of binary variables
)

sparklyr::spark_disconnect(sc)

---

Compute Standardized Differences for Categorical Variables (Spark)

Description

Calculates standardized differences for categorical variables using a Spark DataFrame. Equivalent to stddiff::stddiff.category but operates on Spark data.

Usage

stddiff.category(data, gcol, vcol, verbose = FALSE)

Arguments

data	A Spark DataFrame (tbl_spark) containing the variables.
gcol	Integer; column index of the binary grouping variable.
vcol	Integer vector; column indices of the categorical variables to analyze.
verbose	Logical; if TRUE, prints progress messages. Default is FALSE.

Details

For categorical variables with K levels, the standardized difference is computed using a multivariate approach that accounts for all K-1 levels simultaneously (excluding the reference level). Category levels are sorted lexicographically; the first level alphabetically serves as the reference.

Value

A numeric matrix with one row per category level and columns:

• p.c: Proportion in control group

• p.t: Proportion in treatment group

• missing.c: Number of missing values in control group (first row only)

• missing.t: Number of missing values in treatment group (first row only)

• stddiff: Standardized difference (first row only)

• stddiff.l: Lower CI bound (first row only)

• stddiff.u: Upper CI bound (first row only)

Row names are formatted as "variable_name level_name".

See Also

stddiff.binary, stddiff.numeric

Examples

sc <- sparklyr::spark_connect(master = "local")

spark_df <- sparklyr::copy_to(sc, as.data.frame(Titanic))

result <- stddiff.category(
  data = spark_df,
  gcol = 4,   # column index of grouping variable
  vcol = c(1) # columns of categorical variables
)

sparklyr::spark_disconnect(sc)

---

Compute Standardized Differences for Numeric Variables (Spark)

Description

Calculates standardized differences for continuous numeric variables using a Spark DataFrame. Equivalent to stddiff::stddiff.numeric but operates on Spark data.

Usage

stddiff.numeric(data, gcol, vcol, verbose = FALSE)

Arguments

data	A Spark DataFrame (tbl_spark) containing the variables.
gcol	Integer; column index of the binary grouping variable.
vcol	Integer vector; column indices of the numeric variables to analyze.
verbose	Logical; if TRUE, prints progress messages. Default is FALSE.

Details

The standardized difference for continuous variables is computed as:

$d = \frac{|\bar{x}_t - \bar{x}_c|}{\sqrt{(s_t^2 + s_c^2)/2}}$

where $\bar{x}$ represents means and $s^2$ represents variances.

This is equivalent to Cohen's d with pooled standard deviation.

Value

A numeric matrix with one row per variable and columns:

• mean.c: Mean in control group

• sd.c: Standard deviation in control group

• mean.t: Mean in treatment group

• sd.t: Standard deviation in treatment group

• missing.c: Number of missing values in control group

• missing.t: Number of missing values in treatment group

• stddiff: Standardized difference

• stddiff.l: Lower bound of 95% confidence interval

• stddiff.u: Upper bound of 95% confidence interval

See Also

stddiff.binary, stddiff.category

Examples

sc <- sparklyr::spark_connect(master = "local")

spark_df <- sparklyr::copy_to(sc, mtcars)

result <- stddiff.numeric(
  data = spark_df,
  gcol = 8,          # column index of grouping variable
  vcol = c(1, 2, 5)  # columns of numeric variables
)

sparklyr::spark_disconnect(sc)

---