Comparison of parametric and Random Forest MICE in imputation of missing data in survival analysis

Anoop D. Shah    Jonathan W. Bartlett    James Carpenter Affiliation: Owen Nicholas and Harry Hemingway

1 Introduction

This is a simulation study comparing various methods for imputation of missing covariate data in a survival analysis in which there are interactions between the predictor variables. We compare our new Random Forest method for MICE (Multivariate Imputation by Chained Equations) with other imputation methods and full data analysis. In our Random Forest method (RFcont), the conditional mean missing values are predicted using Random Forest and imputed values are drawn from Normal distributions centred on the predicted means [1].

We also perform a comparison with the methods recently published by Doove et al. [2]: mice.impute.cart (classification and regression trees in MICE) and mice.impute.rf (MICE using Random Forests).

2 Methods

We used the R packages CALIBERrfimpute, survival, xtable, missForest and randomForest. We created simulated survival datasets with two fully observed predictor variables (x1, x2) and a partially observed predictor (x3), which depends on x1, x2 and their interaction. They were generated as follows:

x1

Standard normal distribution

x2

Standard normal distribution, independent of x1

x3

Derived from x1 and x2: x3=0.5(x1+x2−x1.x2)+e where e is normally distributed with mean 0 and variance 1.

The equation for the log hazard of patient i was given by:

hi=β1⁢x1⁢i+β2⁢x2⁢i+β3⁢x3⁢i (1)

where all the β coefficients were set to 0.5.

We used an exponential distribution to generate a survival time for each patient. We also generated an observation time for each patient, as a random draw from a uniform distribution bounded by zero and the 50th percentile of survival time. If the observation time was less than the survival time, the patient was considered as censored (event indicator 0, and the patient’s follow-up ends on their censoring date), otherwise the event indicator was 1, with follow-up ending on the date of event.

Linear regression model relating x3 to x1 and x2:

> # Chunk 4>> summary(lm(x3 ~ x1*x2, data = mydata))

Call:lm(formula = x3 ~ x1 * x2, data = mydata)Residuals: Min 1Q Median 3Q Max-3.8146 -0.6762 0.0019 0.6729 4.1377Coefficients: Estimate Std. Error t value Pr(>|t|)(Intercept) -0.002775 0.007022 -0.395 0.693x1 0.497160 0.007079 70.234 <2e-16 ***x2 0.502151 0.006968 72.069 <2e-16 ***x1:x2 -0.498811 0.006914 -72.140 <2e-16 ***---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 0.9929 on 19996 degrees of freedomMultiple R-squared: 0.4318,Adjusted R-squared: 0.4317F-statistic: 5066 on 3 and 19996 DF, p-value: < 2.2e-16

All true log hazard ratios were assumed to be 0.5, with hazard ratios = 1.65. We checked that the hazard ratios in the simulated data were as expected for a large sample:

> # Chunk 6>> # Cox proportional hazards analysis> myformula <- as.formula(Surv(time, event) ~ x1 + x2 + x3)> # Analysis with 10,000 simulated patients (or more> # if the variable REFERENCE_SAMPLESIZE exists)> if (!exists('REFERENCE_SAMPLESIZE')){+ REFERENCE_SAMPLESIZE <- 10000+ }> # Use parallel processing, if available, to create> # datasets more quickly.> if ('parallel' %in% loadedNamespaces() &&+ !is.null(getOption('mc.cores')) &&+ .Platform$OS.type == 'unix'){+ REFERENCE_SAMPLESIZE <- REFERENCE_SAMPLESIZE %/%+ getOption('mc.cores')+ simdata <- parallel::mclapply(1:getOption('mc.cores'),+ function(x) makeSurv(REFERENCE_SAMPLESIZE))+ simdata <- do.call('rbind', simdata)+ } else {+ simdata <- makeSurv(REFERENCE_SAMPLESIZE)+ }> summary(coxph(myformula, data = simdata))

Call:coxph(formula = myformula, data = simdata) n= 10000, number of events= 3066 coef exp(coef) se(coef) z Pr(>|z|)x1 0.49836 1.64602 0.01937 25.73 <2e-16 ***x2 0.45959 1.58343 0.01924 23.89 <2e-16 ***x3 0.53740 1.71156 0.01734 30.99 <2e-16 ***---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 exp(coef) exp(-coef) lower .95 upper .95x1 1.646 0.6075 1.585 1.710x2 1.583 0.6315 1.525 1.644x3 1.712 0.5843 1.654 1.771Concordance= 0.766 (se = 0.004 )Likelihood ratio test= 2897 on 3 df, p=<2e-16Wald test = 2420 on 3 df, p=<2e-16Score (logrank) test = 2405 on 3 df, p=<2e-16

We created datasets containing 200 simulated patients. For each dataset, we first analysed the complete dataset with no values missing, then artificially created missingness in variable x3, imputed the missing values using various methods, and analysed the imputed datasets. We combined parameter estimates from multiply imputed datasets using Rubin’s rules.

2.1 Missingness mechanism

Missingness was imposed in x3 dependent on x1, x2, the event indicator and the marginal Nelson-Aalen cumulative hazard, using a logistic regression model. The linear predictors were offset by an amount chosen to make the overall proportion of each variable missing approximately 0.2, i.e.:

P⁢(miss)i=exp⁡(l⁢pi+offset)1+exp⁡(l⁢pi+offset) (2)
l⁢pi=0.1⁢x1⁢i+0.1⁢x2⁢i+0.1×cumhazi+0.1×eventi (3)

where ‘event’ is the event indicator and ‘cumhaz’ is the marginal Nelson-Aalen cumulative hazard.

We analysed the datasets with missing data using different methods of multiple imputation. We calculated the marginal Nelson-Aalen cumulative hazard and included it in all imputation models, along with the event indicator and follow-up time.

> # Chunk 7>> # Setting analysis parameters: To analyse more than 3 samples,> # set N to the desired number before running this program> if (!exists('N')){+ N <- 3+ }> # Number of imputations (set to at least 10 when> # running an actual simulation)> if (!exists('NIMPS')){+ NIMPS <- 4+ }> # Use parallel processing if the 'parallel' package is loaded> if ('parallel' %in% loadedNamespaces() &&+ .Platform$OS.type == 'unix'){+ cat('Using parallel processing\n')+ results <- parallel::mclapply(1:N, doanalysis)+ } else {+ results <- lapply(1:N, doanalysis)+ }

Using parallel processing

We used the following methods of multiple imputation. The number of imputations was 4. In each case, the imputation model for x3 contained x1, x2, the event indicator and the marginal Nelson-Aalen cumulative hazard:

missForest

– from the missForest package, which completes a dataset in an iterative way using Random Forest prediction. It was run with maximum 10 iterations (default) and 100 trees per forest (default).

CART MICE

– Clasification and regression tree MICE method from the mice package (mice.impute.cart).

RF MICE (Doove)

– Random Forest MICE method from Doove et al. [2], which is available as function mice.impute.rf in the mice package, with 10 or 100 trees.

RFcont MICE

– Random Forest MICE method from the CALIBERrfimpute package with 5, 10, 20 or 100 trees.

Parametric MICE

– normal-based linear regression with default settings, in which the imputation model for x3 is of the form:

x3=β0+β1.x1+β2.x2+β3.event+β4.cumhaz+e

where e is the residual variance.

We analysed 3 samples. We calculated the following for each method and each parameter:

  • •

    Bias of log hazard ratio

  • •

    Standard error of bias (Monte Carlo error)

  • •

    Mean square error

  • •

    Standard deviation of estimated log hazard ratio

  • •

    Mean length of 95% confidence intervals

  • •

    Coverage of 95% confidence intervals (proportion containing the true log hazard ratio)

3 Results

All the true log hazard ratios were set at 0.5.

3.1 Fully observed variables

Log hazard ratio for the continuous fully observed variable x1:

Standard Mean SD of Mean 95% 95% CI
Bias error of bias square error estimate CI length coverage
Full data 0.0698 0.141 0.0446 0.244 0.549 0.667
missForest 0.0423 0.159 0.0524 0.275 0.551 0.667
CART MICE 0.0547 0.156 0.0515 0.27 0.554 0.667
RF Doove MICE 10 0.0503 0.155 0.0509 0.269 0.553 0.667
RF Doove MICE 100 0.06 0.149 0.0478 0.257 0.563 0.667
RFcont MICE 5 0.0598 0.153 0.0506 0.266 0.561 0.667
RFcont MICE 10 0.0509 0.156 0.0513 0.27 0.56 0.667
RFcont MICE 20 0.0574 0.162 0.0557 0.28 0.564 0.667
RFcont MICE 100 0.0623 0.152 0.05 0.263 0.553 0.667
Parametric MICE 0.0402 0.15 0.0469 0.261 0.563 0.667

Log hazard ratio for the continuous fully observed variable x2:

Standard Mean SD of Mean 95% 95% CI
Bias error of bias square error estimate CI length coverage
Full data 0.0212 0.0434 0.00422 0.0752 0.54 1
missForest -0.00755 0.0465 0.00438 0.0805 0.551 1
CART MICE -0.00467 0.0554 0.00615 0.0959 0.557 1
RF Doove MICE 10 0.00551 0.0488 0.00479 0.0845 0.558 1
RF Doove MICE 100 0.0121 0.0424 0.00373 0.0734 0.565 1
RFcont MICE 5 -0.00327 0.0414 0.00344 0.0718 0.556 1
RFcont MICE 10 -0.00227 0.0501 0.00502 0.0867 0.565 1
RFcont MICE 20 0.00155 0.0369 0.00273 0.064 0.559 1
RFcont MICE 100 0.00142 0.0437 0.00383 0.0758 0.556 1
Parametric MICE -0.00417 0.0484 0.0047 0.0838 0.564 1

3.2 Partially observed variable

Log hazard ratio for the continuous partially observed variable x3:

Standard Mean SD of Mean 95% 95% CI
Bias error of bias square error estimate CI length coverage
Full data -0.0252 0.0943 0.0184 0.163 0.478 1
missForest -0.00595 0.112 0.0252 0.194 0.528 1
CART MICE -0.0848 0.124 0.0379 0.215 0.535 0.667
RF Doove MICE 10 -0.0971 0.0866 0.0244 0.15 0.565 1
RF Doove MICE 100 -0.106 0.0745 0.0223 0.129 0.512 1
RFcont MICE 5 -0.119 0.101 0.0346 0.175 0.527 0.667
RFcont MICE 10 -0.0838 0.0987 0.0265 0.171 0.536 1
RFcont MICE 20 -0.033 0.11 0.0252 0.19 0.516 1
RFcont MICE 100 -0.0515 0.0872 0.0179 0.151 0.536 1
Parametric MICE -0.117 0.109 0.0376 0.189 0.576 1

The following graph shows the bias for RFcont MICE methods by number of trees (bias estimated from 3 simulations; the lines denote 95% confidence intervals):

3.3 Pairwise comparisons between methods

3.3.1 Comparison of bias

Difference between absolute bias (negative means that the first method is less biased). P values from paired sample t tests. Significance level: * P <0.05, ** P <0.01, *** P <0.001.

RFcont MICE 10 vs RFcont MICE 100 vs RFcont MICE 10 vs
Coefficient parametric MICE parametric MICE RFcont MICE 100
x1 0.0108 0.0222 -0.0114
x2 -0.0019 -0.00275 0.000853
x3 -0.0334 -0.0658 0.0323
RF Doove MICE 10 vs RF Doove MICE 10 vs RF Doove MICE 10 vs
Coefficient RFcont MICE 10 CART MICE RF Doove MICE 100
x1 -0.000645 -0.00439 -0.00972
x2 0.00324 0.00084 -0.00656
x3 0.0132 0.0123 -0.00874

3.3.2 Comparison of precision

Ratio of variance of estimates (less than 1 means that the first method is more precise). P values from F test. Significance level: * P <0.05, ** P <0.01, *** P <0.001.

RFcont MICE 10 vs RFcont MICE 100 vs RFcont MICE 10 vs
Coefficient parametric MICE parametric MICE RFcont MICE 100
x1 1.07 1.02 1.06
x2 1.07 0.818 1.31
x3 0.816 0.637 1.28
RF Doove MICE 10 vs RF Doove MICE 10 vs RF Doove MICE 10 vs
Coefficient RFcont MICE 10 CART MICE RF Doove MICE 100
x1 0.993 0.997 1.09
x2 0.949 0.776 1.33
x3 0.77 0.489 1.35

3.3.3 Comparison of confidence interval length

Ratio of mean length of 95% confidence intervals (less than 1 means that the first method produces smaller confidence intervals). P values from paired sample t test. Significance level: * P <0.05, ** P <0.01, *** P <0.001.

RFcont MICE 10 vs RFcont MICE 100 vs RFcont MICE 10 vs
Coefficient parametric MICE parametric MICE RFcont MICE 100
x1 0.9951 0.9821 1.013
x2 1.003 0.986 * 1.017
x3 0.9301 0.9306 0.9995
RF Doove MICE 10 vs RF Doove MICE 10 vs RF Doove MICE 10 vs
Coefficient RFcont MICE 10 CART MICE RF Doove MICE 100
x1 0.9864 * 0.9983 0.982
x2 0.9868 1.002 0.9876
x3 1.055 1.056 * 1.103

3.3.4 Comparison of confidence interval coverage

Difference between percentage coverage of 95% confidence intervals (positive means that the first method has greater coverage). P values for pairwise comparisons by McNemar’s test. Significance level: * P <0.05, ** P <0.01, *** P <0.001.

RFcont MICE 10 vs RFcont MICE 100 vs RFcont MICE 10 vs
Coefficient parametric MICE parametric MICE RFcont MICE 100
x1 0.0 0.0 0.0
x2 0.0 0.0 0.0
x3 0.0 0.0 0.0
RF Doove MICE 10 vs RF Doove MICE 10 vs RF Doove MICE 10 vs
Coefficient RFcont MICE 10 CART MICE RF Doove MICE 100
x1 0.0 0.0 0.0
x2 0.0 0.0 0.0
x3 0.0 33.3 0.0

4 Discussion

In this simulation, parametric MICE using the default settings yielded a biased estimate for the coefficient for the partially observed variable x3. This is because the interaction between x1 and x2 was not included in the imputation models. The estimate using the CART or Random Forest MICE methods were less biased, more precise and had shorter confidence intervals with greater coverage. Omissions of interactions between predictors can potentially result in bias using parametric MICE even if, as in this case, the interaction is not present in the substantive model.

4.1 CART versus Random Forest MICE

CART MICE produced estimates for the x3 coefficient that were less precise than the Random Forest MICE methods, and coverage of 95% confidence intervals was only 93%.

4.2 Comparison of Random Forest MICE methods

Coefficients estimated after imputation using CART or Random Forest MICE methods were slightly biased. The bias was statistically significant but small in magnitude. Using RFcont MICE, the x3 coefficient was biased towards the null with 5 or 10 trees and biased away from the null with 20 or more trees; bias was minimised using 10 or 20 trees.

4.3 missForest

Parameters estimated after imputation using missForest were biased and the coverage of 95% confidence intervals was less than 95%. Failure to draw from the correct conditional distribution leads to bias and underestimation of the uncertainty when statistical models are fitted to imputed data.

4.4 Implications for further research

This simulation demonstrates a situation in which Random Forest MICE methods have an advantage over parametric MICE. Both Doove’s method (RF) and our method (RFcont) performed well, and on some performance measures Doove’s method was superior.

It would be useful to compare these methods in simulations based on real datasets.

5 Appendix: R code

5.1 R functions

This R code needs to be run in order to load the necessary functions before running the script (Section 5.2).

5.1.1 Data generating functions

makeSurv <- function(n = 2000, loghr = kLogHR){# Creates a survival cohort of n patients. Assumes that censoring is# independent of all other variables# x1 and x2 are random normal variablesdata <- data.frame(x1 = rnorm(n), x2 = rnorm(n))# Create the x3 variabledata$x3 <- 0.5 * (data$x1 + data$x2 - data$x1 * data$x2) + rnorm(n)# Underlying log hazard ratio for all variables is the samedata$y <- with(data, loghr * (x1 + x2 + x3))data$survtime <- rexp(n, exp(data$y))# Censoring - assume uniform distribution of observation times# up to a maximumobstime <- runif(nrow(data), min = 0,max = quantile(data$survtime, 0.5))data$event <- as.integer(data$survtime <= obstime)# Generate integer survival timesdata$time <- ceiling(100 * pmin(data$survtime, obstime))# Observed marginal cumulative hazard for imputation modelsdata$cumhaz <- nelsonaalen(data, time, event)# True log hazard and survival time are not seen in the data# so remove themdata$y <- NULLdata$survtime <- NULLreturn(data)}<bytecode: 0x5cfb25bd78a8>

makeMarSurv <- function(data, pmissing = kPmiss){# Introduces missing data dependent on event indicator# and cumulative hazard and x1 and x2logistic <- function(x){exp(x) / (1 + exp(x))}predictions <- function(lp, n){# uses the vector of linear predictions (lp) from a logistic model# and the expected number of positive responses (n) to generate# a set of predictions by modifying the baselinetrialn <- function(lptrial){sum(logistic(lptrial))}stepsize <- 32lptrial <- lp# To avoid errors due to missing linear predictors (ideally# there should not be any missing), replace with the meanif (any(is.na(lptrial))){lp[is.na(lptrial)] <- mean(lptrial, na.rm = TRUE)}while(abs(trialn(lptrial) - n) > 1){if (trialn(lptrial) > n){# trialn bigger than requiredlptrial <- lptrial - stepsize} else {lptrial <- lptrial + stepsize}stepsize <- stepsize / 2}# Generate predictions from binomial distributionas.logical(rbinom(logical(length(lp)), 1, logistic(lptrial)))}data$x3[predictions(0.1 * data$x1 + 0.1 * data$x2 +0.1 * data$cumhaz + 0.1 * data$event, nrow(data) * pmissing)] <- NAreturn(data)}<bytecode: 0x5cfb24e4a298>

5.1.2 Functions to analyse data

coxfull <- function(data){# Full data analysiscoefs <- as.data.frame(summary(coxph(myformula, data = data))$coef)# return a data.frame of coefficients (est), upper and lower 95% limitsout <- data.frame(est = coefs[, 'coef'],lo95 = coefs[, 'coef'] + qnorm(0.025) * coefs[, 'se(coef)'],hi95 = coefs[, 'coef'] + qnorm(0.975) * coefs[, 'se(coef)'],row.names = row.names(coefs))out$cover <- kLogHR >= out$lo95 & kLogHR <= out$hi95out}

coximpute <- function(imputed_datasets){# Analyses a list of imputed datasetsdocoxmodel <- function(data){coxph(myformula, data = data)}mirafits <- as.mira(lapply(imputed_datasets, docoxmodel))coefs <- as.data.frame(summary(pool(mirafits)))if ('term' %in% colnames(coefs)){row.names(coefs) <- as.character(coefs$term)}if (!('lo 95' %in% colnames(coefs))){# newer version of mice# use normal approximation for now, as assume large sample# and large degrees of freedom for t distributionout <- data.frame(est = coefs$estimate,lo95 = coefs$estimate + qnorm(0.025) * coefs$std.error,hi95 = coefs$estimate + qnorm(0.975) * coefs$std.error,row.names = row.names(coefs))} else if ('lo 95' %in% colnames(coefs)){# older version of miceout <- data.frame(est = coefs$est,lo95 = coefs[, 'lo 95'], hi95 = coefs[, 'hi 95'],row.names = row.names(coefs))} else {stop('Unable to handle format of summary.mipo object')}# Whether this confidence interval contains the true hazard ratioout$cover <- kLogHR >= out$lo95 & kLogHR <= out$hi95out}

domissf <- function(missdata, reps = NIMPS){# Imputation by missForestout <- list()for (i in 1:reps){invisible(capture.output(out[[i]] <- missForest(missdata)$ximp))}out}

mice.impute.cart <- function (y, ry, x, wy = NULL, minbucket = 5, cp = 1e-04, ...){ install.on.demand("rpart") if (is.null(wy)) { wy <- !ry } minbucket <- max(1, minbucket) if (dim(x)[2] == 0) { x <- cbind(x, 1) dimnames(x) <- list(NULL, "int") } xobs <- data.frame(x[ry, , drop = FALSE]) xmis <- data.frame(x[wy, , drop = FALSE]) yobs <- y[ry] if (!is.factor(yobs)) { fit <- rpart::rpart(yobs ~ ., data = cbind(yobs, xobs), method = "anova", control = rpart::rpart.control(minbucket = minbucket, cp = cp, ...)) leafnr <- floor(as.numeric(row.names(fit$frame[fit$where, ]))) fit$frame$yval <- as.numeric(row.names(fit$frame)) nodes <- predict(object = fit, newdata = xmis) donor <- lapply(nodes, function(s) yobs[leafnr == s]) impute <- vapply(seq_along(donor), function(s) sample(donor[[s]], 1), numeric(1)) } else { cat.has.all.obs <- table(yobs) == sum(ry) if (any(cat.has.all.obs)) { return(rep(levels(yobs)[cat.has.all.obs], sum(wy))) } xy <- cbind(yobs, xobs) xy <- droplevels(xy) fit <- rpart::rpart(yobs ~ ., data = xy, method = "class", control = rpart::rpart.control(minbucket = minbucket, cp = cp, ...)) nodes <- predict(object = fit, newdata = xmis) impute <- apply(nodes, MARGIN = 1, FUN = function(s) { sample(colnames(nodes), size = 1, prob = s) }) } impute}<bytecode: 0x5cfb257d6a48><environment: namespace:mice>

mice.impute.rfdoove10 <- function(y, ry, x, ...){mice::mice.impute.rf(y = y, ry = ry, x = x, ntrees = 10)}

mice.impute.rfdoove100 <- function(y, ry, x, ...){mice::mice.impute.rf(y = y, ry = ry, x = x, ntrees = 100)}

mice.impute.rfcont5 <- function(y, ry, x, ...){CALIBERrfimpute::mice.impute.rfcont(y = y, ry = ry, x = x, ntree_cont = 5)}

mice.impute.rfcont10 <- function(y, ry, x, ...){CALIBERrfimpute::mice.impute.rfcont(y = y, ry = ry, x = x, ntree_cont = 10)}

mice.impute.rfcont20 <- function(y, ry, x, ...){CALIBERrfimpute::mice.impute.rfcont(y = y, ry = ry, x = x, ntree_cont = 20)}

mice.impute.rfcont100 <- function(y, ry, x, ...){CALIBERrfimpute::mice.impute.rfcont(y = y, ry = ry, x = x, ntree_cont = 100)}

domice <- function(missdata, functions, reps = NIMPS){mids <- mice(missdata, defaultMethod = functions,m = reps, visitSequence = 'monotone',printFlag = FALSE, maxit = 10)lapply(1:reps, function(x) complete(mids, x))}

doanalysis <- function(x){# Creates a dataset, analyses it using different methods, and outputs# the result as a matrix of coefficients / SE and coveragedata <- makeSurv(kSampleSize)missdata <- makeMarSurv(data)out <- list()out$full <- coxfull(data)out$missf <- coximpute(domissf(missdata))out$rf5 <- coximpute(domice(missdata, 'rfcont5'))out$rf10 <- coximpute(domice(missdata, 'rfcont10'))out$rf20 <- coximpute(domice(missdata, 'rfcont20'))out$rf100 <- coximpute(domice(missdata, 'rfcont100'))out$rfdoove10 <- coximpute(domice(missdata, 'rfdoove10'))out$rfdoove100 <- coximpute(domice(missdata, 'rfdoove100'))out$cart <- coximpute(domice(missdata, 'cart'))out$mice <- coximpute(domice(missdata, 'norm'))out}

5.1.3 Functions to compare methods

pstar <- function(x){if (!is.na(x)){if (x < 0.001){'***'} else if (x < 0.01){'**'} else if (x < 0.05){'*'} else {''}} else {''}}<bytecode: 0x5cfb25485d90>

compareBias <- function(method1, method2){# Generates a table comparing bias# Comparison statistic is the difference in absolute bias# (negative means first method is better)compareBiasVar <- function(varname){# All coefficients should be kLogHRbias1 <- sapply(results, function(x){x[[method1]][varname, 'est']}) - kLogHRbias2 <- sapply(results, function(x){x[[method2]][varname, 'est']}) - kLogHRif (sign(mean(bias1)) == -1){bias1 <- -bias1}if (sign(mean(bias2)) == -1){bias2 <- -bias2}paste(formatC(mean(bias1) - mean(bias2), format = 'fg', digits = 3),pstar(t.test(bias1 - bias2)$p.value))}sapply(variables, compareBiasVar)}<bytecode: 0x5cfb250b5448>

compareVariance <- function(method1, method2){# Generates a table comparing precision between two methods# Comparison statistic is ratio of variance# (smaller means first method is better)compareVarianceVar <- function(varname){e1 <- sapply(results, function(x){x[[method1]][varname, 'est']})e2 <- sapply(results, function(x){x[[method2]][varname, 'est']})paste(formatC(var(e1) / var(e2), format = 'fg', digits = 3),pstar(var.test(e1, e2)$p.value))}sapply(variables, compareVarianceVar)}<bytecode: 0x5cfb26591060>

compareCIlength <- function(method1, method2){# Generates a table comparing coverage percentage between two methods# Comparison statistic is the ratio of confidence interval lengths# (less than 1 = first better)compareCIlengthVar <- function(varname){# Paired t test for bias (difference in estimate)len1 <- sapply(results, function(x){x[[method1]][varname, 'hi95'] -x[[method1]][varname, 'lo95']})len2 <- sapply(results, function(x){x[[method2]][varname, 'hi95'] -x[[method2]][varname, 'lo95']})paste(formatC(mean(len1) / mean(len2),format = 'fg', digits = 4),pstar(t.test(len1 - len2)$p.value))}sapply(variables, compareCIlengthVar)}<bytecode: 0x5cfb27258708>

compareCoverage <- function(method1, method2){# Generates a table comparing coverage percentage between two methods# Comparison statistic is the difference in coverage# (positive = first better)compareCoverageVar <- function(varname){# Paired t test for bias (difference in estimate)cov1 <- sapply(results, function(x){x[[method1]][varname, 'cover']})cov2 <- sapply(results, function(x){x[[method2]][varname, 'cover']})paste(formatC(100 * (mean(cov1) - mean(cov2)), format = 'f',digits = 1),pstar(binom.test(c(sum(cov1 == TRUE & cov2 == FALSE),sum(cov1 == FALSE & cov2 == TRUE)))$p.value))}sapply(variables, compareCoverageVar)}<bytecode: 0x5cfb246c3d88>

5.1.4 Functions to compile and display results

getParams <- function(coef, method){estimates <- sapply(results, function(x){x[[method]][coef, 'est']})bias <- mean(estimates) - kLogHRse_bias <- sd(estimates) / sqrt(length(estimates))mse <- mean((estimates - kLogHR) ^ 2)ci_len <- mean(sapply(results, function(x){x[[method]][coef, 'hi95'] - x[[method]][coef, 'lo95']}))ci_cov <- mean(sapply(results, function(x){x[[method]][coef, 'cover']}))out <- c(bias, se_bias, mse, sd(estimates), ci_len, ci_cov)names(out) <- c('bias', 'se_bias', 'mse', 'sd', 'ci_len', 'ci_cov')out}<bytecode: 0x5cfb25b6f3c0>

showTable <- function(coef){methods <- c('full', 'missf', 'cart', 'rfdoove10','rfdoove100', 'rf5', 'rf10', 'rf20', 'rf100', 'mice')methodnames <- c('Full data', 'missForest', 'CART MICE','RF Doove MICE 10', 'RF Doove MICE 100',paste('RFcont MICE', c(5, 10, 20, 100)),'Parametric MICE')out <- t(sapply(methods, function(x){getParams(coef, x)}))out <- formatC(out, digits = 3, format = 'fg')out <- rbind(c('', 'Standard', 'Mean', 'SD of', 'Mean 95%','95% CI'), c('Bias', 'error of bias', 'square error', 'estimate','CI length', 'coverage'), out)out <- cbind(c('', '', methodnames), out)rownames(out) <- NULLprint(xtable(out), floating = FALSE, include.rownames = FALSE,include.colnames = FALSE, hline.after = c(0, 2, nrow(out)))}<bytecode: 0x5cfb24f857b0>

maketable <- function(comparison){# comparison is a function such as compareCoverage, compareBiascompare <- cbind(comparison('rf10', 'mice'),comparison('rf100', 'mice'),comparison('rf10', 'rf100'))compare <- cbind(rownames(compare), compare)compare <- rbind(c('', 'RFcont MICE 10 vs', 'RFcont MICE 100 vs','RFcont MICE 10 vs'),c('Coefficient', 'parametric MICE','parametric MICE', 'RFcont MICE 100'),compare)rownames(compare) <- NULLprint(xtable(compare), include.rownames = FALSE,include.colnames = FALSE, floating = FALSE,hline.after = c(0, 2, nrow(compare)))cat('\n\\vspace{1em}\n')compare <- cbind(comparison('rfdoove10', 'rf10'),comparison('rfdoove10', 'cart'),comparison('rfdoove10', 'rfdoove100'))compare <- cbind(rownames(compare), compare)compare <- rbind(c('', 'RF Doove MICE 10 vs', 'RF Doove MICE 10 vs','RF Doove MICE 10 vs'),c('Coefficient', 'RFcont MICE 10','CART MICE', 'RF Doove MICE 100'),compare)rownames(compare) <- NULLprint(xtable(compare), include.rownames = FALSE,include.colnames = FALSE, floating = FALSE,hline.after = c(0, 2, nrow(compare)))}<bytecode: 0x5cfb235d89e8>

5.2 R script

Run this script after loading the functions above.

library(CALIBERrfimpute)library(missForest)library(survival)library(xtable)library(parallel) # Use parallel processing on Unix# Initialise constantskPmiss <- 0.2 # probability of missingnesskLogHR <- 0.5 # true log hazard ratio# Set number of patients in simulated datasetsNPATS <- 2000# Set number of samplesN <- 1000# Set number of imputationsNIMPS <- 10# Perform the simulationresults <- mclapply(1:N, doanalysis)# Show resultsshowTable('x1'); showTable('x2'); showTable('x3')# Names of the variables in the comparisonvariables <- c('x1', 'x2', 'x3')# Show comparisons between methodsmaketable(compareBias)maketable(compareVariance)maketable(compareCIlength)maketable(compareCoverage)

References

  • [1] Shah AD, Bartlett JW, Carpenter J, Nicholas O, Hemingway H. Comparison of Random Forest and Parametric Imputation Models for Imputing Missing Data Using MICE: A CALIBER Study. American Journal of Epidemiology 2014. doi: 10.1093/aje/kwt312
  • [2] Doove LL, van Buuren S, Dusseldorp E. Recursive partitioning for missing data imputation in the presence of interaction effects. Computational Statistics and Data Analysis 2014;72:92–104. doi: 10.1016/j.csda.2013.10.025