Interval Regression with Sample Selection
This ‘vignette’ is largely based on Petersen, Henningsen and Henningsen (2017).
1 Model Specification
The general specification of an interval regression model with sample selection is:
| (1) | ||||
| (2) | ||||
| (3) | ||||
| (4) | ||||
where subscript indicates the observation, is a latent outcome variable, is a partially observed categorical variable that indicates in which interval lies, is the number of intervals, are the boundaries of the intervals (whereas frequently but not necessarily and ), is a binary variable that indicates whether is observed, is a latent variable that indicates the “tendency” that is one, and are (column) vectors of explanatory variables for the selection equation and outcome equation, respectively, and are random disturbance terms that have a joint bivariate normal distribution, and and are (column) vectors and and are scalars of unknown model parameters.
2 Log-Likelihood Function
The probability that is unobserved is:
| (9) | ||||
| (10) | ||||
| (11) |
The probability that is observed and indicates that lies in the th interval is:
| (12) | ||||
| (13) | ||||
| (14) |
The log-likelihood contribution of the th observation is:
| (15) | ||||
where indicates the cumulative distribution function of the univariate standard normal distribution and indicates the cumulative distribution function of the bivariate standard normal distribution.
3 Restricting coefficients and
The parameter needs to be in the interval . In order to restrict to be in this interval, we estimate instead of so that the derived parameter is always in the interval . We use the delta method to calculate approximate standard errors of the derived parameter , whereas the corresponding element of the Jacobian matrix is:
| (16) |
The parameter needs to be strictly positive, i.e. . In order to restrict to be strictly positive, we estimate instead of or so that the derived parameters and are always strictly positive. We use the delta method to calculate approximate standard errors of the derived parameters and , whereas the corresponding elements of the Jacobian matrix are:
| (17) | ||||
| (18) |
4 Gradients of the CDF of the bivariate standard normal distribution
In order to facilitate the calculation of the gradients of the log-likelihood function, we calculate the partial derivatives of the cumulative distribution function (CDF) of the bivariate standard normal distribution:
| (19) |
where is the probability density function (PDF) of the bivariate standard normal distribution:
| (20) |
In the following, we check equation (20) by a simple numerical example:
> library( "mvtnorm" )> library( "maxLik" )> x1 <- 0.4> x2 <- -0.3> rho <- -0.6> sigma <- matrix( c( 1, rho, rho, 1 ), nrow = 2 )> dens <- dmvnorm( c( x1, x2 ), sigma = sigma )> print( dens )
[1] 0.1831324
> all.equal( dens, ( 2 * pi * sqrt( 1 - rho^2 ) )^(-1) *+ exp( - ( x1^2 - 2 * rho * x1 * x2 + x2^2 ) / ( 2 * ( 1 - rho^2 ) ) ) )
[1] TRUE
4.1 Gradients with respect to the limits ( and )
| (21) | ||||
| (22) | ||||
| (23) | ||||
| (24) | ||||
| (25) | ||||
| (26) | ||||
| (27) |
where indicates the density function of a normal distribution with mean and variance .
In the following, we use the same simple numerical example as in the beginning of section 4 to check the above derivations. First, we check whether the PDF of the bivariate standard normal distribution, i.e. (part of equation 21), is equal to (part of equation 23) and equal to (part of equations 24 and 25):
> all.equal( dens, dnorm( x1, rho * x2, sqrt( 1 - rho^2 ) ) * dnorm(x2) )
[1] TRUE
> all.equal( dens, ( dnorm( ( x1 - rho * x2 ) / sqrt( 1 - rho^2 ) ) /+ sqrt( 1 - rho^2 ) ) * dnorm(x2) )
[1] TRUE
In the following, we will numerically calculate the derivative of the cumulative distribution function of the bivaraite normal distribution (equation 19) with respect to and check wehther this partial derivative is equal to the right-hand sides of equations (21), (24), (25), and (27):
> funX2 <- function( a2 ) {+ prob <- pmvnorm( upper = c( x1, a2 ), sigma = sigma )+ return( prob )+ }> grad <- c( numericGradient( funX2, x2 ) )> print( grad )
[1] 0.2320142
> funX1 <- function( a1 ) {+ dens <- rep( NA, length( a1 ) )+ for( i in 1:length( a1 ) ) {+ dens[i] <- dmvnorm( c( a1[i], x2 ), sigma = sigma )+ }+ return( dens )+ }> all.equal( grad, integrate( funX1, lower = -Inf, upper = x1 )$value )
[1] TRUE
> funX1a <- function( a1 ) {+ dens <- rep( NA, length( a1 ) )+ for( i in 1:length( a1 ) ) {+ dens[i] <- ( dnorm( ( a1[i] - rho * x2 ) / sqrt( 1 - rho^2 ) ) /+ sqrt(1-rho^2) ) * dnorm(x2)+ }+ return( dens )+ }> all.equal( grad, integrate( funX1a, lower = -Inf, upper = x1 )$value )
[1] TRUE
> funX1b <- function( a1 ) {+ dens <- rep( NA, length( a1 ) )+ for( i in 1:length( a1 ) ) {+ dens[i] <- dnorm( ( a1[i] - rho * x2 ) / sqrt( 1 - rho^2 ) ) /+ sqrt(1-rho^2)+ }+ return( dens )+ }> all.equal( grad,+ integrate( funX1b, lower = -Inf, upper = x1 )$value * dnorm(x2) )
[1] TRUE
> all.equal( grad,+ pnorm( ( x1 - rho * x2 ) / sqrt( 1 - rho^2 ) ) * dnorm( x2 ) )
[1] TRUE
4.2 Gradients with respect to the coefficient of correlation ()
| (28) | ||||
| (29) | ||||
| (30) | ||||
| (31) | ||||
| (32) | ||||
| (33) | ||||
| (34) | ||||
| (35) | ||||
| (36) | ||||
| (37) | ||||
| (38) | ||||
| (39) | ||||
| (40) | ||||
| (41) | ||||
| (42) | ||||
Applying L’Hospital on the last term leads to
| (43) | ||||
| (44) | ||||
| (45) | ||||
| (46) | ||||
| (47) |
This result is in line with Sibuya (1960) and Sungur (1990).
In the following, we will numerically calculate the derivative of the cumulative distribution function of the bivariate normal distribution (equation 29) with respect to and check whether this partial derivative is equal to the right-hand sides of equation (47):
> # Numerical gradient of the PDF w.r.t. rho> funrho <- function( p ) {+ prob <- dmvnorm( x = c( x1, x2 ),+ sigma = matrix( c( 1, p, p, 1 ), nrow = 2 ) )+ return( prob )+ }> grad <- c( numericGradient( funrho, rho ) )> print( grad )
[1] -0.1775883
> # Comparison with analytical gradient for rho> efun <- exp(-(x1^2 - 2 * rho * x1 * x2 + x2^2)/(2*(1 - rho^2)))> all.equal( grad,+ ( (-((2*rho*(-2*rho*x1*x2+x1^2+x2^2) - 2*x1*x2*(1-rho^2)) * efun)/+ (2*(1-rho^2)^(3/2) )) ++ ((rho*efun)/(sqrt(1-rho^2))) ) /+ (2*pi*(1-rho^2)) )
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
> # Numerical gradient of the CDF w.r.t. rho> cdfRho <- function( p, xa = x1, xb = x2 ) {+ prob <- pmvnorm( upper = c( xa, xb ),+ sigma = matrix( c( 1, p, p, 1 ), nrow = 2 ) )+ return( prob )+ }> grad <- c( numericGradient( cdfRho, rho ) )> print( grad )
[1] 0.1831324
> # comparison with analytical gradient> all.equal( grad, dmvnorm( x = c( x1, x2 ),+ sigma = matrix( c( 1, rho, rho, 1 ), nrow = 2 ) ) )
[1] TRUE
> # comparisons with other values> compDerivRho <- function( xa, xb, p ) {+ dn <- c( numericGradient( cdfRho, p, xa = xa, xb = xb ) )+ da <- dmvnorm( x = c( xa, xb ),+ sigma = matrix( c( 1, p, p, 1 ), nrow = 2 ) )+ return( all.equal( dn, da ) )+ }> compDerivRho( x1, x2, rho )
[1] TRUE
> compDerivRho( 0.5, x2, rho )
[1] TRUE
> compDerivRho( 2.5, x2, rho )
[1] TRUE
> compDerivRho( x1, -2, rho )
[1] TRUE
> compDerivRho( x1, x2, 0.2 )
[1] TRUE
> compDerivRho( x1, x2, 0.98 )
[1] TRUE
5 Gradients of the Log-Likelihood Function
5.1 Gradients with respect to the parameters of the selection equation ()
First, we use equation (27), to determine the derivative of the bivariate standard normal distribution with respect to the parameter as part of the loglikelihood function:
| (48) | ||||
| (49) |
Using this result we can now derive the gradient for in the log-likelihood function:
| (50) | ||||
| (51) | ||||
| (52) | ||||
| (53) | ||||
| (54) | ||||
5.2 Gradients with respect to the parameters in the outcome equation ()
Analogous to and by using equation (27) we derive the gradient of :
| (55) |
Using this result we derive the gradient for the outcome parameter for the log-likelihood function:
| (56) | ||||
| (57) | ||||
| (58) | ||||
| (59) | ||||
| (60) | ||||
5.3 Gradients with respect to the coefficient of correlation ()
Given the result that the derivative of the CDF with respect to is equal to the PDF (see equation 47), we can also derive the gradient of the correlation parameter ():
| (61) | ||||
| (62) | ||||
| (63) |
| (64) |
5.4 Gradients with respect to the standard deviation used for normalisation ()
Finally, we derive the gradient for in the same way as we did for and :
| (65) |
| (66) | ||||
| (67) | ||||
| (68) | ||||
| (69) | ||||
| (70) | ||||
| (71) | ||||
| (72) |
Similarly:
| (73) |
| (74) | ||||
| (75) | ||||
| (76) | ||||
| (77) | ||||
| (78) |
6 Example with a Generated Dataset
7 Generate Dataset
> library( "mvtnorm" )> # number of observations> nObs <- 300> # parameters> betaS <- c( 1, 1, -1 )> betaO <- c( 10, 4 )> rho <- 0.4> sigma <- 5> # boundaries of the intervals> bound <- c(-Inf,5,15,Inf)> # set 'seed' of the pseudo random number generator> # in order to always generate the same pseudo random numbers> set.seed(123)> # generate variables x1 and x2> dat <- data.frame( x1 = rnorm( nObs ), x2 = rnorm( nObs ) )> # variance-covariance matrix of the two error terms> vcovMat <- matrix( c( 1, rho*sigma, rho*sigma, sigma^2 ), nrow = 2 )> # generate the two error terms> eps <- rmvnorm( nObs, sigma = vcovMat )> dat$epsS <- eps[,1]> dat$epsO <- eps[,2]> # generate the selection variable> dat$yS <- with( dat, betaS[1] + betaS[2] * x1 + betaS[3] * x2 + epsS ) > 0> table( dat$yS )
FALSE TRUE 91 209
> # generate the unobserved/latent outcome variable> dat$yOu <- with( dat, betaO[1] + betaO[2] * x1 + epsO )> dat$yOu[ !dat$yS ] <- NA> # obtain the intervals of the outcome variable> dat$yO <- cut( dat$yOu, bound )> table( dat$yO )
(-Inf,5] (5,15] (15, Inf] 26 130 53
7.1 Estimation of the Model
In the following estimation, the starting values are obtained by a maximum-likelihood (ML) estimation of a tobit-2 model, where the dependent variable of the outcome equation is set to the mid points of the intervals:
> library( "sampleSelection" )> res <- selection( yS ~ x1 + x2, yO ~ x1, data = dat, boundaries = bound )> res
Call: selection(selection = yS ~ x1 + x2, outcome = yO ~ x1, data = dat, boundaries = bound)Coefficients:S:(Intercept) S:x1 S:x2 O:(Intercept) O:x1 0.9821 0.9668 -1.2862 10.2419 2.6586 logSigma atanhRho sigma sigmaSq rho 1.6307 0.2976 5.1076 26.0879 0.2891
> summary( res )
--------------------------------------------Tobit 2 model with interval outcome (sample selection model)Maximum Likelihood estimationBHHH maximisation, 22 iterationsReturn code 8: successive function values within relative tolerance limit (reltol)Log-Likelihood: -275.395300 observations (91 censored and 209 observed)Intervals of the dependent variable of the outcome equation: YO lower upper count1 (-Inf,5] -Inf 5 262 (5,15] 5 15 1303 (15, Inf] 15 Inf 537 free parameters (df = 293)Probit selection equation: Estimate Std. Error t value Pr(>|t|)(Intercept) 0.9821 0.1085 9.050 < 2e-16 ***x1 0.9668 0.1491 6.484 3.78e-10 ***x2 -1.2862 0.1209 -10.636 < 2e-16 ***Outcome equation: Estimate Std. Error t value Pr(>|t|)(Intercept) 10.2419 0.6683 15.325 < 2e-16 ***x1 2.6586 0.5922 4.489 1.03e-05 *** Error terms: Estimate Std. Error t value Pr(>|t|)logSigma 1.63074 0.07472 21.825 < 2e-16 ***atanhRho 0.29757 0.36189 0.822 0.412sigma 5.10763 0.38163 13.384 < 2e-16 ***sigmaSq 26.08791 3.89850 6.692 1.12e-10 ***rho 0.28909 0.33164 0.872 0.384---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1--------------------------------------------
In the following estimation, the starting values are obtained by a two-step estimation of a tobit-2 model, where the dependent variable of the outcome equation is set to the mid points of the intervals:
> res2 <- selection( yS ~ x1 + x2, yO ~ x1, data = dat, boundaries = bound,+ start = "2step" )> res2
Call: selection(selection = yS ~ x1 + x2, outcome = yO ~ x1, data = dat, start = "2step", boundaries = bound)Coefficients:S:(Intercept) S:x1 S:x2 O:(Intercept) O:x1 0.9821 0.9668 -1.2862 10.2419 2.6586 logSigma atanhRho sigma sigmaSq rho 1.6307 0.2976 5.1076 26.0879 0.2891
> summary( res2 )
--------------------------------------------Tobit 2 model with interval outcome (sample selection model)Maximum Likelihood estimationBHHH maximisation, 22 iterationsReturn code 8: successive function values within relative tolerance limit (reltol)Log-Likelihood: -275.395300 observations (91 censored and 209 observed)Intervals of the dependent variable of the outcome equation: YO lower upper count1 (-Inf,5] -Inf 5 262 (5,15] 5 15 1303 (15, Inf] 15 Inf 537 free parameters (df = 293)Probit selection equation: Estimate Std. Error t value Pr(>|t|)(Intercept) 0.9821 0.1085 9.050 < 2e-16 ***x1 0.9668 0.1491 6.484 3.78e-10 ***x2 -1.2862 0.1209 -10.636 < 2e-16 ***Outcome equation: Estimate Std. Error t value Pr(>|t|)(Intercept) 10.2419 0.6683 15.325 < 2e-16 ***x1 2.6586 0.5922 4.489 1.03e-05 *** Error terms: Estimate Std. Error t value Pr(>|t|)logSigma 1.63074 0.07472 21.825 < 2e-16 ***atanhRho 0.29757 0.36189 0.822 0.412sigma 5.10763 0.38163 13.384 < 2e-16 ***sigmaSq 26.08791 3.89850 6.692 1.12e-10 ***rho 0.28909 0.33164 0.872 0.384---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1--------------------------------------------
The following commands compare the starting values and the estimated coefficients:
> # compare starting values (small differences)> cbind( res$start, res2$start, res$start - res2$start )
[,1] [,2] [,3](Intercept) 0.9818072 0.9800827 0.001724574x1 0.9663185 0.9686013 -0.002282885x2 -1.2866893 -1.2808419 -0.005847461(Intercept) 10.3990738 10.3516223 0.047451510x1 3.7408112 3.7642665 -0.023455273logSigma 4.2797893 4.2813219 -0.001532652atanhRho 0.2362382 0.2550566 -0.018818413
> # combare estimated coefficients (virtually identical)> cbind( coef( res ), coef( res2 ), coef( res ) - coef( res2 ) )
[,1] [,2] [,3](Intercept) 0.9820618 0.9820617 1.626690e-07x1 0.9668069 0.9668068 1.116395e-07x2 -1.2861861 -1.2861863 1.863040e-07(Intercept) 10.2419399 10.2419334 6.516212e-06x1 2.6585966 2.6586014 -4.744749e-06logSigma 1.6307359 1.6307360 -8.697063e-08atanhRho 0.2975695 0.2975745 -4.901548e-06sigma 5.1076321 5.1076326 -4.442140e-07sigmaSq 26.0879061 26.0879107 -4.537764e-06rho 0.2890868 0.2890913 -4.491913e-06
8 Example with the ‘Smoke’ dataset
The following command loads the dataset:
> data( "Smoke" )
The following command creates a vector with the boundaries of the intervals:
> bounds <- c( 0, 5, 10, 20, 50, Inf )
The following command estimates the model with few explanatory variables:
> SmokeRes1 <- selection( smoker ~ educ + age,+ cigs_intervals ~ educ, data = Smoke, boundaries = bounds )
The following command estimates the model with more explanatory variables:
> SmokeRes2 <- selection( smoker ~ educ + age + restaurn,+ cigs_intervals ~ educ + income + restaurn, data = Smoke,+ boundaries = bounds )
The following commands test whether adding further explanatory variables significantly improves the explanatory power of the model:
> library( "lmtest" )> lrtest( SmokeRes1, SmokeRes2 )
Likelihood ratio testModel 1: selection(selection = smoker ~ educ + age, outcome = cigs_intervals ~ educ, data = Smoke, boundaries = bounds)Model 2: selection(selection = smoker ~ educ + age + restaurn, outcome = cigs_intervals ~ educ + income + restaurn, data = Smoke, boundaries = bounds) #Df LogLik Df Chisq Pr(>Chisq)1 7 -940.542 10 -936.30 3 8.4705 0.03723 *---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> waldtest( SmokeRes1, SmokeRes2 )
Wald testModel 1: selection(selection = smoker ~ educ + age, outcome = cigs_intervals ~ educ, data = Smoke, boundaries = bounds)Model 2: selection(selection = smoker ~ educ + age + restaurn, outcome = cigs_intervals ~ educ + income + restaurn, data = Smoke, boundaries = bounds) Res.Df Df Chisq Pr(>Chisq)1 8002 797 3 7.8636 0.04892 *---Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Both tests indicate that—at 5% significance level—the model with more explanatory variables (SmokeRes2) has significantly higher explanatory power than the model with fewer explanatory variables (SmokeRes1).
References
- Can changes in households’ life situations predict participation in energy audits and investments in energy savings?. Note: Unpublished Manuscript. Department of Management Engineering, Technical University of Denmark Cited by: Interval Regression with Sample Selection.
- Bivariate extreme statistics, i. Annals of the Institute of Statistical Mathematics 11 (2), pp. 195 – 210. Cited by: §4.2.
- Dependence information in parameterized copulas. Communications in Statistics - Simulation and Computation 19 (4), pp. 1339 – 1360. Cited by: §4.2.