Regression Models with Stochastic Regressors : An Expository note

Regression models form the core of the discipline of econometrics. One of the basic assumptions of classical linear regression model is that the values of the explanatory variables are fixed in repeated sampling. However, in most of the real life cases, particularly in economics the assumption of fixed regressors is not always tenable. Under a non-experimental or uncontrolled environment, the dependent variable is often under the influence of explanatory variables that are stochastic in nature. There is a huge literature related to stochastic regressors in various aspects. In this paper, a historical perspective on some of the works related to stochastic regressor is being tried to pen down based on literature search.


Introduction
In the realm of present day economic relationship and due to dynamic increase of demands of rapidly increasing population, econometric has become an important task in relatively every field.
Econometric is the art of utilizing the factual techniques for the measurement of economic relations.
An econometric study begins with a set of propositions about some characteristics of the economy.
The theory specifies a set of precise, deterministic relationships among factors.An econometric model is a simplified representation of a real world process.
Regression models form the foundation of the discipline of econometrics.This class of model begins with a simple linear regression model.It is a statistical technique for investigating and modeling relationships between two variables i.e. the explained and the explanatory variable.It helps to estimate the strength and direction of the relationship, allowing predictions about past, present or future events to be made with information about past or present events.In broad sense, there are two types of linear regression models-General Linear Regression Models and Generalized Linear Regression Models.This classification is based on the distribution of the disturbance term.The usual notation of a regression model may be defined as- where  is the vector of responses, X is the data matrix and ̃ is the identically and independently distributed (iid) vector of disturbance terms or errors. is the vector of regression coefficient that The distribution of the random variable e remains the same for all explanatory variables.If e follows Normal distribution with mean 0 and variance  , it is called Classical Linear Normal Regression Model.However, the usual assumption of normality and constant variance are not always satisfied.The alternative approach to this is called the Generalized Linear Regression Models where e follows the distribution of the members of the exponential family of distribution, which includes Normal, Binomial, Poisson, Exponential, Gamma distribution, etc as its member.Also, the distribution of the random variable e remains the same for all explanatory variables.
Another important assumption of regression model is explanatory variables are fixed in repeated samples.However, in many cases the assumption of non-stochastic regressor is not always tenable.Thus, under a non-experimental or uncontrolled environment, the dependent variable is often under the influence of explanatory variables that are stochastic in nature.
A problem of major importance is to estimate the parameters ′and test the null hypothesis  :  = 0. Clearly,  implies that the design variable has no effect on the response Y. Traditionally, assume that e's (1 <i <n) are iid normal N(0, ).Various procedures have been developed for parameter estimation and inference in linear regression.Least Squares method is applied which give the best linear unbiased estimator when the basic assumptions were fulfilled.Since the method of maximum likelihood has all the desirable optimal properties under some general regularity conditions and the distribution of the error terms is known to belong to a certain parametric family of probability distributions, so we can employ this method also to estimate the parameters.
The resulting estimate is identical to the OLS estimate when the distribution of the error term is normal.This work is devoted to historical perspective on stochastic regressions and its further developments.

Regression Models with Stochastic Regressor
Giving more emphasis on the assumption, that the design variable X is traditionally been taken as nonstochastic, it is realized that the assumption of nonstochastic regressor is not always plausible, the regressor may be stochastic in nature.The term stochastic regressor means that the regressors, i.e.
the explanatory variables are random with the change of time.The basic assumption in case of Stochastic regressors are: i) X, Y, e random ii) (X,Y) obtained from iid sampling iii) E(e|X)=0 iv) X takes atleast two values v) Var(e|X) = vi) e is normal.The Variables X, Y and e is already defined in section 1.
It has been recognized, however, that in numerous applications X might be stochastic and e might not be normal.This may give rise to three problems (a) X is nonstochastic and e is nonnormal, (b) X is stochastic and e is normal, and (c) X is stochastic and e is nonnormal.
The basic problem of the stochastic regressor in General Linear Model (GLM) is that the least square estimators may not be unbiased because when taking the expectation of the random vector b  (Islam and Tiku, 2010).

A Bird eye view on works of Stochastic Regressors
There is a huge literature related to stochastic regressors in various aspects.The basic assumption of classical linear regression model is that the values of the explanatory variables are fixed in repeated sampling, Hooper and Zellner (1961) treated the error of prediction of a multiple regression equation with constant regressor variable.Kerridge (1967) investigated the same but assumed that the regressor variables are drawn from a multivariate normal population.The k-dimensional vectors X1, X2, …,Xn are given as a random sample from the distribution N(Ʌ,Σ).Corresponding to this vector Y1, Y2, ... ,Yn was observed.The error of prediction was expressed as the ratio of a normally distributed random variable and the square root of an independent beta-variate.It was realized that though such regression has limitations if the number of regressor variables is large, it may also be useful if the limitations were well understood.
In hypothesis testing on the coefficients of a linear regression model with stochastic regressors, t and F test are applicable if the regressors are statistically independent of the disturbances.It was explained that there are also cases in which economic hypothesis can be stated in terms of the independence of stochastic regressors and disturbances.Wu (1973)  shown that all the tests were consistent.Wu (1974), determined analytically the set of parameters that enters the power functions of the four tests and then compared the power functions of these tests using sampling experiment.Here also he considered two linear models where each row of the disturbances is independently distributed as multivariate normal with mean zero and covariance .This test is proposed for those problems in which the statistical decision making can be formulated as a choice between two alternative hypothesis of whether the regressors and disturbances are independent.He gave four examples of its applications in econometrics including a test of the permanent income hypothesis.Narula (1971) presented that if prediction is the main objective, the mean square error of the predicted response from the observed response would seem to be a reasonable criterion, since it takes into account bias and variability simultaneously.Narula (1974), considered two approaches to improve the predictive mean square error of the predicted response when predictor variables are stochastic and follow a multivariate normal distribution.The first technique, the subset approach used only a subset of the available predictor variables to predict the response.To select the subset a decision rule was given.In the second method, the lambda approach, the regression coefficients were scaled down by a suitable constant.He then suggested an estimator of the constant.The two methods were then compared.Wu (1973Wu ( , 1974)), Revankar and Hartley (1973), Revankar (1978), and Hwang (1978)  In many occurrences it is valuable to know whether stochastic regressors are independent of the disturbance terms in regression models.Reynolds (1982) developed the bayesian approach based on posterior odds ratios to considering the hypothesis of independence between stochastic regressors and disturbances and then compares it with the Wu tests.Here two simultaneous equations are considered, where the disturbances term are assumed independently and identically multivariate normally distributed with zero means and positive definite symmetric covariance matrices .This procedure provides an alternative procedure for testing hypothesis of erogeneity.
The posterior probabilities of hypothesis that may be calculated from the posterior odds ratios also offer the intuitively appealing interpretation of reflecting the degree of belief that may be had in the hypothesis given the data. .He illustrated an example on an empirical model of the demand for health developed by Grossman (1972) and Wei (1982and Wei ( ,1983and Wei ( ,1985)).The results are then applied to yield strong consistency results for order estimation in nonstationary autoregressive models.Lai (1994) established asymptotic properties of nonlinear least square estimates in stochastic regression models.Stochastic regression models, where the random disturbances form a martingale difference sequence with respect to an increasing sequence of σ-fields {  } and  is a random  measurable function of an unknown parameter ɵ, and cover a broad range of nonlinear (and linear) time series stochastic process models.In the linear case they reduced to known results on the linear least squares estimate with stochastic measurable regressors.Hu (1997) used posterior covariance matrices to study the strong consistency of Bayes estimators in stochastic regression models under various assumptions on the stochastic regressors.The random errors were assumed to form a martingale difference sequence.Several results were obtained using a recursion satisfied by the sequence of posterior covariance matrices.These results suggested that the posterior covariance matrix is a useful tool in studying strong consistency problems in stochastic regression models.Three examples from sequential design and adaptive control are discussed.
In numerous situations, a bivariate random vector (X, Y) is such that Y depends on X but not so much the other way round.Vaughan and Tiku (2000) assumed the distribution of X to be the extreme value distribution and the conditional distribution of Y to be normal.Then they derived the MML (modified maximum likelihood) estimators of the five parameters and showed that they are highly efficient.They also developed hypothesis testing procedures.Akkaya and Tiku (2001)  ≤ t ≤ n) and showed that they are asymptotically fully efficient for all values of φ.Their properties for small sample sizes are studied and showed that they are, like the LS (least squares) estimators, almost unbiased for all values of φ but are considerably more efficient.They also developed procedures for testing the two important null hypotheses: (i) δ = 0, and (ii) φ = 0. Testing φ = 1 (the unit root problem) is of enormous practical interest but is considerably more difficult because the variance of yt is now infinite.
Magdalinos and kandilorou ( 2001) encompasses the problems of testing for structural restrictions and for valid orthogonality conditions into a single-equation framework in which the familiar specification techniques of the linear regression model can be applied.Identifying the endogenous variables that induce "simultaneous equation bias" will improve the efficiency of estimation, since only those variables need to be instrumented.
As in the linear regression model the regressors are traditionally assumed to be non stochastic, likewise it is often assumed that the random error is normally distributed.In numerous situations when dependent variable measures life times or reaction times, error term has skew distribution.Islam et al. (2001) considered two families of skew distributions, firstly Weibull with support IR: (0, ∞) on the real line and secondly generalized logistic with support IR: (-∞,∞).Then they derived Modified maximum likelihood estimators and showed that these estimators are efficient and robust.
Then simulations are done for the means and variances for several values of b (parameter) to compare the efficiencies of the MML estimators with the LS estimators.The biases in both the MML and the LS estimators were found to be negligible.Tiku et al. (2001) patronized the salient features of a family of short-tailed symmetric distribution, and introduced the mechanisms for creating inliers.They assumed the error distribution to be one of this family, the methodology of modified likelihood is used to derive MML estimators of parameters in a linear regression model.The estimators are efficient and are the first to achieve robustness to inliers.And the design variable xi are generated from a uniform (0,1) distribution and kept common for all N generated random samples.Then they further extended the case to long-tailed symmetric distributions and showed that the MML estimators have excellent efficiency and robustness properties.In particular, it can be said that they are robust to outliers.
In Regression Models, like Regressors, the errors are also traditionally assumed to be normally distributed.But in more recent years, there has been a realization that nonnormal distributions are more prevalent in practice.Islam and Tiku (2005) considered several nonnormal distributions for the random error e of the equation Yi= +  Xi + , (1 ≤ i ≤ n), both symmetric and skew.Then they obtained the modified maximum likelihood estimators (MMLEs) of  ,  ,  and compared them with the least square estimators (LSEs) and Huber M estimators (1981).They showed that for plausible deviations from an assumed distribution, the MMLEs are more efficient and much easier to compute.
In many statistical applications, the deviations from normality are very common where usually the normality assumption invoked.Therefore there has been enormous practical interest to develop statistical procedures which are robust to plausible deviations from an assumed distribution particularly normal.As not much is known about how to obtain robust parametric estimators for short tailed distributions, so Akkaya and Tiku (2005)  In literature, the risk factor X in binary regression has been treated as a non-stochastic variable.
However, in most situations, X is stochastic.Oral (2006)

Conclusions
In the world of econometrics, regression analysis plays an important role.Linear Regression validity of the test.The least squares principle can be applied over the regression model to estimate the parameters and for Best Linear Unbiased Estimators (BLUE) it must satisfy some basic assumptions.Maximum Likelihood Estimation can be performed when the distributions of the error terms is known to belong to a certain parametric family of probability distributions.However, when the parametric family is normal distributions with mean 0 and variance , the resulting estimate is identical to the Ordinary Least Squares Estimation.Violations of these basic assumptions lead to serious consequences of the model development.One of the important assumptions is that the regressors are fixed in repeated samples.However, in many cases particularly in social sciences the assumption of fixed regressor is not always tenable.It may be stochastic in nature.The practical strategy to follow the assumption of fixed regressor is rationale though the variable may be intrinsically stochastic.The literature survey reveals that the instance of stochastic regressor is truly a fascinating area where the regressor may follow some traditional distributions.The regressor may follow other distribution(s) besides normality.In cases of non-normality, the traditional estimation method is in general problematic.For instance, the likelihood equations have no explicit solutions and solving them by iteration can be problematic.To attenuate these difficulties, Method of Modified Maximum Likelihood (MMLE) is used, introduced by Tiku (1967Tiku ( , 1968Tiku ( , 1980) ) and Tiku and Suresh (1992), Islam al. (2001) and Tiku et al. (2001) developed MMLE for X and e nonnormal and considered the following families of distributions: Weibull, Generalized logistic, Student's t, and short-tailed distributions.There is a huge literature related to stochastic regressors in various aspects.
So, based on literature search a historical perspective on some of the works related to stochastic regressors has been critically discussed here which fulfills the objective of this paper.However, after 2010, so far our knowledge goes no significant attempt has been made by the researchers in this area.
examined four alternative tests of independence.He also examined two finite sample tests and two alternative asymptotic tests, along with power functions of them.It was shown that the entire tests were consistent.Here two linear models were considered where the disturbances are distributed as multivariate normal with mean zero and covariance matrix Σ =   ′   .In 1973, Wu only presented four alternative tests of independence between stochastic regressors and disturbances in a linear regression model and considered two distributions, the gamma (chi-square) with support (0,∞) and generalized logistic with support (−∞,∞) for illustration.They derived MML (modified maximum likelihood) estimators of the parameters in the autoregressive model yt − φyt−1= μ + δ (xt− φxt−1) + at, model is widely used in biological, behavioral and social sciences to describe possible relationships between variables.A large number of procedures have been developed for parameter estimation and inference in linear regression model with respect to distributions, theoretical assumptions for the Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 27 September 2018 doi:10.20944/preprints201809.0539.v1 This is valid for experimental work, in which the experimenter has control over the explanatory variables and can repeatedly observe the outcome of the dependent variable with the same fixed values or some designated values of the explanatory variables.In the social sciences and particularly in economics, explanatory variables in one equation are often generated as the outcome variables of other equations that are stochastic in nature.Thus they neither have the same fixed values in repeated samples nor do they have values that conform to the investigator's desired experimental design.
may not vanish.It

preprints.org) | NOT PEER-REVIEWED | Posted: 27 September 2018 doi:10.20944/preprints201809.0539.v1 at
the University of Chicago.Based on information for both these methods under Ho and H1 suggest that the data do not provide much basis for discriminating whether the consumption services of health are normal or inferior.
and then provided a comparison with use of the Wu tests.The data used for considering this example are a sample of 816 observations of individuals in a 1970 national survey of health and health care conducted by the centre for Health Administration Studies (CHAS) Preprints (www.stochasticexplanatoryvariables.Stochastic regressors present the problem of larger variance for the OV estimator than in the non-stochastic case, and derived conditions under which omission may be better under a minimum mean square error criterion.The error-in-variables problem can be interpreted as a specification error problem with stochastic regressors, and the issue of MSE dominance of the proxy-variable estimator over the OV estimator is considered.It may be optimal to introduce some measurement error, even in situations where all variables are correctly measured.Potscher (1989)gave strong consistency results for model selection procedures based on variants of Akaike's information Criteria for general nonstationary stochastic linear regression models where the errors constitute a not necessary stationary martingale difference sequence using results of Lai

(www.preprints.org) | NOT PEER-REVIEWED | Posted: 27 September 2018 doi:10.20944/preprints201809.0539.v1
Sazak et al. (2006)ch robust estimators by censoring a fixed proportion of ordered statistics in the middle of the random samples.They derived MMLE and showed that MMLE are explicit, and efficient and robust under short-tailedness and inliers.Sazak et al. (2006)considered situations when both X and e has non-normal distributions, i.e. when both the marginal and the conditional distributions are skew.Particularly generalized logistic distribution is taken for illustration for X and e with shape parameter b1(>0) and b2(>0).Then Modified Maximum Likelihood Estimation (MMLE) is used for parameter estimation which yields estimators which are asymptotically efficient.For small n, the MMLE are known to be essentially as efficient as the MLE (Maximum likelihood estimators) and the two are numerically very close to one another.
presented solutions applicable to such situations where there are only four independent parameters.Here, h(x) denotes the probability density function of X and two cases are considered.In the first case h(x) is skewed, e.g., Generalized Logistic (GL), whereas in the second case h(x) is symmetric, e.g., scaled Student t.To study efficiency and robustness properties, MMLE is derived and showed that treating X nonstochastic results in loss of efficiency if X is in fact stochastic.In particular, they showed that the traditional estimators based only on the conditional likelihood LY|X with the observations xi replaced by (xi−̅ )/s, 1 ≤ i ≤ n, give inefficient estimators unless h(x) is normal N (0, ).
Islam and Tiku (2010)mators (LSE) of the parameters in the second order polynomial regression model are obtained.Both are invariant to location and scale of the design variables and the former are considerably more efficient and, for any non-normal distribution, the relative efficiencies of LSEs are the same (almost) and less than 1 for all designs considered.They illustrated with three real life examples.Islam and Tiku (2010)tried to developed estimators of parameters when there are q ≥ 2 stochastic design variables in the multiple linear regression models, which is more general and challenging situations.It may be noted that treating Xj, (1 ≤ j ≤ q) as non-stochastic when, in fact, they are stochastic can unknowingly yield estimators which have considerable bias and are inefficient.