@@ -57,17 +57,19 @@ There is a rich literature on problems that arise in similar settings, our speci
% COUNTERFACTUALS: FUNDAMENTAL PROBLEM
At its core, our task is to answer a `what-if' question, asking ``what would the outcome have been if a different decision had been made'' (a counterfactual), this is often mentioned as the `fundamental problem' in causal inference~\cite{holland1986statistics, bookofwhy}.
% SELECTION BIAS
Settings where a portion of the data is not observed due to some intricate filtering mechanism are said to exhibit {\it selection bias} (see, for example, \citet{hernan2004structural}).
Settings where data samples are chosen through some intricate filtering mechanism are said to exhibit {\it selection bias} (see, for example, \citet{hernan2004structural}).
%WHAT WE DO NOT HAVE THIS???
% MISSING DATA %IMPUTATION
Research on selection bias has achieved results in recovery the structure of the generative model (i.e., the mechanism that results in bias) and estimating causal effects (e.g.,~\citet{pearl1995empirical} and~\citet{bareinboim2012controlling}).
Settings where some variables are not observed for all samples have missing data.
%Research on selection bias has achieved results in recovery the structure of the generative model (i.e., the mechanism that results in bias) and estimating causal effects (e.g.,~\citet{pearl1995empirical} and~\citet{bareinboim2012controlling}).
%OFFLINE POLICY EVALUATION
Similar scenarios appear in the literature for example under offline policy assessment~\cite{Jung2}.
Offline policy assessment refers to evaluation of a decision policy over a dataset recorded under another policy~\cite{Jung2}.
%COUNFOUNDING AND SENSITIVITY ANALYSIS
%WE WANT TO CITE HERE ALL SELECTIVE LABELS PAPERS TO SELL THIS VIEWPOINT
Recently, \citet[KDD2017]{lakkaraju2017selective} referred to the problem of evaluation is such settings as the '{\it selective labels problem}' empahasizing the fact that outcomes in the data are selectively labeled
(see also \cite{dearteaga2018learning,kleinberg2018human}).
(see also \cite{dearteaga2018learning,kleinberg2018human}).
%
\citet{lakkaraju2017selective} also presented {\it contraction}, a method for evaluating decision making mechanisms in a setting where subjects are randomly assigned to decision makers with varying leniency levels.
