related.tex

%!TEX root = sl.tex
% The above command helps compiling in TexShop on a MAc. Hitting typeset complies sl.tex directly instead of producing an error here.


\section{Related work}
\label{sec:related}

%Rikus point: other papers do not really explain why decisions help in predictions

In this paper we considered the overall setting as formulated by~\citet{lakkaraju2017selective}, and building on~\cite{Jung2,mccandless2007bayesian,dearteaga2018learning}, showed that causally informed counterfactual imputation can achieve accurate results.
%In addition to Lakkaraju et al.~\citet{lakkaraju2017selective} which we build upon, several papers consider related problems to ours. 
Note that our setting allowing for unobserved confounding does not fulfill the ignorability or missing at random (MAR), biasing any methods built on these assumptions~\cite{lakkaraju2017selective,DBLP:conf/icml/DudikLL11,bang2005doubly,little2019statistical}.

In our simulations we compared in particular to contraction of~\citet{lakkaraju2017selective}, an approach that is appealing in its simplicity.  However, as our experiments confirm, it is quite sensitive to number of subjects given for a decision maker, decision makers being evaluated and recorded in the data, in terms of their leniency and other specifics.   \citet{kleinberg2018human} present an in-detail account of employing contraction in a real data.

%also note the possibility of using decision in the data to correct for selective labels, assuming expert consistency
 In contrast to our imputation approach, 
De-Arteaga et al.~\cite{dearteaga2018learning} directly impute decisions as outcomes and consider learning automatic decision makers from this augmented data. \citet{kleinberg2018human} use a multiplicative correction term to adjust the bias observed for more conventional imputation. In comparison, our  approach is based on a rigorous causal model accounting for different leniencies and unobservables, and gives accurate results even with random decision makers that violate the expert consistency assumption of \cite{dearteaga2018learning}. % and a particular type of imputation.

In reinforcement learning a related scenario is consider as offline policy evaluation, where the objective is to learn from 
data recorded under some policy, the goodness of some other policies \cite{Jung2,DBLP:conf/icml/ThomasB16}. In particular, Jung et al. \cite{Jung2,jung2018algorithmic} consider sensitivity analysis in a similar scenario as ours, but without directly modelling judges with multiple leniencies.
Mc-Candless et al. perform Bayesian sensitivity analysis while taking into account latent confounding~\cite{mccandless2007bayesian,mccandless2017comparison}.
\citet{kallus2018confounding} obtain improved policies from data possibly biased by a baseline policy.

The importance in-detail causal modeling and evaluating counterfactual outcomes, as observed also here, is particularly prominent in recent work on fairness of automatic decision making~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/NabiMS19,DBLP:conf/icml/JohanssonSS16}. Also identifiability questions in the presence of selection bias or missing data mechanisms require detailed causal modeling~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical}.
%To properly assess decision procedures for their performance and fairness we need to understand the causal relations 
Finally, more applied work on automated decision making and risk scoring, related in particular to recidivism, can be found for example in~\cite{murder,tolan2019why,kleinberg2018human,chouldechova2017fair,brennan2009evaluating}.

%\citet{} do what?

%\cite{madras2019fairness} learn fair and acccurate treatment policies from biased data.
%   \acomment{They cite Lakkaraju, De arteaga as conceptually similar work.}

%\cite{coston2020counterfactual} propose counterfactual measures for performance metrics with doubly robust estimation of these metrics. %The first assumes absense of unobserved variables.

%An issue we do not discuss in this paper is fairness  \citep{DBLP:conf/icml/Kusner0LS1,madras2019fairness,corbett2017algorithmic}.

%Literature on risks and redicivism? \cite{chouldechova2017fair}


%difference on jung and mcCandless
%they have u as binary, no it is normal there are many
%different groups maybe have diffe

%How do they actually differ.
%\acomment{We should refer to Deartega somewhere early on, they have made the same discovery as we put presented it poorly.}


%Recent research has shown the value of counterfactual reasoning in similar setting as this paper, for  fairness of decision making, and applications in online advertising~\cite{DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/Kusner0LS19,DBLP:conf/icml/NabiMS19,DBLP:conf/icml/JohanssonSS16,pearl2000}. 

%Some imputation book?


%\subsection{Imputation}

%\rcomment{Imputation approach presented by \citet{kleinberg2018human} in p.270 is a bit different than the one we have used. Our approach is more similar to Lakkaraju's original paper \cite{lakkaraju2017selective}. Their approach has three stages, 1. use observed outcomes whenever available. 2. impute results convetionally up to the leniency off the most lenient decision maker and 3. impute normally, but multiply the prediction by $\alpha$.}