In this paper we considered the overall setting as formulated by~\citet{lakkaraju2017selective}, and building on~\cite{Jung2,mccandless2007bayesian,dearteaga2018learning}, showed that causally informed counterfactual imputation can achieve accurate results.
%In addition to Lakkaraju et al.~\citet{lakkaraju2017selective} which we build upon, several papers consider related problems to ours.
Note that our setting allowing for unobserved confounding does not fulfill the ignorability or missing at random (MAR), biasing any methods built on these assumptions~\cite{lakkaraju2017selective,DBLP:conf/icml/DudikLL11,bang2005doubly,little2019statistical}.
\todo{MM}{Clarify above sentence?}
Note that our setting allowing for unobserved confounding does not fulfill the ignorability or missing at random (MAR) conditions often assumed when processing missing data, biasing any methods built on these assumptions~\cite{lakkaraju2017selective,DBLP:conf/icml/DudikLL11,bang2005doubly,little2019statistical}.
In our simulations we compared in particular to \contraction of~\citet{lakkaraju2017selective}, an approach that is appealing in its simplicity.
%
...
...
@@ -24,8 +23,7 @@ In contrast to our imputation approach, De-Arteaga et al.~\cite{dearteaga2018lea
%
In comparison, our approach is based on a rigorous causal model accounting for different leniencies and unobservables, and gives accurate results even with random decision makers that violate the expert consistency assumption of \cite{dearteaga2018learning}. % and a particular type of imputation.
In reinforcement learning a related scenario is consider as offline policy evaluation, where the objective is to learn from data recorded under some policy, the goodness of some other policies \cite{Jung2,DBLP:conf/icml/ThomasB16}.
\todo{MM}{I do not understand sentence above.}
In reinforcement learning a related scenario is consider as offline policy evaluation, where the objective is to determine a quality of a policy from data recorded under some other baseline policy \cite{Jung2,DBLP:conf/icml/ThomasB16}.
%
In particular, Jung et al. \cite{Jung2,jung2018algorithmic} consider sensitivity analysis in a similar scenario as ours, but without directly modelling judges with multiple leniencies.
%
...
...
@@ -33,7 +31,7 @@ Mc-Candless et al. perform Bayesian sensitivity analysis while taking into accou
%
\citet{kallus2018confounding} obtain improved policies from data possibly biased by a baseline policy.
The importance in-detail causal modeling and evaluating counterfactual outcomes, as observed also here, is particularly prominent in recent work on fairness of automatic decision making~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/NabiMS19,DBLP:conf/icml/JohanssonSS16}. Several author study selection bias or missing data in the context of identifiability of causal effects and causal structure~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical,Bareinboim2014:selectionbias,smr1999,Mohan2013,Shpitser2015}.
The importance in-detail causal modeling and evaluating counterfactual outcomes, as observed also here, is particularly prominent in recent work on fairness of automatic decision making~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/JohanssonSS16}. Several author study selection bias or missing data in the context of identifiability of causal effects and causal structure~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical,Bareinboim2014:selectionbias,smr1999,Mohan2013,Shpitser2015}.
%To properly assess decision procedures for their performance and fairness we need to understand the causal relations
Finally, more applied work on automated decision making and risk scoring, related in particular to recidivism, can be found for example in~\cite{murder,tolan2019why,kleinberg2018human,chouldechova2017fair,brennan2009evaluating}.
