@@ -69,7 +69,7 @@ Settings where some variables are not observed for all samples have \emph{missin
%WE WANT TO CITE HERE ALL SELECTIVE LABELS PAPERS TO SELL THIS VIEWPOINT
Recently, \citet[KDD2017]{lakkaraju2017selective} referred to the problem of evaluation is such settings as the '{\it selective labels problem}' empahasizing the fact that outcomes in the data are selectively labeled based on the decisions
Recently, \citet[KDD'17]{lakkaraju2017selective} referred to the problem of evaluation is such settings as the '{\it selective labels problem}' empahasizing the fact that outcomes in the data are selectively labeled based on the decisions
(also considered by~\cite{dearteaga2018learning,kleinberg2018human}).
%
\citet{lakkaraju2017selective} also presented {\it contraction}, a method for evaluating decision making mechanisms in a setting where subjects are randomly assigned to decision makers with varying leniency levels.
...
...
@@ -80,7 +80,7 @@ We note, however, that for contraction to work, we need lenient decision makers
% THIS BELONGS TO RELATED WORK
In another recent paper, \citet{Jung2} studied unobserved confounding in the context of creating optimal decision policies.
%
They approached the problem with Bayesian modelling, but they don't consider selective labeling or the possibility that the decisions reflected in the data may be taken by several decision makers with differing levels of leniency.
They approached the problem with Bayesian modelling and sensitivity analysis, but they don't consider selective labeling or the possibility that the decisions reflected in the data may be taken by several decision makers with differing levels of leniency.
\spara{Our contributions}
In this paper, we build upon the problem setting used in~\citet{lakkaraju2017selective} and present a novel, modular framework to evaluate decision makers over selectively labeled data.
...
...
@@ -95,5 +95,7 @@ The results indicate that our method achieves more accurate results with conside
@@ -22,7 +22,7 @@ data recorded under some policy, the goodness of some other policies \cite{Jung2
Mc-Candless et al. perform Bayesian sensitivity analysis while taking into account latent confounding~\cite{mccandless2007bayesian,mccandless2017comparison}.
\citet{kallus2018confounding} obtain improved policies from data possibly biased by a baseline policy.
The importance in-detail causal modeling and evaluating counterfactual outcomes, as observed also here, is particularly prominent in recent work on fairness of automatic decision making~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/NabiMS19,DBLP:conf/icml/JohanssonSS16}. Also identifiability questions in the presence of selection bias or missing data mechanisms require detailed causal modeling~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical}.
The importance in-detail causal modeling and evaluating counterfactual outcomes, as observed also here, is particularly prominent in recent work on fairness of automatic decision making~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/NabiMS19,DBLP:conf/icml/JohanssonSS16}. Several author study selection bias or missing data in the context of identifiability of causal effects and causal structure~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical,Bareinboim2014:selectionbias,smr1999,Mohan2013,Shpitser2015}.
%To properly assess decision procedures for their performance and fairness we need to understand the causal relations
Finally, more applied work on automated decision making and risk scoring, related in particular to recidivism, can be found for example in~\cite{murder,tolan2019why,kleinberg2018human,chouldechova2017fair,brennan2009evaluating}.