Newer
Older
%!TEX root = sl.tex
% The above command helps compiling in TexShop on a MAc. Hitting typeset complies sl.tex directly instead of producing an error here.
\section{Related work}
\label{sec:related}
In addition to Lakkaraju et al.~\citet{lakkaraju2017selective} which we build upon, several papers consider related problems to ours. Note that our setting allowing for unobserved confounding does not fulfill the ignorability or missing at random (MAR), preventing the use of methods assuming them~\cite{lakkaraju2017selective,DBLP:conf/icml/DudikLL11,bang2005doubly,little2019statistical}.
In reinforcement learning a similar scenario is consider as offline policy evaluation, where the objective is to learn from
data recorded under some policy, the goodness of some other policies. In particular, Jung et al. \cite{Jung2,jung2018algorithmic} consider sensitivity analysis in a similar scenario as ours, but without not directly modelling judges with multiple leniencies.
Also Mc-Candless et al. perform Bayesian sensitivity analysis on the priors for a similar model as here employing logistic regression and taking into account latent confounding~\cite{mccandless2007bayesian}.
De-Arteaga et al. also note the possibility of using decision in the data to correct for selective labels, assuming expert consistency~\cite{dearteaga2018learning}. They directly impute decisions as outcomes and consider learning automatic decision makers from this augmented data. In contrast, our approach on decision maker evaluation is based on a rigorous probabilistic model accounting for different leniencies and unobservables. Furthermore, our approach gives accurate results even with random decision makers that clearly violate the expert consistency assumption. \acomment{De-Arteaga uses selective labels terminology}
\acomment{Cite Corbet-Davies somewhere}
To properly assess decision procedures for their performance and fairness we need to understand the causal relations \cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19}
More applied work includes works such as~\cite{murder,tolan2019why}.
\cite{madras2019fairness} do what
\cite{coston2020counterfactual} propose counterfactual measures for performance and fairness metrics with doubly robust estimation of these metrics. The first assumes absense of unobserved variables.
%difference on jung and mcCandless
%they have u as binary, no it is normal there are many
%different groups maybe have diffe
%\acomment{We should refer to Deartega somewhere early on, they have made the same discovery as we put presented it poorly.}
Recent research has shown the value of counterfactual reasoning in similar setting as this paper, for fairness of decision making, and applications in online advertising~\cite{DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/Kusner0LS19,DBLP:conf/icml/NabiMS19,DBLP:conf/icml/JohanssonSS16,pearl2000}.