@@ -11,10 +11,10 @@ We presented \cfbi, an approach based on proper causal modelling that makes full
%
Via thorough experimentation we found that \cfbi allows for accurate and robust evaluations, also in settings that evaluation was not possible before (i.e., for a leniency level higher than the one present in the data).
%
In future work, we will generalize our setting and modeling assumptions -- e.g., to consider more elaborate behaviour for decision makers and additional dependencies between model variables.
In future work, we will generalize our setting and modelling assumptions -- e.g., to consider more elaborate behaviour for decision makers and additional dependencies between model variables.
% In future work, we will generalize our setting and modeling assumptions.
% One direction is to consider a more elaborate structure behind the behaviour of the decision makers, e.g., via hierarchical Bayesian modeling.
% In future work, we will generalize our setting and modelling assumptions.
% One direction is to consider a more elaborate structure behind the behaviour of the decision makers, e.g., via hierarchical Bayesian modelling.
% Another direction is to consider additional dependencies between model variables, e.g., dependence between the decision makers and the features of the cases assigned to them.
%
% More generally, this line of research opens promising research directions that could enable better evaluation in new application fields, ensuring the accuracy and fairness of automatic decision making procedures that can be then adopted in the society.
In \cite{kleinberg2018human}, a multiplicative correction term is used to adjust the bias observed for more conventional imputation.
%
In comparison, \cfbi uses rigorous causal modeling to account for leniency and unobservables, and gives accurate results even with random decision makers that violate the expert consistency assumption of \cite{dearteaga2018learning}. % and a particular type of imputation.
In comparison, \cfbi uses rigorous causal modelling to account for leniency and unobservables, and gives accurate results even with random decision makers that violate the expert consistency assumption of \cite{dearteaga2018learning}. % and a particular type of imputation.
%% MM: I take out the following, they are less related and we are out of space
\iffalse
...
...
@@ -58,11 +58,11 @@ Mc-Candless et al. perform Bayesian sensitivity analysis while taking into accou
\fi
\iffalse
The effectiveness of causal modeling and use of counterfactuals is also demonstrated in recent work on algorithmic fairness~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/JohanssonSS16}.
The effectiveness of causal modelling and use of counterfactuals is also demonstrated in recent work on algorithmic fairness~\cite{DBLP:conf/icml/NabiMS19,DBLP:conf/icml/Kusner0LS19,coston2020counterfactual,madras2019fairness,corbett2017algorithmic,DBLP:journals/jmlr/BottouPCCCPRSS13,DBLP:conf/icml/JohanssonSS16}.
%
Several works study selection bias or missing data in the context of identifiability of causal effects and causal structure~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical,Bareinboim2014:selectionbias,smr1999,Mohan2013,Shpitser2015}.
%
%Also identifiability questions in the presence of selection bias or missing data mechanisms require detailed causal modeling~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical}.
%Also identifiability questions in the presence of selection bias or missing data mechanisms require detailed causal modelling~\cite{bareinboim2012controlling,hernan2004structural,little2019statistical}.
%To properly assess decision procedures for their performance and fairness we need to understand the causal relations
Finally, more applied work on automated decision making and risk scoring, related in particular to recidivism, can be found for example in~\cite{murder,tolan2019why,kleinberg2018human,chouldechova2017fair,brennan2009evaluating,royal}.