%!TEX root = sl.tex
% The above command helps compiling in TexShop on a MAc. Hitting typeset complies sl.tex directly instead of producing an error here.

\section{Conclusions and Future work}

In this paper we considered the task of evaluating (automated) decision makers, which is crucially needed in replacing human decisions with automated ones.
%
The challenge is that the evaluation often needs to be based on data where present decisions imply selective labeling and missing data, biasing any standard statistical data analysis results.  
%
We presented \cfbi, an approach based on proper causal modelling that makes full use of the available data, and demonstrated that automated decision makers can be evaluated on data that are selectively labeled.
%
Contrary to previous work, our proposed approach allows for more accurate evaluations, with less variation, also in settings that evaluation was not possible before (i.e., for a leniency level higher than the one present in the data).


In future work, we will generalize our setting and modeling assumptions.
One direction is to consider a more elaborate structure behind the behaviour of the decision makers, e.g., via hierarchical Bayesian modeling.
Another direction is to consider additional dependencies between model variables, e.g., dependence between the decision makers and the features of the cases assigned to them.
%, which are currently considered independent.
%WE DO NOT WANT TO ADVERTISE AT THIS POINT A MAJOR LIMITATION OF OUR MODEL, BETTER TO SAY IT VAGUELY
%
% Since our approach predicts outcomes based on decisions made by educated decision makers, it is still unclear how much this benefits the estimation the statistical models the automated decision makers are ultimately based on~\cite{dearteaga2018learning}. % MM: not sure what this means.
%
More generally, this line of research opens promising research directions that could enable better evaluation in new application fields, ensuring the accuracy and fairness of automatic decision making procedures that can be then adopted in the society.