@@ -97,7 +97,7 @@ The data which such evaluation is performed on has two major challenges, biasing
%
First, in most cases the data does not include all factors that play a role in the decisions recorded in it.
%
Second, the past decision in the data skew the data, and, in particular, any possible outcomes recorded in it.
Second, the past decision in the data may skew the data, and, in particular, any possible outcomes recorded in it.
%Another major challenge in such cases is that often past decisions have skewed the data on which the evaluation is performed.
%
For example, when a bank decides whether a customer should be granted a loan, it is desired to grant loans to customers who would honor its conditions, but not to ones who would violate them.
...
...
@@ -106,14 +106,17 @@ However, we can directly evaluate only the decision to grant the loan, while we
%
%THIS IS NOT SKEW THIS IS MISSING DATA
%
Such selection bias appears in the decisions of both human and AI decision makers -- and should be properly taken into account for evaluation.
Such bias appears in the decisions of both human and AI decision makers -- and should be properly taken into account for evaluation.
%
%Further complications arise since commonly not all features that the decisions are based on are observed. DISCUSS UNOBSERVABLES IN THE INTRO?
%
In this paper, we develop a Bayesian approach towards this end, using what we call counterfactual-based imputation.
% to infer unobserved outcomes.
In this paper, we develop a Bayesian approach towards this end.
%
Compared to previous state-of-the-art, the quality of decisions is estimated more accurately and with lower variance.
We use a detailed causal model of the decision making process, taking into account also unobserved features.
Based on this model, we evaluate counterfactual outcomes to correct any aforementioned biases.
% to infer unobserved outcomes.
%
Compared to previous methods for this setting, the approach estimates the quality of decisions more accurately and with lower variance.
%
The approach is also %demonstrated to be
robust to different variations in the decision mechanisms in the data.
