\maketitle% typeset the header of the contribution
\begin{abstract}
We develop a Bayesian approach to evaluate AI decision systems over data from past decisions.
We develop a Bayesian approach to evaluate AI decision systems using data from past decisions.
%
% Such settings arise commonly when newly developed AI systems are evaluated against previous systems they aim to replace.
%
...
...
@@ -77,15 +77,15 @@ And second, past decisions may have led to unobserved outcomes.
%
This is the case, for example, when a bank decides whether a customer should be granted a loan, and the outcome of interest is whether the customer will repay the loan.
%
In this case, the data includes the outcome only for customers who were granted the loan, but not for those who were not.
In this case, the data includes the outcome (was loan repayed or not) only for customers who were granted the loan, but not for those who were not.
%
To address these challenges, we formalize the decision making process with a causal model, taking into account also the unobserved features.
To address these challenges, we formalize the decision making process with a causal model, considering also unobserved features.
%
Based on this model, we compute counterfactuals to impute missing outcomes, which in turn allows us to produce accurate evaluations.
%
As we demonstrate over real and synthetic data, our approach estimates the quality of decisions more accurately and with lower variance compared to previous methods.
%
Our approach is also %demonstrated to be
It is also %demonstrated to be
robust to different variations in the decision mechanisms in the data.
