@@ -163,17 +163,18 @@ It is designed specifically to estimate the true failure rate of a machine decis
Contraction bases its evaluation only on the cases assigned to the most lenient decision maker $\human_l$ in the data.
%
%The algorithm removes cases with negative decision from that decision maker. % R_q
The approach assumes $\machine$ makes a negative decision for all cases for which the most lenient decision maker $\human_l$ makes a negative decision.
Because of the lower leniency of the evaluated decision maker $\machine$, the approach assumes $\machine$ makes a negative decision for all cases for which $\human_l$ makes a negative decision.
%
The cases with a positive decision by $\human_l$ are sorted according to the lowest leniency level at which they receive a positive decisions by the evaluated decision maker $\machine$. % R sort q
%
The sorted list is then {\it contracted} so that only cases in the \leniencyValue fraction
The cases with a positive decision by $\human_l$ are sorted according to the lowest leniency level at which they receive positive decisions by $\machine$. % R sort q
% I PUT HERE MATCH BECAUSE THE FRACTION NEEDS CONSIDER
The sorted list is then {\it contracted} to match the leniency level $\leniencyValue$ at which \machine is evaluated. Because all outcomes for subjects in this list are available in the data,
%so that only cases in the \leniencyValue fraction
%of the least dangerous
of cases are considered in the computation of the failure rate estimate: % R_b
The estimate is the number of negative outcomes in that contracted list (for which outcomes are observed) divided by the number of cases assigned $\human_l$. Because the cases are assigned randomly to all decision makers, this estimate gives an estimate of the failure rate on the whole dataset.
%of cases are considered in the computation of the failure rate estimate: % R_b
we can estimate the failure rate by the number of negative outcomes for cases in the contracted list divided by the number of cases assigned to $\human_l$. Because the cases are assigned randomly to all decision makers, this estimates the failure rate on the whole dataset.
%
\rcomment{Rewrote it to the best of my abilities. Ok now?}
\todo{MM}{Re-write the last two sentences.}
%\rcomment{Rewrote it to the best of my abilities. Ok now?}
