@@ -68,6 +68,7 @@ One major challenge in such evaluation is that the inner workings of the system
...
@@ -68,6 +68,7 @@ One major challenge in such evaluation is that the inner workings of the system
%
%
Another challenge is that often decisions skew the data on which the evaluation is performed.
Another challenge is that often decisions skew the data on which the evaluation is performed.
%
%
\mcomment{On one hand, since we use judicial data in our experiments, it makes sense to use the bail-or-jail dat in the abstract. On the other hand, this does not connect with the motivation we provide to evaluate the decision of (computer/ML/AI) systems, since jail-or-bail decisions are not currently made by such systems. The bank loan example might look better in the abstract.}
% For example, when deciding whether a defendant should be granted bail or rather be led to jail, a decision is deemed successful if it grants bail to defendants who would honor the conditions of the bail and leads to jail ones who would violate them.
% For example, when deciding whether a defendant should be granted bail or rather be led to jail, a decision is deemed successful if it grants bail to defendants who would honor the conditions of the bail and leads to jail ones who would violate them.
%
%
% However, in such cases, we are only able to directly evaluate the mechanism when it grants bail, while we cannot observe the potential bail violations by defendants who were led to jail.
% However, in such cases, we are only able to directly evaluate the mechanism when it grants bail, while we cannot observe the potential bail violations by defendants who were led to jail.
