@@ -257,7 +257,7 @@ Given the above framework, the goal is to create an evaluation algorithm that ca
\end{itemize}
\item[Evaluator:] ~ \\
~ \\
\hskip 3em \textbf{Input:}
\hskip 3em \textbf{Input:}
\begin{itemize}
\item Data sample $(X, T, Y)$
\item something about $\M$ and something about Decider(r)
...
...
@@ -274,15 +274,19 @@ The above framework is now separated into three different modules: data generati
The next module, namely the decider, assigns decisions for each observation with a given/defined way. This 'decision' can be either the most likely value for y (argmax likelihood y, usually binary 0 or 1), probability of an outcome or an ordering of the defendants.
The evaluator module takes a data sample, some information about the data generation and some information about the decider as input. The data sample includes features $X,~T$ and $Y$ where $Y \in\{0, 1, NA\}$ as specified before. The "some information about $\M$" might be knowledge on the distribution of some of the variables or their interdependencies. In our example, we know that the $X$ is a standard Gaussian and independent from the other variables. From the decider it is known that its decisions are affected by leniency and private properties X. Next we try to simulate the decision-maker's process within the data sample. But to do this we need to learn the predictive model $\B$ with the restriction that Z can't be observed. With this setting, we now need to define an algorithm which outputs $\mathbb{E}[FR~|~input]$.
The evaluator module takes a data sample, some information about the data generation and some information about the decider as input. The data sample includes features $X,~T$ and $Y$ where $Y \in\{0, 1, NA\}$ as specified before. The "some information about $\M$" might be knowledge on the distribution of some of the variables or their interdependencies. In our example, we know that the $X$ is a standard Gaussian and independent from the other variables. From the decider it is known that its decisions are affected by leniency and private properties X. Our objective is to simulate the decision-maker's process within the data sample. But to do this we need to learn the predictive model $\B$ with the restriction that Z can't be observed. With this setting, we now need to define an algorithm which outputs $\mathbb{E}[FR~|~input]$ for a given model $\B$ and leniency $r$.
\begin{quote}
\emph{MM:} For example, consider an evaluation process that knows (i.e., is given as input) the decision process and what decisions it took for a few data points. The same evaluation process knows only some of the attributes of those data points -- and therefore it has only partial information about the data generation process. To make the example more specific, consider the case of decision process $\s$ mentioned above, which does not know W -- and consider an evaluation process that knows exactly how $\s$ works and what decisions it took for a few data points, but does not know either W or Z of those data points. This evaluation process outputs the expected value of FR according to the information that's given to it.
\end{quote}
\textcolor{red}{For what does the evaluator provide the estimate of failure rate for?}
\textbf{Example} For illustration, let's consider the contraction algorithm in this framework. Data is generated from three Gaussians and the outcome Y is decided as in algorithm \ref{alg:data_with_Z}. Now the deciders are humans assigning the decisions with the ordering tactic as presented in algorithm \ref{alg:data_with_Z}. The evaluator module now takes the whole of the data as input, knows that X affects the results and that the variables are Gaussians. Now the evaluator module ....
\textbf{Example} For illustration, let's consider the contraction algorithm in this framework. Data is generated from three Gaussians and the outcome Y is decided as in algorithm \ref{alg:data_with_Z}. Now the deciders are humans assigning the decisions as in algorithm \ref{alg:data_with_Z}. Next, the evaluator module
\begin{itemize}
\item takes the whole of the observable data ($\{x_i, t_i, y_i\}$) as input
\item knows that X affects the results and the decisions
\item knows that the variables are Gaussians.
\end{itemize}
Now the evaluator module splits the data to training and test sets and trains a predictive model $\B$ on the training set. Finally the contraction algorithm outputs an estimate of the failure rate following its definition (see algorithm \ref{alg:contraction}) using predictions from $\B$.
