Curve descriptions

paper/sl.tex

@@ -200,10 +200,18 @@ Moreover, it is easy to see based on the derivations of Eq.\ref{eqn:gp} that our
 
 \section{Results}
 
-Below we present our results in various settings.
+Below we present our results in various settings. Models are evaluated in contrast to the following quantities:
+\begin{itemize}
+\item {\it True evaluation:} Depicts the true performance of the predictive model. Constructed by sorting all the labels in the test data (even the ones hidden from the models) by the predicted probabilities and the simulating the leniency at a given level.
+\item {\it Labeled outcomes:} Similar to {\it true evaluation} but only available labels with positive decisions $(\decision = 1)$ are used.
+\item {\it Human evaluation:} Human decision makers with similar leniency levels are binned and treated as a single decision maker.
+\item {\it Contraction:} Contraction curve was constructed as explained by Lakkaraju et al \cite{lakkaraju2017selective}.
+\item {\it Causal model, ep:} Curve presents the predicted probability $\prob{\outcome = 0 | \doop{\leniency = \leniencyValue}}$ at various levels of acceptance rate.
+\end{itemize}
 
 \subsection{Without unobservables}
 
+\subsubsection{Data creation}
 The causal model for this scenario corresponds to that depicted in Figure \ref{fig:causalmodel}.
 For the analysis, we assigned 500 subjects for each of the 100 judges randomly.
 Every judge's leniency rate $\leniency$ was sampled uniformly from a half-open interval $[0.1; 0.9)$. 
@@ -215,7 +223,7 @@ Next, probabilities for negative results $\outcome = 0$ were calculated as
 
 and consequently $\outcome \sim \text{Bernoulli}(1 - p_{y_0})$.
 The decision variable $\decision$ was set to 0 if the value $p_{y_0}$ resided in the top $(1-\leniencyValue)\cdot 100 \%$ of the subjects appointed for that judge.
-Results for estimating the causal quantity $\prob{\outcome = 0 | \doop{\leniency = \leniencyValue}}$ with various levels of leniency $\leniencyValue$ are presented in Figure \ref{fig:without_unobservables}.
+Results for estimating the causal quantity $\prob{\outcome = 0 | \doop{\leniency = \leniencyValue}}$ with various levels of leniency $\leniencyValue$ under this model are presented in Figure \ref{fig:without_unobservables}.
 
 \begin{figure}
 \begin{center}
@@ -225,6 +233,7 @@ Results for estimating the causal quantity $\prob{\outcome = 0 | \doop{\leniency
 \label{fig:without_unobservables}
 \end{figure}
 
+
 % \textbf{Acknowledgments.}