1st chapter of results

.gitignore

@@ -27,7 +27,7 @@ sources_ignored
 # these rules might exclude image files for figures etc.
 # *.ps
 # *.eps
-# *.pdf
+*.pdf
 
 ## Generated if empty string is given at "Please type another file name for output:"
 .pdf

paper/sl.tex

@@ -124,6 +124,15 @@ which should be equal to
 \begin{equation}
 	F(\featuresValue_0) = \int {\prob{\featuresValue}  \indicator{\prob{\outcome = 0| \decision = 1, \features = \featuresValue} > \prob{\outcome = 0| \decision = 1, \features = \featuresValue_0}} d\featuresValue}
 \end{equation}
+
+\note[RL]{
+	Should the inequality be reversed? With some derivations
+	\begin{equation}
+	F(\featuresValue_0) = \int {\prob{\featuresValue}  \indicator{\score{\featuresValue} > \score{\featuresValue_0} } ~ d\featuresValue}
+\end{equation}
+}
+
+
 The bail-or-jail scenario is just one example of settings that involve a decision $\decision \in\{0,1\}$ that is based on individual features \features and leniency (acceptance rate) \leniency -- and where a behavior of interest \outcome is observed only for the cases where \decision = 1.
 The diagram of the causal model is shown in Figure~\ref{fig:causalmodel}.
 Our results are applicable to other scenarios with same causal model.
@@ -142,6 +151,7 @@ We will use existing machine-learning techniques from the literature to learn fu
 The challenge we face is to estimate accurately the performance of the decision system -- given that we are in a `selective labels' setting.
 Performance is measured {\it for a given leniency level} as the rate at which bail is granted {\it and} the defendant violates it.
 In other words, performance is measured as the probability that a decision lead to undesired outcome.
+
 \section{Analysis}
 
 We wish to calculate the probability of undesired outcome (\outcome = 0) at a fixed leniency level.
@@ -170,17 +180,9 @@ the {\it generalized performance} \generalPerformance of that model is given by
 Equation~\ref{eqn:gp} can be calculated for a given model \datadistr{\featuresValue} = \prob{\features = \featuresValue} of individual features.
 Alternatively, we can have an empirical measure \empiricalPerformance of performance over the $\datasize$ data points in dataset \dataset, given by the following equation.
 \begin{equation}
-\empiricalPerformance = \frac{1}{\datasize} \sum_{(\featuresValue, \outcomeValue)\in\dataset}  \indicator{\outcomeValue = 0} \indicator{F(\featuresValue) < r} 
-\label{eqn:gp}	
-\end{equation}
-
-\note[MM]{
-	Use the following for empirical performance?
-	\begin{equation}
 \empiricalPerformance = \frac{1}{\datasize} \sum_{(\featuresValue, \outcomeValue)\in\dataset}  \score{\featuresValue} \indicator{F(\featuresValue) < r} 
 \label{eqn:gp}	
 \end{equation}
-}
 
 \subsection{Comments}
 Roughly speaking, the above formulas should work well if `bail' cases (\decision = 1) cover well the area spanned by the observed features of defendants -- i.e., we do not have large areas of \features with no or too few bail cases.
@@ -196,6 +198,33 @@ Lack of data for large areas of \features is a potential problem for the {\it co
 Unlike contraction, though, our approach does not require to have data at all leniency levels.
 Moreover, it is easy to see based on the derivations of Eq.\ref{eqn:gp} that our approach would work identically in the case where defendants are not assigned to judges at random (i.e., if there was a causal relation $\features\rightarrow\leniency$).
 
+\section{Results}
+
+Below we present our results in various settings.
+
+\subsection{Without unobservables}
+
+The causal model for this scenario corresponds to that depicted in Figure \ref{fig:causalmodel}.
+For the analysis, we assigned 500 subjects for each of the 100 judges randomly.
+Every judge's leniency rate $\leniency$ was sampled uniformly from a half-open interval $[0.1; 0.9)$. 
+Private features $\features$ were defined as i.i.d standard Gaussian random variables.
+Next, probabilities for negative results $\outcome = 0$ were calculated as 
+\[
+\prob{\outcome = 0| \features = \featuresValue} = \dfrac{1}{1+\exp\{-\featuresValue\}}
+\]
+
+and consequently $\outcome \sim \text{Bernoulli}(1 - \prob{\outcome = 0| \features = \featuresValue})$.
+The decision variable $\decision$ was set to 0 if the value $\prob{\outcome = 0| \features = \featuresValue}$ resided in the top $(1-\leniencyValue)\cdot 100 \%$ of the subjects appointed for that judge.
+Results for estimating the causal quantity $\prob{\outcome = 0 | \doop{\leniency = \leniencyValue}}$ with various levels of leniency $\leniencyValue$ are presented in Figure \ref{fig:without_unobservables}.
+
+\begin{figure}
+\begin{center}
+\includegraphics[width=\columnwidth]{img/without_unobservables.png}
+\end{center}
+\caption{$\prob{\outcome = 0 | \doop{\leniency = \leniencyValue}}$ with varying levels of acceptance rate. Error bars denote standard error of the mean across simulations.}
+\label{fig:without_unobservables}
+\end{figure}
+
 % \textbf{Acknowledgments.}