From fdde45eebdd879f25ac917a172639c491c59e980 Mon Sep 17 00:00:00 2001
From: Antti Hyttinen <ajhyttin@gmail.com>
Date: Wed, 7 Aug 2019 11:00:48 +0300
Subject: [PATCH] Trying to make sense of the setting.
---
paper/sl.tex | 15 ++++++---------
1 file changed, 6 insertions(+), 9 deletions(-)
diff --git a/paper/sl.tex b/paper/sl.tex
index 561d710..99e471d 100755
--- a/paper/sl.tex
+++ b/paper/sl.tex
@@ -357,10 +357,7 @@ In this section we present our results from experiments with synthetic and reali
We experimented with synthetic data sets to examine accurateness, unbiasedness and robustness to violations of the assumptions.
-We sampled $N=50k$ samples of $X$, $Z$, and $W$ as independent standard Gaussians. We then drew the outcome $Y$ from a Bernoulli distribution with parameter $p = 1 - \invlogit(\beta_xx+\beta_zz+\beta_ww)$ so that $P(Y=0|X, Z, W) = \invlogit(\beta_xx+\beta_zz+\beta_ww)$ where the coefficients for X, Z and W were set to $1$, $1$, $0.2$ respectively. This process follows the suggestion of Lakkaraju et al. \cite{lakkaraju2017selective}.
-
-
-\acomment{How were the leniencies drawn?} \acomment{Explain the how we have several different judges?}
+We sampled $N=50k$ samples of $X$, $Z$, and $W$ as independent standard Gaussians. We then drew the outcome $Y$ from a Bernoulli distribution with parameter $p = 1 - \invlogit(\beta_xx+\beta_zz+\beta_ww)$ so that $P(Y=0|X, Z, W) = \invlogit(\beta_xx+\beta_zz+\beta_ww)$ where the coefficients for X, Z and W were set to $1$, $1$, $0.2$ respectively. We sampled $50$ leniency levels $R$ uniformly from $[0,1]$. We assigned the randomly subjects such that a single leniency level was assigned for $1000$ subjects. In the example, this mimics having 50 judges deciding each for $1000$ defendants. The data was divided in half to form a training set and a test set. This process follows the suggestion of Lakkaraju et al. \cite{lakkaraju2017selective}. \acomment{Check before?}
%This is one data generation module.
% It can be / was modified by changing the outcome producing mechanism. For other experiments we changed the outcome generating mechanism so that the outcome was assigned value 1 if
@@ -369,13 +366,13 @@ The \emph{default} decision maker in the data fits a logistic regression model $
$T=1$ to $R$ percent of subjects given by the leniency with highest probability of $Y=1$ in the test set.
- We used a number of different decision mechanism. A \emph{limited} works as the default but uses regression model $Y \sim \invlogit(\beta_xx)$ .
-A \emph{biased} decision maker works similarly but the logistic regression model is .. where biases decision...
-Given leniency $R$, a \emph{random} decision maker decides on $T=1$ probability $R$.
+ We used a number of different decision mechanism. A \emph{limited} works as the default but uses regression model $Y \sim \invlogit(\beta_xx)$. Hence it is unable to observe $Z$.
+A \emph{biased} decision maker works similarly as limited but the logistic regression model is .. where biases decision.
+Given leniency $R$, a \emph{random} decision maker decides on $T=1$ probability given by $R$.
In contrast, Lakkaraju et al. essentially order the subjects and decide $T=1$ with the percentage given by the leniency $R$. We see this as unrealistic: the decisions
-on a subject should not depend on the decision on other subject. In the example this would induce unethical behaviour, a judge would need to jail somebody today in order to release a defendant tomorrow.
-We treat the observations as independent and the still the leniency would be a good estimate of the acceptance rate. The acceptance rate converges to the leniency.
+on a subject should not depend on the decision on other subject. In the example this would induce unethical behaviour: a single judge would need to jail defendant today in order to release a defendant tomorrow.
+We treat the observations as independent and the still the leniency would be a good estimate of the acceptance rate. The acceptance rate converges to the leniency. \acomment{As a reviewer I would perhaps ask to see the results for the Lakkaraju mechanism.}
%This is a decider module. We experimented with different combinations of decider and data generating modules to show X / see Y. (to see that our method is robust against non-informative, biased and bad decisions . Due to space constraints we defer these results...)
--
GitLab