@@ -24,12 +24,12 @@ Let's describe how we do the following.
\item Decision makers \human can be one of two types (one type used in each experiment) :
\begin{itemize}
\item Random.
\item Using same parameters as model for \outcome, i.e., $\gamma_x =\beta_x$ and $\gamma_z =\beta_z$.
\item Using same parameters as model for \outcome, i.e., $\gamma_x =\beta_\obsFeaturesValue$ and $\gamma_\unobservableValue=\beta_\unobservableValue$.
\end{itemize}
\item Decision makers \machine(\leniencyValue):
\begin{itemize}
\item Random.
\item Decision maker with $\gamma_x =\beta_x$ and $\gamma_z=0$.
\item Decision maker with $\gamma_x =\beta_\obsFeaturesValue$ and $\gamma_\unobservableValue=0$.
\item Decision makers learned (naively) from a separate set of labeled data (what we've called the `training set').
\item Decision maker learned from Stan.
\end{itemize}
...
...
@@ -48,7 +48,7 @@ We also make a plot to show how our Counterfactuals method infers correctly valu
\subsubsection*{The effect of unobservables}
Perform the same experiment but with $\beta_z \gg\beta_x$.
Perform the same experiment but with $\beta_\unobservableValue\gg\beta_\obsFeaturesValue$.
\subsubsection*{The effect of observed leniency}
...
...
@@ -64,13 +64,13 @@ We experimented with synthetic data sets to examine accurateness, unbiasedness a
We sampled $N=7k$ samples of $X$, $Z$, and $W$ as independent standard Gaussians.
\todo{Michael to Riku}{W does not appear in our model.}
We then drew the outcome $Y$ from a Bernoulli distribution with parameter $p =1-\invlogit(\beta_xx+\beta_zz+\beta_ww)$ so that $P(Y=0|X, Z, W)=\invlogit(\beta_xx+\beta_zz+\beta_ww)$ where the coefficients for X, Z and W were set to $1$, $1$ and $0.2$ respectively. Then the leniency levels $R$ for each of the $M=14$ judges were assigned pairwise so that each of the paiirs had leniencies $0.1,~0.2,\ldots, 0.7$.
We then drew the outcome $Y$ from a Bernoulli distribution with parameter $p =1-\invlogit(\beta_\obsFeaturesValue\obsFeaturesValue+\beta_\unobservableValue\unobservableValue+\beta_ww)$ so that $P(Y=0|X, Z, W)=\invlogit(\beta_\obsFeaturesValue\obsFeaturesValue+\beta_\unobservableValue\unobservableValue+\beta_ww)$ where the coefficients for X, Z and W were set to $1$, $1$ and $0.2$ respectively. Then the leniency levels $R$ for each of the $M=14$ judges were assigned pairwise so that each of the paiirs had leniencies $0.1,~0.2,\ldots, 0.7$.
\todo{Michael to Riku}{We have assumed all along that the outcome \outcome causally follows from leniency \leniency. So we cannot suddenly say that we assign leniency after we have generated the outcomes. Let's follow strictly our model definition. If what you describe above is equivalent to what we have in the model section, then let's simply say that we do things as in the model section. Otherwise, let's do a light re-arrangement of experiments so that we follow the model exactly. In any case, in this section we should not have any model description -- we should only say what model parameters we used for the previous defined model.}
The subjects were assigned randomly to the judges so each received $500$ subjects. The data was divided in half to form a training set and a test set. This process follows the suggestion of Lakkaraju et al. \cite{lakkaraju2017selective}. \acomment{Check before?}
The \emph{default} decision maker in the data predicts a subjects' probability for recidivism to be $P(\decision=0~|~\obsFeatures, \unobservable)=\invlogit(\beta_xx+\beta_zz)$. Each of the decision-makers is assigned a leniency value, so the decision is then assigned by comparing the value of $P(\decision=0~|~\obsFeatures, \unobservable)$ to the value of the inverse cumulative density function $F^{-1}_{P(\decision=0~|~\obsFeatures, \unobservable)}(r)=F^{-1}(r)$. Now, if $F^{-1}(r) < P(\decision=0~|~\obsFeatures, \unobservable)$ the subject is given a negative decision $\decision=0$ and a positive otherwise. \rcomment{Needs double checking.} This ensures that the decisions are independent and that the ratio of positive decisions to negative decisions converges to $r$. Then the outcomes for which the decision was negative, were set to $0$.
The \emph{default} decision maker in the data predicts a subjects' probability for recidivism to be $P(\decision=0~|~\obsFeatures, \unobservable)=\invlogit(\beta_\obsFeaturesValue\obsFeaturesValue+\beta_\unobservableValue\unobservableValue)$. Each of the decision-makers is assigned a leniency value, so the decision is then assigned by comparing the value of $P(\decision=0~|~\obsFeatures, \unobservable)$ to the value of the inverse cumulative density function $F^{-1}_{P(\decision=0~|~\obsFeatures, \unobservable)}(r)=F^{-1}(r)$. Now, if $F^{-1}(r) < P(\decision=0~|~\obsFeatures, \unobservable)$ the subject is given a negative decision $\decision=0$ and a positive otherwise. \rcomment{Needs double checking.} This ensures that the decisions are independent and that the ratio of positive decisions to negative decisions converges to $r$. Then the outcomes for which the decision was negative, were set to $0$.
We used a number of different decision mechanisms. A \emph{limited} decision-maker works as the default, but predicts the risk for a negative outcome using only the recorded features \obsFeatures so that $P(\decision=0~|~\obsFeatures, \unobservable)=\invlogit(\beta_xx)$. Hence it is unable to observe $Z$. A \emph{biased} decision maker works similarly as the default decision-maker but the values for the observed features \obsFeatures observed by the decision-maker are altered. We modified the values so that if the value for \obsFeaturesValue was greater than $1$ it was multiplied by $0.75$ to induce more positive decisions. Similarly, if the subject's \obsFeaturesValue was in the interval $(-2,~-1)$ we added $0.5$ to induce more negative decisions. Additionally the effect of non-informative decisions were investigated by deploying a \emph{random} decision-maker. Given leniency $R$, a random decision-maker give a positive decision $T=1$ with probability given by $R$.
We used a number of different decision mechanisms. A \emph{limited} decision-maker works as the default, but predicts the risk for a negative outcome using only the recorded features \obsFeatures so that $P(\decision=0~|~\obsFeatures, \unobservable)=\invlogit(\beta_\obsFeaturesValue\obsFeaturesValue)$. Hence it is unable to observe $Z$. A \emph{biased} decision maker works similarly as the default decision-maker but the values for the observed features \obsFeatures observed by the decision-maker are altered. We modified the values so that if the value for \obsFeaturesValue was greater than $1$ it was multiplied by $0.75$ to induce more positive decisions. Similarly, if the subject's \obsFeaturesValue was in the interval $(-2,~-1)$ we added $0.5$ to induce more negative decisions. Additionally the effect of non-informative decisions were investigated by deploying a \emph{random} decision-maker. Given leniency $R$, a random decision-maker give a positive decision $T=1$ with probability given by $R$.
In contrast, Lakkaraju et al. essentially order the subjects and decide $T=1$ with the percentage given by the leniency $R$. We see this as unrealistic: the decisions
on a subject should not depend on the decision on other subject. In the example this would induce unethical behaviour: a single judge would need to jail defendant today in order to release a defendant tomorrow.
