Skip to content
Snippets Groups Projects
sl.tex 5.14 KiB
Newer Older
  • Learn to ignore specific revisions
  • \documentclass[sigconf,anonymous]{acmart}
    % \documentclass[sigconf]{acmart}
    
    %
    % TO DO
    % - Use 'case' instead of 'subject', unless we talk explicitly about people
    % - Use 'recent' intead of 'state-of-the-art' to refer to lakkaraju.
    
    % For camera-ready version: change these
    \settopmatter{printacmref=false}
    \settopmatter{printccs=false}
    \setcopyright{none}
    
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    \sloppy
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    
    \usepackage{tikz}
    \usepackage{tikz-cd}
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    \usetikzlibrary{shapes}
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    \usetikzlibrary{arrows,automata, positioning}
    
    
    % Packages
    \usepackage{type1cm}     % type1 computer modern font
    \usepackage{graphicx}     % advanced figures
    \usepackage{xspace}     % fix space in macros
    \usepackage{balance}     % to better equalize the last page
    \usepackage{multirow}     % multi rows for tables
    \usepackage[font={bf}, tableposition=top]{caption}     % captions on top for tables
    \usepackage{bold-extra}     % bold + {small capital, italic}
    \usepackage{siunitx}          % \num for decimal grouping
    \usepackage[vlined,linesnumbered,ruled,noend]{algorithm2e}     % algorithms
    \usepackage{booktabs}     % nicer tables
    %\usepackage[hyphens]{url}     % handle long urls
    %\usepackage[bookmarks, pdftex, colorlinks=false]{hyperref}     % clickable references
    %\usepackage[square,numbers]{natbib}     % better references
    \usepackage{microtype}    % compress text
    \usepackage{units}     % nicer slanted fractions
    \usepackage{mathtools}     % amsmath++
    %\usepackage{amssymb}     % math symbols
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    \usepackage{amsmath}
    
    \usepackage{relsize}
    \usepackage{caption}
    \captionsetup{belowskip=6pt,aboveskip=2pt} % to save space.
    %\usepackage{subcaption}
    % \usepackage{multicolumn}
    \usepackage[]{inputenc}
    \usepackage{xfrac}
    \RequirePackage{graphicx,color}
    \usepackage[font={small}]{subfig} % subfig, 4 figures in a row
    \usepackage{pifont}
    \usepackage{footnote} % show footnotes in tables
    \makesavenoteenv{table}
    
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    \newtheorem{problem}{Problem}
    
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %\newcommand{\ourtitle}{Evaluating Decision Makers over Selectively Labeled Data}
    
    
    
    %\newcommand{\ourtitle}{A Causal Approach to\\Evaluating Decision Makers over Selectively Labeled Data}
    
    \newcommand{\ourtitle}{Evaluating Decision Makers over Selectively Labeled Data:\\
    A Causal Modeling Approach
    }
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    % A in the Presence of Unobservables and Selective Labels
    % A Causal Treatment for Unobservables and Selective Labels
    % Incomplete Data
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %-unobservables
    %-selective labels
    
    %-causal
    %-bayesian
    
    
    
    \input{macros}
    
    \usepackage{chato-notes}
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %\author{Michael Mathioudakis}
    %\affiliation{%
     % \institution{University of Helsinki}
     % \city{Helsinki} 
     % \country{Finland} 
    %}
    %\email{michael.mathioudakis@helsinki.fi}
    
    
    
    \begin{abstract}
    
    Today, AI systems replace humans in an increasing number of decisions affecting people's lives.
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    Therefore, it is important to evaluate the performance of such systems {\it offline}, i.e., before they are deployed in real settings --
    and compare it to the performance of human decisions they aim to replace.
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    The data which such evaluation is performed on has two major challenges, biasing any direct evaluations of considered decision makers.
    %
    First, in most cases the data does not include all factors that play a role in the decisions recorded in it.
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    Second, the past decision in the data may skew the data, and, in particular, any possible outcomes recorded in it.
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %Another major challenge in such cases is that often past decisions have skewed the data on which the evaluation is performed. 
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    For example, when a bank decides whether a customer should be granted a loan, it is desired to grant loans to customers who would honor its conditions, but not to ones who would violate them.
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    However, we can directly observe outcomes only for the decisions to grant the loan, while we cannot observe whether customers who were not granted the loan would indeed violate its conditions. 
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %
    %THIS IS NOT SKEW THIS IS MISSING DATA
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    Such difficulties appear in the recorded decisions of both human and AI decision makers -- and should be properly taken into account for evaluation.
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %Further complications arise since commonly not all features that the decisions are based on are observed. DISCUSS UNOBSERVABLES IN THE INTRO?
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    In this paper, we develop a Bayesian approach towards this end.
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    We use a proper causal model of the decision making process, taking into account also the unobserved features.
    Based on this model, we compute counterfactual outcomes, which in turn allow us to produce accurate evaluations of decision maker policies.
     %to correct any aforementioned biases. 
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
     % to infer unobserved outcomes.
    %
    Compared to previous methods for this setting, the approach estimates the quality of decisions more accurately and with lower variance. 
    
    Michael Mathioudakis's avatar
    Michael Mathioudakis committed
    %
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    The approach is also %demonstrated to be 
    robust to different variations in the decision mechanisms in the data.
    
    \end{abstract}
    
    
    \begin{document}
    
    
    \fancyhead{}
    \maketitle
    
    \renewcommand{\shortauthors}{Authors}
    
    
    
    \input{introduction}
    
    \input{setting}
    
    \input{imputation}
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    
    
    Riku-Laine's avatar
    Riku-Laine committed
    \input{experiments} 
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    
    
    \input{related} 
    
    \input{conclusions}
    
    Riku-Laine's avatar
    Riku-Laine committed
    
    
    % \textbf{Acknowledgments.}
    %The computational resources must be mentioned. 
    
    
    %\clearpage
    % \balance
    \bibliographystyle{ACM-Reference-Format}
    \bibliography{biblio}
    %\balancecolumns % GM June 2007
    
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    %\clearpage
    
    Antti Hyttinen's avatar
    Antti Hyttinen committed
    
    
    \end{document}