Bias all the way down

A recent study on racial bias in health algorithms found evidence of bias in a widely used algorithm: Black patients with the same level of risk as White patients were less frequently being defined as in need of additional care by the algorithm.

When asked, ‘how can we prevent things like this in the future?’ senior author of the study Sendhil Mullainathan suggests that more questions need to be asked at the prototype stage.

I disagree: the prototype stage is far too late to start worrying about bias.

By the time we get to the prototype stage, bias is potentially too deeply embedded to be fixed. Post hoc decisions will only work in exceptional cases. Why? Because bias goes all the way down. Bias enters AI way before machine learning starts. Indeed, bias in tech is both foundational and layered. Each level of bias, it would seem, is preceded by another level of bias, and some of the layers will simply be too deep to allow redesign to be successful.

Mullainathan’s solution is machine learning retraining. But this solution is available in only limited circumstances. Choice in data set exhibits bias, and a change in data sets will exhibit a different bias. In some cases, a data set will be available that exhibits a minimal bias or an acceptable bias. But this will not be the case for many data sets, some of which are deliberately biased towards the user. Additional limitations on this type of solution are plentiful.

One level of bias enters at the design stage. Airbnb’s business model requires trust between hosts and guests who have never met. Their designed solution was to manufacture trust through the use of profile photos and real names. Yet this design choice neglected the history of racism in the hotel industry, and famously enabled and reproduced existing racist practices. Their post hoc re-design solutions are improving things, but are moving slowly, and remain not altogether satisfactory. There are, I suggest, a number of reasons why the problem is so pernicious.

Another level of bias enters tech at the coding stage. For example, we encode categories of data for AI to learn. How we define the categories in an AI or machine learning algorithm will determine how the data is sorted. Facial recognition software has been criticized for relying on binary classifications of gender. The resulting errors are predictable, but also fundamentally difficult to resolve. Re-training the AI is not an option, when the categorization has been made salient, but the relevant category has been omitted.

Defining the categories is a choice by human beings, and this choice is not free from bias. Choice of categories is not value-neutral. Choice of categories embeds a history. In these and other ways, determining which categories are salient is a choice, one that can be affected by bias, and one that has further implications for downstream biases.

But perhaps the foundational level of bias in tech is bias in hiring. Because, the diversity in the room (or the lack thereof) has implications for which questions are asked, which solutions are considered, and how the ideas under consideration are evaluated.

Bias in tech is a problem, and something we all agree that we want to avoid. Yet, bias in tech is also ubiquitous, and moreover pernicious. Each layer of bias affects the subsequent layers. Many of the fixes are post-hoc, and seem themselves to be oblivious to the depth of the problem. Bias is deeply embedded in tech as in other areas of enterprise. But when we have bias all the way down, tech is not going to be its own solution. The layers of bias are interconnected and intertwined, and to the extent that we can even identify the source of bias, we will rarely – if ever – be able to disentangle it.


Leave a comment

Filed under Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s