Technology is not neutral, and can used to force trans people into boxes that don’t fit them. Trans identities, and in particular non-binary identities, are often erased, invalidated or overridden.

The song Binary  by The Spook School was flooding my mind during a recent conversation with the CEO of a technology startup. The company provides cloud-based computer vision software for a variety of purposes: detecting objects; detecting patterns; detecting faces. But one of their offerings is facial analysis — a moment in front of their camera will generate a report of characteristics, including age, current emotion, and gender.

It’s a learning algorithm that has been trained on a bank of images of many cis people, and spits out a probability that the subject is either male or female.  This can then be used by a product designer to whatever end.

The key point is not, of course, simply that it erases non-binary identities. Even if the developers tried to add one or several non-binary categories, or tried to train their algorithm with the faces of trans people, the feature is fundamentally broken: unless the software asks what someone’s identity is, trans people will always run afoul of the system. The conversation was productive, and the CEO accepted that the software was broken – but there’s a wider conversation to be had.

In 2016, Santander’s fraud department blocked bank accounts if a customer sounded “the wrong gender” on the telephone – and voice-pattern analysis for telephone banking is on the rise. Bank staff are still told to flag up where customer gender is “suspect”. With the advent of smart billboards, “customer-responsive adverts” that are programmed with “gender detection” can out trans people in the street by simple failure of this algorithm.

Inherently -phobic technologies are not new: the apple watch and automatic toilet taps have famously failed on black skin, and insurance company and police algorithms disproportionately target minority populations. But let’s not have yet another “neutral” technology with the power to wreck us financially and endanger us in the street.