Preventing racist, sexist and biased museum machines


After being arrested on charges of attempted burglary, facing possession of cocaine and marijuana, Dylan Fugett was evaluated as low risk to reoffend. This wasn’t a conclusion reached by a judge, or even a police officer – instead by a machine: a forecasting criminal risk assessment technology gaining popularity in states struggling to keep up with prison populations. Across town, Bernard Parker, a young African American man, wasn’t so fortunate. Despite only a single prior charge of resisting arrest, he was rated a 10/10 risk of reoffending. Fugett went on to reoffend multiple times on drugs charges, Parker did not.

Far too many cases like theirs, where offenders’ criminal profiles point to a different conclusion, indicated an inherent bias in the machine learning models used to make these assessments, featuring ethnicity influences. A similar fate faced the contestants of the world’s first beauty contest to be judged by a machine. Of the 44 winners, only a single person of color was selected by the machine’s highly flawed definition of beauty. Or consider the flaws inherent in the many recruitment algorithms supposedly taking prejudice out of hiring, yet only making a bad situation worse given they have been trained on historical data of teams suffering a severe lack of diversity.

These are just a few examples behind a growing concern of a lack of ethics in artificial intelligence and inherent bias passed down from human to machine. With many visitor attractions beginning to dabble in various forms of the technology, the cultural sector needs to get proactive in how it prevents examples like these from arising in its own ranks. Institutions such as museums have led the way in diversity related corrections, such as the introduction of gender neutral restrooms – but risk taking a step back if discriminatory practices are not pre-empted in the digital realm. If anything, this industry should look to lead the way in ethical considerations in the artificial intelligence space.

Left unchecked, with technology like machine vision and bots creeping into the visitor experience, visitor attractions risk alienating the public with offensive, harmful experiences on diversity factors such as ethnicity, gender and orientation. Consider Richard Lee, a person of Asian descent, who found himself unable to apply for a passport when the machine vision algorithm responsible for verifying his photo interpreted Lee’s facial characteristics as a subject whose eyes were closed. In examples where visitor attractions are using artificial intelligence to create art or experiences based on a capture of the visitor’s face, this is exactly the sort of scenario that could replicate itself in a cultural sector setting.

It’s not just visitor centric applications, but collection facing innovation too. At the University of Washington, research training neural networks on a collection of images found women were a third more likely to appear in photographs relating to cooking. The result for their technology’s predictions meant the machine was 68% more likely to place a woman in an image related to the kitchen, even when presented with a photograph of a man.  Or the infamous example from Google, whose photo categorization tool labelled an image of an African American couple as gorillas and disappointingly only ‘fixed’ it with a quick workaround rather than addressing the issue. With a growing number of museums using artificial intelligence to catalog, interpret and make connections between collection images, this same situation could easily play out. Given the sheer volume of vast collections spanning millions of images, it’s not necessarily an issue that would be detected prior to a technology’s release and could easily slip through to be found instead by the visiting public – and quickly make its way into the press.

It’s not necessarily that the machine is inherently bad, more so a case of taking the cliché of ‘bad data in, bad data out’ to another level. MIT proved this point by training a robot on a strict diet of macabre Reddit images. The machine, aptly named ‘Norman’ as a nod to the film Psycho, sees death and destruction everywhere it looks – even with non descript ink blots that a normal artificial intelligence would otherwise decipher as everyday scenes. Should a museum dealing with a delicate subject matter such as war or genocide train a model specifically on its collection, a similar outcome may ensue.

The same is true for artificial intelligences trained by the general public. Microsoft’s bot Tay, an attempt to engage the public in conversation, took less than 24 hours to be utterly corrupted by the scum of the internet, tweeting out offensive remarks ranging from transphobia to antisemitism. With multiple museums rolling out chat bots, or in some cases, physical robots, inappropriate or intentionally damaging public use cannot be discounted (as any museum with a digital interactive allowing visitors to draw pictures would have already discovered).  

The wider issue of data ethics surrounding artificial intelligence deals with more than just bias. As with any technology advancement, some parts of society will find a way to criminally exploit it. Google’s recent demonstration of its new voice Assistant proved so real, it scheduled a hair appointment without the listener even being aware they were talking to a machine. Pushing the boundary further to break the trust in the intimacy of voice, start up Lyrebird launched a technology that mimics the voices of real people – proving the point by training it to sound just like Donald Trump, Hillary Clinton and Barack Obama. While the world worries about misuse of this technology from an identity theft standpoint, in the era of fake news, academic institutions should also be concerned with the fight against intentionally misleading rewrites to historic record. Fortunately, the examples of this aren’t all bad, with clever innovators like Netsafe designing artificial intelligence to beat scammers at their own game.

So what can a visitor attraction do to prevent ethical issues seeping into their digital landscape? Silicon Valley is slowly learning that diversity needs to be inherent to the development team – a bunch of young guys in hoodies aren’t the answer to design for diverse user experiences. Albeit unintentionally, they’re less likely to build in protections, feed appropriate training data sets or test for user profiles alternate to their own. As the Santa Cruz Museum of Art and History puts it, design should instead be ‘OFBYFORALL’.

Equally, ethics are not just a matter for engineers, with organizations such as Facebook appointing executive oversight committees at the insistence of investors. Given the rise of the Privacy Officer as a result of legislative change, it can be expected such roles might morph into a wider scope of responsibilities as a Data Ethics Officer.  A similar trend is evident in technology giants releasing artificial intelligence ethics statements beside reworked privacy policies, such as Google’s – prompted in part by the company’s difficult handling of its involvement in military applications. Though still the domain of mainstream technology companies, such organizational changes would be positive steps for the visitor attractions sector to take proactively as a natural follow on from recent privacy improvements.

It can be expected these changes provoked by the data ethics requirements of artificial intelligence will eventually include whistleblower, complaint handling and audit procedures, potentially by external auditors. Statistician Cathy O’Neil has gone as far to create a certification seal of approval for well behaved algorithms, in the same way food might be labelled organic or a bank might have a AAA rating. For socially significant issues such as criminal justice, credit scores and surveillance, researchers at AI Now have teamed up with ACLU to keep an eye on civil liberties. Some in the technology sector have gone as far as to suggest the need to study the way in which machines think in a similar way to the evaluation of human psychology.

Legally, under the new General Data Protection Regulation (GDPR), artificial intelligence requires explicit permission if using personal information and outcomes must be able to evidence the basis behind their findings. For example, if a subscriber were targeted under personalization for a particular membership offer by their cultural institution, they have the right to request the reasoning for that conclusion. As a result, any institution employing such technology should feel comfortable offering full public transparency of its algorithm’s featured factors should this occasion arise. Remaining compliant with sharing machine learning findings on request will become very awkward for visitor attractions if the result shows a lack of respect for diversity. Given the challenge of regulation keeping up with the pace of technology change, no doubt this is not the end of the story for legislative oversight.

Yet transparency in machine learning is also beneficial for the institution. For example, in predicting metrics such as visitor attendance, Dexibit details aspects such as expected accuracy and model fit, plus featured factors and their emphasis – tracking actual performance against expected forecast too. This detail involves the assisted professional in designing model enhancements – rather than just blind use – and in itself, is useful intelligence to inform strategic and operational planning decisions, helping the visitor attraction act in an agile manner.

Pulling together the triangle of artificial intelligence ethics for visitor attractions invokes fairness and inclusivity, privacy and security, trust and transparency. Together, these three principles will help the cultural sector use artificial intelligence for good, yet with care to deliver that good in the right way.