Das, Batra, and their colleagues then try to get a sense of how the network makes its decisions by investigating where in the pictures it chooses to look. What they have found surprises them: When answering the question about drapes, the network doesn’t even bother looking for a window. Instead, it first looks to the bottom of the image, and stops looking if it finds a bed. It seems that, in the dataset used to train this neural net, windows with drapes may be found in bedrooms.
While this approach does reveal some of the inner workings of the deep net, it also reinforces the challenge presented by interpretability. “What machines are picking up on are not facts about the world”, Batra says. “They’re facts about the dataset.” That the machines are so tightly tuned to the data they are fed makes it difficult to extract general rules about how they work. More importantly, he cautions, if you don’t know how it works, you don’t know how it will fail. And when they do they fail, in Batra’s experience, “they fail spectacularly disgracefully.”
Aaron M. Bornstein
While machine learning has shown promise lately in the field of artificial intelligence, the complexity of the algorithms prevents even the people designing them from understanding how exactly does their machine work. This may not seem important at first, except… Not understanding the reasons behind the success can have many unintended, unpredictable consequences. It means you can’t improve the results, because you don’t know the factors contributing to the efficiency of the machine. It also means it’s very hard to spot false answers, as described in the example above. That may not seem such a big deal when AI is beating humans at board games, but if you employ these methods on more sensitive areas like treating diseases or managing city traffic the results could be catastrophic. AI could provide excellent results in some cases (on datasets similar to the one used for its training) but fail miserably in others – a common concern with self-driving cars. Placing faith in empirical success is not exactly a good method for advancing in a complex field – without an underlying theory progress can stall at any moment without a new path to follow.
Related: researchers fool many AI facial recognition systems with cheap paper eyeglass frames:
The CMU work builds on previous research by Google, OpenAI, and Pennsylvania State University that has found systematic flaws with the way deep neural networks are trained. By exploiting these vulnerabilities with purposefully malicious data called adversarial examples, like the image printed on the glasses in this CMU work, researchers have consistently been able to force AI to make decisions it wouldn’t otherwise make.
Dave Gershgorn
Post a Comment