Twitter started using a saliency algorithm in 2018 to crop images. We did this to improve consistency in the size of photos in your timeline and to allow you to see more Tweets at a glance. The saliency algorithm works by estimating what a person might want to see first within a picture so that our system could determine how to crop an image to an easily-viewable size. Saliency models are trained on how the human eye looks at a picture as a method of prioritizing what’s likely to be most important to the most people. The algorithm, trained on human eye-tracking data, predicts a saliency score on all regions in the image and chooses the point with the highest score as the center of the crop.
Here’s what we found:
Rumman Chowdhury
- In comparisons of men and women, there was an 8% difference from demographic parity in favor of women.
- In comparisons of black and white individuals, there was a 4% difference from demographic parity in favor of white individuals.
- In comparisons of black and white women, there was a 7% difference from demographic parity in favor of white women.
- In comparisons of black and white men, there was a 2% difference from demographic parity in favor of white men.
A single-digit deviation in any direction seems insignificant to me. A fine example of how people on Twitter (and in general) can latch onto marginal issues and blow them completely out of proportion – in this case how Twitter’s automated image cropping could sometimes select the faces of white people over black people.
Amusingly, the solution Twitter found to this issue is… to simply reduce its use of algorithms!
In March, we began testing a new way to display standard aspect ratio photos in full on iOS and Android — meaning without the saliency algorithm crop. The goal of this was to give people more control over how their images appear while also improving the experience of people seeing the images in their timeline. After getting positive feedback on this experience, we launched this feature to everyone. This update also includes a true preview of the image in the Tweet composer field, so Tweet authors know how their Tweets will look before they publish. This release reduces our dependency on ML for a function that we agree is best performed by people using our products. We’re working on further improvements to media on Twitter that builds on this initial effort, and we hope to roll it out to everyone soon.
The broader point here is that machine learning can never fully cover any possible edge case. You either have to accept these fringe cases (when the damage is minimal, as it was the case here in my opinion), correct for them manually (if the impact is important, for example hiring decisions and credit scores), or abandon the algorithmic solution (as Twitter has done here).
Post a Comment