"My goal is for computers to understand images as humans do," says Assistant Professor James Hays of Brown University's Department of Computer Science (Brown CS), "and for computers to use this understanding to help people interact with and create imagery in new ways."
Hays has just been named an Alfred P. Sloan Research Fellow in one of the oldest and most competitive fellowship programs in the country. He joins multiple previous Brown CS recipients, including Paul Valiant, Ben Raphael, Chad Jenkins, and Amy Greenwald. The fellowships, which take the form of a $50,000 grant used over a two-year period, honor and promote the science of outstanding researchers early in their academic careers who show outstanding promise for fundamental contributions to new knowledge.
"Receiving the Sloan Fellowship is a huge honor," he says. "It supports the research that my students and I pursue to help computers and people interact with image data. We want images to be as easy to edit as text and just as useful as text for anyone in the world." Working at what he describes as "Internet scale", Hays and his team utilize terabytes of data and global crowdsourcing to build tools that affect every aspect of how we create images and make use of the data they hold.
"With images," he notes, "almost everyone is good at discriminative tasks: we can tell good art from bad, or whether something is real or fake. But so few of us are good at generative tasks. There's a strange mismatch there. Why can't we synthesize the images that we can imagine?"
Beginning with his earliest doctoral work, that mismatch is where Hays has aimed his research. He and his team recently used the data gained from thousands of historical images to introduce the first method to recognize and imitate historical color processes. For example, users can make a World War II photo look as if it were taken by a modern digital camera, and vice-versa.
"We hope to take this much further," says James, "so that we can restore grainy, black and white historical photos to appear vivid and modern. This would be a new way to experience history."
And there's much more: an early collaboration between Hays and Adobe was the first use of a photo database to learn a model of sharp image edge patches, repairing blur better than any existing technique. His work in image geolocalization, attempting to localize query photos by matching them to a database of ground-level training images, has ramifications for both photo organization and forensics. Recent experiments with crowdsourced sketching have revealed shared, iconic representations of objects that can be recognized by both humans and machines, leading to forthcoming exploration of crowd active learning and gamification of classifier design.
"The future," says James, "isn't about improvement in resolution. It's not science fiction to think that my kids will expect photos to be 3d and allow re-rendering from different viewpoints. They could be part of the first generation where everyone in the world is able to capture and share whatever is in their mind in the image space."
