Scene completion is the task of filling in holes in a picture, and algorithms for doing it automatically are starting to become impressive. In the series of pictures above, the original picture (left) has an obnoxious roof taking up a lot of the picture. In the next picture, the user erased the roof, leaving a gaping hole in the picture. A current algorithm for filling in holes uses other similar pictures to figure out what should go in the hole, so the next picture shows similar pictures from a picture database. The final picture has the hole filled in with a reasonable guess.

The algorithm that generated the above sequence of pictures is due to James Hays and Alexei Efros and is explained in this paper. It is driven by a database of 2.3 million images of landscape, travel, and city photography. The first stage of the algorithm is to get the “gist” of the scene, which is a descriptor that can distinguish between cities, hillsides, beaches, and so forth. The algorithm uses this to find a set of images from its database that are of the same general type of thing. This prevents the algorithm from filling in hillsides with car roofs.

Once it has a set of images of similar things, it looks at the region around the hole and tries to find similar regions in the list of pictures. Next it chooses a small set of these best fitting regions and blends each of them back into the original hole, making a set of composite pictures without holes. Finally, it presents each of these composite pictures to the user so they can choose which one they like the best.

The algorithm works very well most of the time. However, it still has a few drawbacks. First, it is currently only trained on city, landscape, and travel pictures. As the range of pictures grows, the problem will become more difficult. Second, the program currently runs on a cluster of 15 machines, instead of a single desktop PC. Third, it still makes serious mistakes occasionally (they show a failure case in the article where the bottom half of a lady was filled in with what looks like a trash can). These drawbacks are serious enough to show that scene completion is not yet ready for the mass market, but the overriding message I took from the article was that it will be ready quite a bit sooner than I would have guessed.