The game Pokémon Go for smartphones and tablets has given Augmented Reality a considerable push forwards. Researchers from the Max Planck Institute for Information Technology in Saarbrücken have succeeded in raising this technique to a new quality level: the actual digital image is no longer supplemented with visual information, but the image itself is photo-realistically altered in real time. In this way a surface can have another color and/or another texture – so that it appears to have been made from a different material than what it really is.

That sounds easier than it is. The color value that a camera assigns to each individual pixel is – to put it simply – always the product of reflection and lighting. The problem however, is that both these components are only available indirectly from the image and therefore cannot be measured.

The researchers have solved this problem in a clever way. At a very high speed, for every pixel the lighting and the amount of reflection is estimated and only one of these factors is changed. As a result a realistic impression remains.

For this estimation of indirect pixels a mathematical optimization method is used. Because the software computes the new values in parallel the changes can be made so quickly that even the live stream from an ordinary webcam can be photo-realistically altered.