With GauGAN you can transform segmentation maps (paint-style doodles) into beautiful landscape pictures. The demo was announced on Twitter last month by Ming-Yu Liu. GauGAN was created using PyTorch deep learning framework and gets its name from the use of generative adversarial networks (GANs). The basic idea behind GANs is that two models compete, in this case one attempts to create a realistic image and a second tries to detect the fake images. The result is higher fidelity images with less training data.

Despite lacking an understanding of the physical world, GANs can produce convincing results because of their structure as a cooperating pair of networks: a generator and a discriminator. The generator creates images that it presents to the discriminator. Trained on real images, the discriminator coaches the generator with pixel-by-pixel feedback on how to improve the realism of its synthetic images.

The NVIDIA team has a great blog post about GauGAN and GANs, they are also giving talks this Sunday at SIGGRAPH about their work. To learn more take a look at their abstract or try implementing the code on GitHub.

Last but not least, here’s some more GauGAN fun: