Transforming Photos With Neural Nets

The opening picture is a painting by my sister-in-law, Louise Gray, an artist living in Portland, OR. I became interested in how a Deep Learning neural net can transform photos after seeing an article on Convolutional Neural Nets in my Eurekalert! newsfeed. In researching the concept further I found on github.com two very creative ways of implementing a neural net on my computer.

A neural net is a system consisting of a trained deep learning model with a method to transform a content image, such as a photo, based on the style of an artist. The deep learning model consists of layers of nodes (or neurons) with each layer trained to recognize finer levels of details. The model in this case was developed at the Visual Geometry Group, Department of Engineering Science, University of Oxford. It was trained using thousands of images. The one I used is VGG-19.

The first system I set up was developed by Justin Johnson and is published on Github:

https://github.com/jcjohnson/neural-style

To implement it I used an old hard drive to install Ubuntu 14.04 and followed the detailed installation instructions on Justin’s site. I used CUDA 7.5 (code to utilize the graphics processors on my GeForce GT 740 w/ 4 gb of memory) and Cudnn 5.1.

I had the system analyze Louise’s painting for style and had it apply her style to a photo I took in Ontario’s Chutes Provincial Park:

Chutes

The pictures below show how with each pass through the system the photo is transformed more and more. Neural-style applies both the stylistic aspects of Louise’s painting as well as the colors.

Chutes_Grey_lbfgs_400

400 passes

Chutes_Grey_lbfgs_800

800 Passes

Chutes_Grey_lbfgs

1000 Passes

You can see how the system is picking up the coloration of the rocks, the style of her trees/leaves, and the characteristics of the lines separating the rocks in her painting. The river in the photograph becomes more washed out to be similar to the water in the lake of her painting.

The second system was developed by Chuan (Chris) Li, a postdoc researcher at Mainz University. His system is an extension of Justin Johnson’s system by including Markov Random Fields as a way of applying style to photos. The github site for his system is:

https://github.com/chuanli11/CNNMRF

Since I am a complete novice at setting the parameters for the systems (and there are many!) I found I got good results with Chris’s system with mostly the default settings. While there are most likely a way to have the system apply more of the color schemes of the paintings to the photo, I found I liked the results from his system.

Louise_Grey_cnnmrf

Example of applying the Markov Random Fields to the Convolutional Neural Net

Here you can see that the system is picking up more of Louise’s style of painting the trees as well as the curves of the lines separating the rocks.

Finally I took one of my favorite photos of an egret and applied the style of a Picasso painting. Here is the photo:

DSC_5281

And the Picasso Painting:

picasso

And the end result:

Egret_cnnmrf

For those who want to experiment with neural nets to transform photos there is an app for iPhones on iTunes called Prisma. It uses a similar process to apply a filter to photos. I do not have any Apple products so I can’t comment on how well it works, but I understand an Android version will be coming out shortly.

One other interest of mine is to take a video, separate it out to individual frames and run the frames through the neural net to apply a style to them and then reconstruct it back into a video. My test run turned out to be very noisy and is not quite ready to share. I will continue to work on this. Others have been pretty successful in working with video. One promising solution is the work of Manuel Ruder:(http://github.com/manuelruder/artistic-videos) There is a link to an example of his work at the web site.

I hope you find this interesting. Type at you later.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s