Transforming Photos With Neural Nets

The opening picture is a painting by my sister-in-law, Louise Gray, an artist living in Portland, OR. I became interested in how a Deep Learning neural net can transform photos after seeing an article on Convolutional Neural Nets in my Eurekalert! newsfeed. In researching the concept further I found on github.com two very creative ways of implementing a neural net on my computer.

A neural net is a system consisting of a trained deep learning model with a method to transform a content image, such as a photo, based on the style of an artist. The deep learning model consists of layers of nodes (or neurons) with each layer trained to recognize finer levels of details. The model in this case was developed at the Visual Geometry Group, Department of Engineering Science, University of Oxford. It was trained using thousands of images. The one I used is VGG-19.

The first system I set up was developed by Justin Johnson and is published on Github:

https://github.com/jcjohnson/neural-style

To implement it I used an old hard drive to install Ubuntu 14.04 and followed the detailed installation instructions on Justin’s site. I used CUDA 7.5 (code to utilize the graphics processors on my GeForce GT 740 w/ 4 gb of memory) and Cudnn 5.1.

I had the system analyze Louise’s painting for style and had it apply her style to a photo I took in Ontario’s Chutes Provincial Park:

Chutes

The pictures below show how with each pass through the system the photo is transformed more and more. Neural-style applies both the stylistic aspects of Louise’s painting as well as the colors.

Chutes_Grey_lbfgs_400

400 passes

Chutes_Grey_lbfgs_800

800 Passes

Chutes_Grey_lbfgs

1000 Passes

You can see how the system is picking up the coloration of the rocks, the style of her trees/leaves, and the characteristics of the lines separating the rocks in her painting. The river in the photograph becomes more washed out to be similar to the water in the lake of her painting.

The second system was developed by Chuan (Chris) Li, a postdoc researcher at Mainz University. His system is an extension of Justin Johnson’s system by including Markov Random Fields as a way of applying style to photos. The github site for his system is:

https://github.com/chuanli11/CNNMRF

Since I am a complete novice at setting the parameters for the systems (and there are many!) I found I got good results with Chris’s system with mostly the default settings. While there are most likely a way to have the system apply more of the color schemes of the paintings to the photo, I found I liked the results from his system.

Louise_Grey_cnnmrf

Example of applying the Markov Random Fields to the Convolutional Neural Net

Here you can see that the system is picking up more of Louise’s style of painting the trees as well as the curves of the lines separating the rocks.

Finally I took one of my favorite photos of an egret and applied the style of a Picasso painting. Here is the photo:

DSC_5281

And the Picasso Painting:

picasso

And the end result:

Egret_cnnmrf

For those who want to experiment with neural nets to transform photos there is an app for iPhones on iTunes called Prisma. It uses a similar process to apply a filter to photos. I do not have any Apple products so I can’t comment on how well it works, but I understand an Android version will be coming out shortly.

One other interest of mine is to take a video, separate it out to individual frames and run the frames through the neural net to apply a style to them and then reconstruct it back into a video. My test run turned out to be very noisy and is not quite ready to share. I will continue to work on this. Others have been pretty successful in working with video. One promising solution is the work of Manuel Ruder:(http://github.com/manuelruder/artistic-videos) There is a link to an example of his work at the web site.

I hope you find this interesting. Type at you later.

Cowboy Studio Shoulder Brace

When out shooting wildlife my best lens is a Nikon AF-S FX Nikkor 200-400 f/4G ED VRII zoom lens. Since I use a Nikon D5100 camera that has an APS-C sensor, due to the 1.6 crop factor the 35mm equivalent would be 320-640mm. I chose the lens because it is very sharp, the image quality is great, the f/4.0 aperature lets in plenty of light and the reach of the lens lets me take shots very far away. The downside is that when mounted on my camera the rig weighs about 11 pounds.

The weight isn’t much of an issue when I can use my tripod. However, lugging around a tripod, setting it up, getting it level and such is a bit of a hassle. On the other hand, trying to get a steady shot at a distance while hand holding the camera and lens is very difficult. I have done it trying to shoot birds in flight, but being a bit older my hands normally shake a bit making hand-holding my rig all that much more challenging.

I have tried using my monopod, but it too light weight to handle the 11 pounds. I researched other solutions, but many of the portable platforms like shoulder or chest mounts with gyroscopes were out of my price range.

 

DSC_4494

Cowboy Studio Shoulder Brace

I found the Cowboy Studio shoulder brace online and since it was ~$30 I thought I would give it a try. It is mostly for videographers, but since it is rated to hold up to 13 pounds, I thought it might work for me. Although it took some getting used to I have been able to take shots that I would otherwise have missed.

I attach the Cowboy Studio to the tripod foot on the lens that is closest to the camera (there are two mount points on the 200-400mm lens), which give a pretty good weight distribution. I usually attach my Joby shoulder strap to the further tripod mount on the lens to carry the camera, lens and brace. The curved top piece in the picture goes over my right shoulder and the bottom piece rests against my chest. This brings the camera viewfinder in from of my eye. Even though the brace is made out of hard plastic I haven’t had any issue with stability or it breaking. Since the bottom brace rests on my chest, it does move when I take a breath.

Jim and Bertha

Jim and Bertha Photo B Grant

I call the big lens “Bertha”. Yeah, we’re the people who name our cars, camper and so on. . .

The opening picture of the bald eagle was taken in Alaska while I was on a whale watching trip in the Inner Passage. I was standing on the back of a motorized 40 foot catamaran in three foot seas shooting at the eagle from almost 100 yards. The image is cropped significantly and post processed in Lightroom.

DSC_0628

Whale Tail

I did get some shots of humpback whales in the Inner Passage. This was taken from 150 yards or so. You can see the shallow depth of field by looking at how the water is slightly out of focus in front of and behind the tail.

DSC_0537

Out of focus eagle

While I am not fond of this shot I have included to show one of the challenges in shooting birds in flight with such a cumbersome rig. The camera picked up the rock and vegetation behind the eagle as it was taking off. The autofocus can’t keep up with objects moving rapidly towards the camera.

DSC_0850

Great Horned Owl

This photo of a great horned owl was taken in Alaska while on a jet boat trip up one of the rivers. I was standing in the back of the jet boat maybe 60 feet from the bird. This image was not cropped, but it was post processed in Lightroom. The day was fairly overcast, so I was shooting wide open (f/4.0) and the ISO pushed up to 320.

None of these shots would probably be possible without the vibration reduction (Nikon’s term for image stabilization) built into the lenses I use. I tend to shut off the VR when I am using the tripod, but engage it when I use the brace. I have tried to use the brace with some of my other lenses, especially when I am using a 1.4X teleconverter to extend the reach. I have found it is easier and more versatile to just hand hold the camera and lens in those situations.

DSC_5281_picc

Egret

It is always nice when a bird holds still for its picture. I have a 13″ X 19″ print of this on the wall of my office.

In summary, if you have a moderate to heavy lens in your collection and want to try a different platform from a tripod or monopod, I would recommend the Cowboy Studio Shoulder Brace. At under $30.00, what do you have to lose?

Floors, Doors and Knockers

While on a cruise around Italy my wife, B, along with my sister, Judy Lewis and her husband, Bill Lewis, took a different excursion than me and my brother and sister-in-law. B and Bill were on the motor coach seated behind two ladies from the United Kingdom. The streets in town were very narrow and so their window was close to the buildings along the street.

From time to time one or other of the ladies would look out the window and loudly exclaim:

Look at those knockers!

In the area that Bill grew up “knockers” were one of the terms used to mean female breasts. As a result he and B were chuckling every time the ladies pointed out the lovely door knockers on the buildings.

On a subsequent trip to Europe B and I spent some time appreciating the workmanship and overall beauty not only for doors and their ornamentation, but also floors, manhole covers, and the like. While not particularly challenging to photograph (as compared to a bird in flight) the images are satisfying and help remind us of our trip.

DSC_2263

Melk Abbey, Melk Austria

The Melk Abbey in Melk, Austria is on a hill overlooking the Danube River. The interior of the abbey is stunning. We were there in December, so the gardens were closed. We intend to go back in springtime to tour the grounds of the abbey.

The abbey was heated with huge wood stoves in many of the rooms. There were passages within the walls through which people could stoke the fires and empty the ashes from the passage so that wood debris and ashes would not dirty the main halls. This miniature door was the access to the inner passages. The size, detail of the hinges and inlay of the door itself caught our eye.

DSCN2615

Passau, Germany

Seeing doors like this in Germany makes me wish I didn’t live in a condo where all of the outer doors have to be uniform. This is an outer door, so was taken without a flash. The main challenge in getting interior doors is reflection of the flash on the shiny wood finishes.

DSC_2361

Intricately Carved Interior Door

I believe this was also in Passau, Germany. I love the hand carved detail. Some flash reflection is noticeable, but overall a nice representation of the woodworking from the area.

DSC_2822

Mosaic Floor – Cologne, Germany

Here we have an intricate mosaic floor in the cathedral in Cologne Germany. The cathedral was completed in 1880, but I am not sure how old the mosaic is. Having done just one small tile project, I am in awe at this level of craftsmanship.

Here are examples of a beautiful courtyard, a tile floor and a palace carpet. Love the patterns.

Finally, how about these knockers?

DSC_2137DSC_2293DSC_2524

DSC_2813

Lion’s Head Knocker

I find that focusing in on these types of architectural details deepens my appreciation of beauty to be found in every day objects.

 

Getting Warmed Up

Welcome to the blog. I am looking forward to sharing with you some of the experiences I have had in taking pictures these many years. I have found there are many steps along the way of creating interesting photos. Some are relatively easy to master and others I have struggled with most of my life.

I am not a professional photographer, but an avid hobbyist. I have won one photo contest that was sponsored by my employer. I think I got a gift card for winning. Here is the winning photo taken at Madeira Beach in Florida at sunset:

OLYMPUS DIGITAL CAMERA

Sunset with waves at Madeira Beach

I think one of the most challenging aspects of photography is the transition when I am immersed in a setting regardless of whether it is in nature, inside a building, or in an urban area,  making the switch from just “being in the moment” to looking at the setting as a scene with my artistic eye. What part of the scene am I trying to capture? How is the lighting going to affect my approach to taking a shot? Am I going to isolate one part of the scene and zoom in or am I going to try to capture the whole scene?

As these questions crop up in my head I begin to feel isolated from the setting and involved with the task of taking a picture. It is like I am stepping out of the frame. It tends to be more jarring to me when I am immersed in nature.

The next challenge is getting to know the tools I use for photography. These include my cameras, lenses, platforms (tripod, monopod, shoulder brace, and straps), as well as various software I use for editing and transforming photos. While current digital cameras can be set to use auto settings, getting to know all of the functions for aperture, shutter speed, white balance, focusing options, filters, effects, histograms and such can be quite daunting.

The next step along the way is to use the knowledge of the tools to capture the scene the way I want it. Generally I am looking for a crisp shot with the right level of detail within the correct depth of field. In some cases some action blurring, such as a waterfall taken at a slower shutter speed, is what I go for. In any case I am trying to get something that will work for me in post processing.

Speaking of post processing, there are purists who believe that photos should be finished after the shutter button is pressed and the image is saved to the memory card. Obviously from the image at the top of this page, I am of the school that most everything is possible in post processing. I routinely use an image clarifier program, editing software, and recently been experimenting with convolutional neural nets that can learn the style of a particular artist and then apply that style to a photograph. In the case of the image above I passed a photo of me through a neural net trained on Vincent Van Gogh’s self portrait with a grey hat.

 

new-003-01

As I progress with this blog I will go into much more detail about neural nets as well as the other aspects to photography I have mentioned. Welcome again, and I hope you enjoy my stories.