Machine Learning: Google and Style Transfer

You knew we didn't post "Auto Mechanics In the Style of  Michelangelo--UPDATED" just because the pictures were pretty didn't you. Even though, as this example in the style of Rembrandt shows, the photographer nails it, from the book, to the lighting to the folds in the coveralls:

A bit of Rembrandt inspired garageiness:

 
No, it was so I could refer back to them.
And they were pretty.
In a manly sort of way. Very manly.

Moving right along, that is a type of style transfer, something that takes a very good eye for detail and almost virtuoso skill with the camera.

And it's something that computers are learning how to do, sans camera.

From Google's Research blog, Oct. 26, 2016

Supercharging Style Transfer

Posted by Vincent Dumoulin^*, Jonathon Shlens and Manjunath Kudlur, Google Brain Team

Pastiche. A French word, it designates a work of art that imitates the style of another one (not to be confused with its more humorous Greek cousin, parody). Although it has been used for a long time in visual art, music and literature, pastiche has been getting mass attention lately with online forums dedicated to images that have been modified to be in the style of famous paintings. Using a technique known as style transfer, these images are generated by phone or web apps that allow a user to render their favorite picture in the style of a well known work of art. Although users have already produced gorgeous pastiches using the current technology, we feel that it could be made even more engaging.

Right now, each painting is its own island, so to speak: the user provides a content image, selects an artistic style and gets a pastiche back. But what if one could combine many different styles, exploring unique mixtures of well known artists to create an entirely unique pastiche? Learning a representation for artistic style In our recent paper titled “A Learned Representation for Artistic Style”, we introduce a simple method to allow a single deep convolutional style transfer network to learn multiple styles at the same time. The network, having learned multiple styles, is able to do style interpolation, where the pastiche varies smoothly from one style to another. Our method enables style interpolation in real-time as well, allowing this to be applied not only to static images, but also videos...

...In the video above, multiple styles are combined in real-time and the resulting style is applied using a single style transfer network. The user is provided with a set of 13 different painting styles and adjusts their relative strengths in the final style via sliders. In this demonstration, the user is an active participant in producing the pastiche. A Quick History of Style Transfer While transferring the style of one image to another has existed for nearly 15 years [1] [2], leveraging neural networks to accomplish it is both very recent and very fascinating. In “A Neural Algorithm of Artistic Style” [3], researchers Gatys, Ecker & Bethge introduced a method that uses deep convolutional neural network (CNN) classifiers. The pastiche image is found via optimization: the algorithm looks for an image which elicits the same kind of activations in the CNN’s lower layers - which capture the overall rough aesthetic of the style input (broad brushstrokes, cubist patterns, etc.) - yet produces activations in the higher layers - which capture the things that make the subject recognizable - that are close to those produced by the content image. From some starting point (e.g. random noise, or the content image itself), the pastiche image is progressively refined until these requirements are met.

Content image: The Tübingen Neckarfront by Andreas Praefcke, Style painting: “Head of a Clown”, by Georges Rouault

 (click to enlarge)

The pastiches produced via this algorithm look spectacular:

Figure adapted from L. Gatys et al. "A Neural Algorithm of Artistic Style" (2015). 

(click to enlarge)

This work is considered a breakthrough in the field of deep learning research because it provided the first proof of concept for neural network-based style transfer. Unfortunately this method for stylizing an individual image is computationally demanding. For instance, in the first demos available on the web, one would upload a photo to a server, and then still have plenty of time to go grab a cup of coffee before a result was available. This process was sped up significantly by subsequent research [4, 5] that recognized that this optimization problem may be recast as an image transformation problem, where one wishes to apply a single, fixed painting style to an arbitrary content image (e.g. a photograph). The problem can then be solved by teaching a feed-forward, deep convolutional neural network to alter a corpus of content images to match the style of a painting. The goal of the trained network is two-fold: maintain the content of the original image while matching the visual style of the painting....MUCH MORE 

I did not recognize painting B, top right, it's a Turner, The Shipwreck of the Minotaur. Here's the Wikipedia image, the sailors and soldiers look totally effed.

Earlier today:
Interview: Manuela Veloso Head of Machine Learning, Carnegie Mellon University

For pastiches I usually use Bohemian Rhapsody as the example:

(not Queen) Bohemian Gravity
Definitely not Queen.

To quote myself, from "Media: William Shatner Covers Queen's 'Bohemian Rhapsody'":
We've commented on the structural genius of Bohemian Rhapsody a couple times, links below the video....

Lyrical Genius:

Hold all Tickets. We had thought "The Day the Nasdaq Died" the pastiche* of the decade:

The Day the NASDAQ Died

Humble Pie (sung to the tune of American Pie)...

*..."Bohemian Rhapsody", by Queen is unusual as it is a pastiche in both senses of the word, as there are many distinct styles imitated in the song, all 'hodge-podged' together to create one piece of music....
(Wikipedia)

GOT THAT? WE HAVE A PASTICHE OF A DOUBLE PASTICHE!!!
Genius.

Previously:
March 2008:
Lyrical Genius:

CDO Remake of Bohemian Rhapsody

Is this the real price?
Is this just fantasy?
Financial landslide
No escape from reality
Open your eyes
And look at your buys and see.
I’m now a poor boy
High-yielding casualty...MORE

A few years later Alphaville stepped up:
September 2011
Genius: "Athenian Rhapsody (updated)"

Pastiche time baby!*
From FT Alphaville:

Alan Beattie, FT International Economy Editor and FT Alphaville’s new A&R man, has sent us a rock opera by two parodists who wish to remain anonymous.

We’ve done all we can to confirm one of them isn’t Ben Elton but we can’t give you our full reassurance.
Our sincerest apologies to Mercury, May, Taylor and Deacon....MORE