"Living the Meme: AI as Funny as Humans for Generating Image Captions" (NVDA)

From NVIDIA's blog:

It’s possible to get grad-school credit for writing memes. At least if you use deep learning to do it.
Just ask Lawrence Peirson.

The 23-year-old is pursuing a theoretical astrophysics Ph.D. at Stanford, but decided to enroll in a couple AI courses this year. He and classmate E. Meltem Tolunay came up with a neural network that captions memes for a class project, now published in a whitepaper aptly titled “Dank Learning.” (“Dank,” for the uninitiated, is a synonym for “cool.”)

There are lots of examples of training deep learning models to produce literal captions for an image — for example, accurately captioning an image as “man riding a surfboard” or “child with ice cream cone.” With memes, Peirson’s challenge was to see if a neural network could go beyond literal interpretation and create humorous captions.
Though he was initially skeptical that the memes would be funny, Peirson found that the deep learning model produced “some quite interesting and original humor.”

Attaining Deep Meme
The deep learning network captioned this meme, a variation on the popular advice animals template.To collect training data for the deep learning model, Peirson scraped around 400,000 user-generated memes from the website memegenerator.net. The site provides meme templates and allows users to come up with their own captions.
The dataset included around 3,000 base images, each with many different captions. Since the input data was crowdsourced, there was a wide range in quality of meme captions the deep learning model processed.

“With 400k memes, most aren’t going to be that funny, but at least they teach the system what a meme is, what joke is relevant,” he said.

Internet memes have circulated around the web for years, with a strong foothold in websites like Reddit, Facebook, 9GAG and Quick Meme. The most popular can get more than 2 million unique captions created.

Memes often reference pop culture, current events or esoteric bits of a particular internet subculture. (Peirson runs a meme page called “The specific heat capacity of europium at standard temperature and pressure.”)

As a result, they imbibe both the good and bad of digital culture — the paper notes a bias in the training data towards expletive, racist and sexist memes. Peirson sees the need to filter these out in future work, but points out that “it’s a big problem in natural language processing in general,” not one specific to memes.

The deep learning model was programmed in CUDA and used an NVIDIA TITAN Xp GPU. Peirson and Tolunay tried using both unlabeled data and data labeled with the meme title (for example, success kid or trollface), but saw no significant difference in meme quality.

“They’re very funny in a ‘it sort of makes sense, but not really’ way,” Peirson said. “Memes lend themselves to that kind of humor.”

The deep learning network captioned this meme, a variation on the popular advice animals template.

One Does Not Simply Declare a Meme Dank To evaluate the deep learning model’s success, the collaborators calculated a perplexity score, which checks whether the neural network can identify clear patterns in the data. They calculated this metric for a few hundred memes with preset formats, such as the Boromir meme, which always begins with the phrase “one does not simply.”

But the true test of a meme is whether it’s funny....MORE