Sunday, September 6, 2015

A Neural Algorithm of Artistic Style with Chainer

See this picture. Isn't it awesome?
I implemented the algorithm to synthesize a painting and a picture in a way that represents the photo in the painting style. Here is more images that show how the learning goes. 

Well, it’s two weeks since classes started. If I summarize my phd student life so far in a word, it’s busy. I have lots of reading and writing far more than in Japanese universities. However, if I only do my work, I will be depressed. I need to do something that I can enjoy.

So, what do I like? I love technologies! That’s why I implemented an interesting algorithm proposed recently. It's from a paper called  A Neural Algorithm of ArtisticStyle. If you are not computer science student, this news article is helpful to know what I implemented.

Some thoughts when playing:
  • The paper didn’t mention how they optimized explicitly, so I used simple gradient decent at first, but it was too slow. Then I used Adam to minimize the loss, which I think one of the latest optimization methods. I think momentum gradient descent is also fine.
  • The difference from paper is that I did (could) not use average-pooling instead of max-pooling in VGG pooling layers. I wanted to change max-pooling to average-pooling, but I did not know how to change a part of layer of the imported caffe model in chainer. I need to figure out how to change network structure of a imported caffe model in chainer.
  • There are several parameters that make the output difference, but I am still not sure what is the best parameter value that is used for all paints and pictures.

Here is the code on github:
You can easily run both on GPU and CPU. I recommend you to use GPU if available. It is so much faster.

This was the first time I played with computer vision. To me, it seems like progress of deep learning in computer vision is already off the peak. Now the research is shifting to natural language processing. Although I am not a phd student specializing deep learning,  I look forward to the progress! 


  1. Hi Satoshi,

    Thank you for sharing your article. It looks brilliant.

    I have a question, I'm doing something wrong. I've downloaded the Github code and run the program using the example command 'python -g 0 -c kinkaku.jpg -s style.png -d kinkaku' however the pictures in the log are just grey noise images. The log has not output the image with the style?

    Do I need to run the neural network in a different command to get the styled image?

    Or possibly I currently have 'VGG_ILSVRC_19_layers.caffemodel' file in the same directory as the code? Does this need to be somewhere else?

    Any help would be much appreciated.

    1. Thank you for reporting the issue. Solution is, please use chainer 1.3.2

      pip uninstall chainer
      pip install chainer==1.3.2

      Your command is correct. I tested again with the latest chainer (1.6), and had the same problem. I guess something is wrong with the latest chainer. So I downgraded chaienr to 1.3.2, which is the version I used when writing the codes. Then the code worked properly again.

    2. and images should become clearer like this:

    3. Amazing. Thank you for the quick reply. I've installed chainer 1.3.2 and will be running it tonight. I'll keep you updated.