Universal Style Transfer via Feature Transforms: Difference between revisions
No edit summary |
No edit summary |
||
| Line 7: | Line 7: | ||
=Related Work= | =Related Work= | ||
Gatys et al. developed a new method for generating textures from sample images in 2015 [1] and extended their approach to style transfer by 2016 [2]. They proposed the use of a pre-trained convolutional neural network (CNN) to separate content and style of input images. A CNN was chosen due to its abilitiy to extract high level features from images. These features can be interpreted in two ways. Within layer <math> l </math> there are <math> N_l </math> features maps of size <math> M_l </math>. With a particular input image, the output of the feature maps is given by <math> F^l < | Gatys et al. developed a new method for generating textures from sample images in 2015 [1] and extended their approach to style transfer by 2016 [2]. They proposed the use of a pre-trained convolutional neural network (CNN) to separate content and style of input images. A CNN was chosen due to its abilitiy to extract high level features from images. These features can be interpreted in two ways. Within layer <math> l </math> there are <math> N_l </math> features maps of size <math> M_l </math>. With a particular input image, the output of the feature maps is given by <math> F^l </math>. Starting with a white noise image and an reference (content) image, the features can be transferred by minimizing | ||
<math> \mathcal{L} = \frac{1}{2} \sum_{i,j} \left( F_{i,j}^l - P_{i,j}^l \right)^2 </math> | <math> \mathcal{L} = \frac{1}{2} \sum_{i,j} \left( F_{i,j}^l - P_{i,j}^l \right)^2 </math> | ||
Revision as of 22:04, 23 October 2017
Under construction!
Introduction
When viewing an image, whether it is a photograph or a painting, two types of mutually exclusive data are present. First, there is the content of the image, such as a person in a portrait. However, the content does not uniquely define the image. Consider a case where multiple artists paint a portrait of an identical subject, the results would vary despite the content being invariant. The cause of the variance is rooted in the style of each particular artist. Therefore, style transfer between two images results in the content being unaffected but just the style being copied. Typically one image is termed the content/reference image, whose style is discarded and the other image is called the style image, whose style, but not content is copied.
Deep learning techniques have been shown to be effective methods for implementing style transfer. Previous methods have been successful but with several key limitations. Either they are fast, but have very few styles that can be transferred or they can handle arbitrary styles but are no longer efficient. The presented paper establishes a compromise between these two extremes by using only whitening and colouring transforms to transfer a particular style. No training of the underlying deep network is required per style.
Related Work
Gatys et al. developed a new method for generating textures from sample images in 2015 [1] and extended their approach to style transfer by 2016 [2]. They proposed the use of a pre-trained convolutional neural network (CNN) to separate content and style of input images. A CNN was chosen due to its abilitiy to extract high level features from images. These features can be interpreted in two ways. Within layer [math]\displaystyle{ l }[/math] there are [math]\displaystyle{ N_l }[/math] features maps of size [math]\displaystyle{ M_l }[/math]. With a particular input image, the output of the feature maps is given by [math]\displaystyle{ F^l }[/math]. Starting with a white noise image and an reference (content) image, the features can be transferred by minimizing
[math]\displaystyle{ \mathcal{L} = \frac{1}{2} \sum_{i,j} \left( F_{i,j}^l - P_{i,j}^l \right)^2 }[/math]
The 19-layer VGG network was chosen by Gatys et al. and is still commonly used in more recent works as will be shown in the presented paper.
References
[1] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis using convolutional neural networks. In NIPS, 2015. [2] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016.