Visible to the public Improving Style Transfer with Calibrated Metrics

TitleImproving Style Transfer with Calibrated Metrics
Publication TypeConference Paper
Year of Publication2020
AuthorsYeh, M., Tang, S., Bhattad, A., Zou, C., Forsyth, D.
Conference Name2020 IEEE Winter Conference on Applications of Computer Vision (WACV)
Keywordscalibration, Coherence, Coherence statistic measures, content image, Covariance matrices, E statistics, EC scores, Effectiveness statistic measures, Gatys' loss, histogram loss, integrated circuits, neural style transfer, Neural Style Transfer methods, optimisation, Optimization, Predictive Metrics, pubcrawl, quantitative evaluation procedure, rendering (computer graphics), Resiliency, Scalability, Standards, statistical analysis, style image, style weights, transferred image, universal style transfer
AbstractStyle transfer produces a transferred image which is a rendering of a content image in the manner of a style image. We seek to understand how to improve style transfer.To do so requires quantitative evaluation procedures, but current evaluation is qualitative, mostly involving user studies. We describe a novel quantitative evaluation procedure. Our procedure relies on two statistics: the Effectiveness (E) statistic measures the extent that a given style has been transferred to the target, and the Coherence (C) statistic measures the extent to which the original image's content is preserved. Our statistics are calibrated to human preference: targets with larger values of E and C will reliably be preferred by human subjects in comparisons of style and content, respectively.We use these statistics to investigate relative performance of a number of Neural Style Transfer (NST) methods, revealing a number of intriguing properties. Admissible methods lie on a Pareto frontier (i.e. improving E reduces C, or vice versa). Three methods are admissible: Universal style transfer produces very good C but weak E; modifying the optimization used for Gatys' loss produces a method with strong E and strong C; and a modified cross-layer method has slightly better E at strong cost in C. While the histogram loss improves the E statistics of Gatys' method, it does not make the method admissible. Surprisingly, style weights have relatively little effect in improving EC scores, and most variability in transfer is explained by the style itself (meaning experimenters can be misguided by selecting styles). Our GitHub Link is available1.
Citation Keyyeh_improving_2020