有限色彩空间下的线稿上色

目的 线稿上色是由线条构成的黑白线稿草图涂上颜色变为彩色图像的过程,在卡通动画制作和艺术绘画等领域中是非常关键的步骤。全自动线稿上色方法可以减轻绘制过程中烦琐耗时的手工上色的工作量,然而自动理解线稿中的稀疏线条并选取合适的颜色仍较为困难。方法 依据现实场景中特定绘画类型常有固定用色风格偏好这一先验,本文聚焦于有限色彩空间下的线稿自动上色,通过约束色彩空间,不仅可以降低语义理解的难度,还可以避免不合理的用色。具体地,本文提出一种两阶段线稿自动上色方法。在第1阶段,设计一个灰度图生成器，对输入的稀疏线稿补充线条和细节,以生成稠密像素的灰度图像。在第2阶段,首先设计色彩推理模块,从输入的颜色先验中推理得到适合该线稿的色彩子空间,再提出一种多尺度的渐进融合颜色信息的生成网络以逐步生成高质量的彩色图像。结果 实验在3个数据集上与4种线稿自动上色方法进行对比,在上色结果的客观质量对比中，所提方法取得了更高的峰值信噪比（peak signal to noise ratio,PSNR）和结构相似性（structural similarity index measure,SSIM）值以及更低的均方误差;在上色结果的色彩指标对比中,所提方法取得了最高的色彩丰富度分数;在主观评价和用户调查中,所提方法也取得了与人的主观审美感受更一致的结果。此外,消融实验结果也表明了本文所使用的模型结构及色彩空间限制有益于上色性能的提升。结论 实验结果表明,本文提出的有限色彩空间下的线稿自动上色方法可以有效地完成多类线稿的自动上色,并且可以简单地通过调整颜色先验以获得更多样的彩色图像。

Sketch colorization with finite color space prior

Objective In the art field,an exquisite painting typically takes considerable effort from sketch drawing in the early stage to coloring and polishing.With the rise of animation,painting,graphic design,and other related industries,sketch colorization has become one of the most tedious and repetitive processes.Although some computer-aided design tools have appeared in past decades,they still require humans to accomplish colorization operations,and drawing an exquisite painting is difficult for ordinary users.Meanwhile,automatic sketch colorization is still a difficult problem.Therefore,both the academia and industry are in urgent need of convenient and efficient sketch colorization methods.With the development of deep neural networks(DNNs),DNN-based colorization methods have achieved promising performance in recent years.However,most studies have focused on grayscale image colorization for natural images,which is quite different from sketch colorization.At present,only a few studies have focused on automatic sketch colorization,and they typically require user guidance or are designed for certain types,such as anime characters.However,automatically understanding sparse lines and selecting appropriate colors remain extremely difficult and ill-posed problems.Thus,disharmonious colors frequently appear in recent automatic sketch colorization results,e.g.,red grass and black sun.Therefore,most sketch colorization methods reduce difficulty through user guidance,which can be roughly divided into three types:reference image-based,color hints-based,and text expression-based.Although these semi-automatic methods can provide more reasonable results,they still require inefficient user interaction processes.Method In practice,we observe the phenomenon that colors used in paintings of a particular style are typically fixed and finite,rather than arbitrary colorization.Therefore,this study focuses on automatic sketch colorization with finite color space prior,which can effectively reduce the difficulty of understanding semantics and avoid undesired colors.In particular,a two-stage multi-scale colorization network is designed.In the first stage,a subnetwork is proposed to generate a dense grayscale image from the input sparse sketch.It adopts a commonly used U-Net structure.The U-Net structure can obtain a large receptive field,and thus,is helpful for understanding high-level semantics.In the second stage,a multi-scale generative adversarial network is designed to colorize the grayscale image in accordance with the input color palette.In this study,we adopt three scales.At the minimum scale,the input color palette prior is used to guide color reasoning,which contains several specified dominant colors.Then,the sketch content features extracted by the content encoder are converted into spatial attention and multiplied by the color features extracted by the color encoder to perceive preliminary color reasoning.The structure of two other scales is similar to the minimum scale.Color guidance information is gradually fused to generate higher-quality and higherresolution results.Adversarial learning is adopted.Hence,three scales of discriminators that correspond to each generator and a discriminator for grayscale generation are used.Intermediate loss is designed to guide the generation of grayscale image in the first stage.Pixel-wise loss,adversarial loss,and TV loss are used in the second stage.Considering the application scenarios,we construct three datasets,namely,the Regong Tibetan painting dataset,the Thangka(Tibetan scroll painting)elements dataset,and a specific cartoon dataset.Each dataset contains images,color palettes,and corresponding sketches.Result Our model is implemented with the Tensorflow platform.For the three datasets,all the images with the size of 256 × 256 pixels are divided into training and testing sets with a ratio of 9:1.The proposed method is compared with the baseline model Pix2Pix and several automatic sketch colorization methods:AdvSegLoss,PaintsChainer,and Style2Paints V4.5.Notably,PaintsChainer and Style2Paints are colorization tools that can be used directly,and they require humans to input sketches manually.Hence,we only retrain Pix2Pix and AdvSegLoss with their official implementations on the same datasets for fair comparison.For quantitative evaluation,peak signal to noise ratio(PSNR),structural similarity index measure(SSIM),and mean squared error(MSE)are used to measure the objective quality of the output coloring images.Meanwhile,the colorfulness score is used to measure the color vividness of coloring images.The proposed method nearly achieves the highest PSNR and SSIM values and the lowest MSE values on the three datasets,and it obtains the best colorfulness scores on all datasets.Although quantitative evaluation has shown the effectiveness of the proposed method,comparing colorization results through human subjective feelings is more meaningful.Hence,subjective results and user studies are provided to evaluate the performance of the proposed method.For qualitative evaluation,the proposed method can reproduce harmonious colors and obtain better visual perception than comparison methods.Furthermore,owing to the decoupled color reasoning and fusion modules in our model,the color of the output image can be changed flexibly with different input color priors.In a user study,most users prefer our results on both aspects of color selection and overall image quality.Moreover,we conduct ablation experiments,including training a single-stage model;that is,coloring an image without generating a grayscale image first and using only single-scale in the second stage.The performance of the single-stage model is the poorest,proving that learning automatic colorization directly is an extremely difficult task.The performance of the single-scale model is poorer than that of the multi-scale model,verifying the effectiveness of the multiscale strategy.Conclusion Experimental results on three datasets show that the proposed automatic sketch colorization method based on finite color space prior can colorize sketch effectively and easily generate various colorful results by changing color prior.