Learning Inclusion Matching for Animation Paint Bucket Colorization

S-lab, Nanyang Technological University
Teaser Image

Our project streamlines the animation colorization process by requiring painters to colorize just one frame, after which the algorithm autonomously propagates the color to subsequent frames. © drawn by Nicca (Sriprachum Kongwisawamit), used with artist permission.

Paint Bucket Colorization

Colorizing line art is a pivotal task in the production of hand-drawn cel animation. Digital painters use a paint bucket tool to manually color each segment enclosed by lines, based on RGB values predetermined by color designers. This frame-by-frame process, named Paint Bucket Colorization, is both arduous and time-intensive.


Abstract

Current automated colorization methods mainly focus on segment matching. This technique migrates colors from a reference to the target frame by aligning features within line-enclosed segments across frames. However, issues like occlusion and wrinkles in animations often disrupt these direct correspondences, leading to mismatches. In this work, we introduce a new learning-based inclusion matching pipeline, which directs the network to comprehend the inclusion relationships between segments rather than relying solely on direct visual correspondences. Our method features a two-stage estimation module that integrates a coarse color warping module with an inclusion matching module, enabling more nuanced and accurate colorization.

To facilitate the training of this network, we have developed a unique dataset, referred to as PaintBucket-Character. This dataset includes rendered line arts alongside their colorized counterparts, featuring various 3D characters. Extensive experiments demonstrate the effectiveness and superiority of our method over existing techniques.

Dataset

Video


Visual Comparison

Result based on previous frame's ground truth

Compared with previous Cadmium application and AnimeRun, our method can achieve more robust colorization results.
© Nicca

Loading...

Color propagation result based on the first frame

By propagating the color from the previous frame, our method can colorize a frame sequence based on the given first frame.
© Clip Studio Paint

Loading...

More results




Failure cases

Large deformation

Our method still struggles with exaggerated motion or large deformation, such as the moment when the robot hand clenches into a fist.
© Clip Studio Paint and Mecha-Ude

Occlusion

Our method necessitates the presence of all segments from the target frame in the reference frame, making it difficult to colorize new segments in the scenes like turning around.
© Kyoto Animation


BibTeX

@article{InclusionMatching2024,
  title     = {Learning Inclusion Matching for Animation Paint Bucket Colorization},
  author    = {Dai, Yuekun and Zhou, Shangchen and Li, Qinyue and Li, Chongyi and Loy, Chen Change},
  journal   = {CVPR},
  year      = {2024},
}