Sketch-based Creativity Support Tools using Deep Learning
Sketching is a natural and effective visual communication medium commonly
used in creative processes. Recent developments in deep-learning models
drastically improved machines' ability in understanding and generating visual
content. An exciting area of development explores deep-learning approaches used
to model human sketches, opening opportunities for creative applications. This
chapter describes three fundamental steps in developing deep-learning-driven
creativity support tools that consumes and generates sketches: 1) a data
collection effort that generated a new paired dataset between sketches and
mobile user interfaces; 2) a sketch-based user interface retrieval system
adapted from state-of-the-art computer vision techniques; and, 3) a
conversational sketching system that supports the novel interaction of a
natural-language-based sketch/critique authoring process. In this chapter, we
survey relevant prior work in both the deep-learning and
human-computer-interaction communities, document the data collection process
and the systems' architectures in detail, present qualitative and quantitative
results, and paint the landscape of several future research directions in this
exciting area.
Authors
Forrest Huang, Eldon Schoop, David Ha, Jeffrey Nichols, John Canny