Shutterstock, Inc.
User drawing based image search

Abstract:

Various aspects of the subject technology relate to systems, methods, and machine-readable media for user drawing based image search. These aspects include an image retrieval system using a convolutional neural network trained to identify how users draw semantic concepts and using an image search engine to search against images having a similar concept. The aspects include mapping between concepts of the user drawing space and concepts of the image space such that images associated with the same concept are identified. For each input user drawing, the drawing is first processed through a concept classifier to identify a corresponding concept, and then through a feature extractor to form a corresponding feature vector. The results from the concept classifier and the feature extractor may be combined to search against a collection of images having a similar concept to determine a listing of images ranked by visual and semantic similarity to the input.