Automated and unsupervised generation of real-world training data

Abstract

The technology disclosed uses a combination of an object detector and an object tracker to process video sequences and produce tracks of real-world images categorized by objects detected in the video sequences. The tracks of real-world images are used to iteratively train and re-train the object detector and improve its detection rate during a so-called “training cycle”. Each training cycle of improving the object detector is followed by a so-called “training data generation cycle” that involves collaboration between the improved object detector and the object tracker. Improved detection by the object detector causes the object tracker to produce longer and smoother tracks tagged with bounding boxes around the target object. Longer and smoother tracks and corresponding bounding boxes from the last training data generation cycle are used as ground truth in the current training cycle until the object detector's performance reaches a convergence point.

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 62/554,424, entitled “AUTOMATED AND UNSUPERVISED GENERATION OF REAL-WORLD TRAINING DATA”, filed on Sep. 5, 2018, which is incorporated by reference for all purposes as if fully set forth herein.

U.S. Nonprovisional patent application Ser. No. 15/295,926, entitled “Visual Interactive Search, Scalable Bandit-Based Visual Interactive Search And Ranking For Visual Interactive Search”, filed on Oct. 17, 2016, which is incorporated by reference for all purposes as if fully set forth herein;

U.S. Nonprovisional patent application Ser. No. 15/295,930, entitled “Selection Of Initial Document Collection For Visual Interactive Search”, filed on Oct. 17, 2016, which is incorporated by reference for all purposes as if fully set forth herein;

U.S. Nonprovisional patent application Ser. No. 15/373,897, entitled “Bayesian Visual Interactive Search”, filed on Dec. 9, 2016, which is incorporated by reference for all purposes as if fully set forth herein; and

U.S. Nonprovisional patent application Ser. No. 15/619,299, entitled “Content Embedding Using Deep Metric Learning Algorithms”, filed on Jun. 9, 2017, which is incorporated by reference for all purposes as if fully set forth herein.

FIELD OF THE TECHNOLOGY DISCLOSED

The technology disclosed relates to automated and unsupervised generation of real-world training data, and in particular, relates to training convolutional neural networks (CNNs) using automatically generated real-world training images.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to like parts throughout the different views. Also, the drawings are not necessarily to scale, with an emphasis instead generally being placed upon illustrating the principles of the technology disclosed. In the following description, various implementations of the technology disclosed are described with reference to the following drawings, in which:

FIG. 1 shows one implementation of training an object detector.

FIG. 2 is a block diagram that illustrates various aspects of the technology disclosed.

FIG. 3 depicts one implementation of training a triplet network using tagged real-world training data.

FIG. 4 illustrates one implementation of using the trained triplet network to embed catalog data (images) in a catalog embedding space.

FIG. 5 shows one implementation of using the trained triplet network to improve the “Shop the Look” service.

FIG. 6 is a simplified block diagram of a computer system that can be used to implement the technology disclosed.

DESCRIPTION

Regarding “real-world data,” one of the most challenging aspects of generating training data is that the training data should resemble an underlying distribution of “real-world data.” “Real-world data” is data that is similar to what a user is trying to match when a user is presented with documents or images on a screen.

Roughly described, the technology disclosed relates to an overall process of providing a service using a trained model. The trained model uses algorithms for generating predictions in the form of images and/or screens that are believed to draw the customer to their target image (e.g., an image in their mind that they are trying to reach, such as a specific product). The images and/or screens are produced using embeddings created by the trained model.

The outcome of the service is only as good as the trained model. Use of better or more comprehensive training data allows for the creation of a better (e.g., more accurate or realistic) model, because the model is only as “smart” as the data that was used for training. This is why it is important to improve the training data generation process. Training data should satisfy two important aspects—(i) comprehensiveness, i.e., having richly tagged real-world images that are captured in a wide spectrum of uncontrolled environments (e.g., arbitrary poses, textures, backgrounds, occlusion, illumination) so that the model is proficient at handling a diverse array of image requests from the customers during production and (ii) scale, i.e., having large amounts of such tagged real-world images so that the model is adequately trained. There exists a shortage of such training data because colleting and tagging real-world images is tedious, time consuming, and error prone.

To overcome these difficulties, an automatic and unsupervised framework is proposed for collecting and labelling real-world images in real-world video sequences. The proposed framework uses a combination of an object detector and an object tracker to process video sequences and produce tracks of real-world images categorized by objects detected in the video sequences. The tracks of real-world images are used to iteratively train and re-train the object detector and improve its detection rate during a so-called “training cycle”. Each training cycle of improving the object detector is followed by a so-called “training data generation cycle” that involves collaboration between the improved object detector and the object tracker. Improved detection by the object detector causes the object tracker to produce longer and smoother tracks tagged with bounding boxes around the target object. Longer and smoother tracks and corresponding bounding boxes from the last training data generation cycle are used as ground truth in the current training cycle until the object detector's performance reaches a convergence point (or some other termination condition is achieved).

Over time, longer and smoother tracks and corresponding bounding boxes are collected and stored in a tagged real-world training database with minimal or no manual intervention. This database is then used to train a convolutional neural network (CNN)-based model (also called “embedding model”) that learns an embedding function for image retrieval. The trained model is further used to generate embeddings for catalog images so they can populate an embedding space. In one implementation, the embedding model can be trained using a triplet network architecture. The triplet network can include three convolutional neural networks that share weights and are branched in parallel. The convolutional neural networks are the underlying neural networks of the embedding model. The triplet network, and thus the underlying convolutional neural networks, are trained by making the embedding produced by each convolutional neural network enforce a metric defined at the image level, i.e., if image B is closer to image A than image C, then embedding B should be closer to embedding A than embedding C.

When inputs can be grouped into categories and there are several examples in each category, the embeddings can be learned using a classification task. The goal of the classification task is to put, as close as possible, embeddings of inputs of the same category. To operationalize the classification task, the embedding model can be augmented to include one or more fully-connected layers and a terminal softmax classification layer.

During production, the trained model produces a real-time embedding of a user selected image. The real-time embedding is then compared to embeddings of catalog images in the embedding space to produce a ranked set of catalog images that are presented as suggestions to the user.

In one use case, the technology disclosed uses the framework to improve the “Shop the Look” service. With Shop the Look service, users search the Internet for a product (e.g., an outfit like “cocktail dress” or furniture piece like “echelon round side table”) and the search platform pulls in images that match the product description. Once users tap on one of those images, the search platform suggests a gallery of exact (or visually similar) items featured in the selected image, as well as a link to purchase them. More information about the Shop the Look service can be found at “https://adwords.googleblog.com/2016/09/shop-look-on-google.html”, which is incorporated by reference for all purposes as if fully set forth herein.

Typically, in the Shop the Look service, the user selected image is a real-world image (“wild” image) and the suggested gallery includes synthetic images that are taken in a controlled and constrained environment. For example, the wild image is generally of a person wearing a dress and the suggested gallery is composed of catalog images of the dress and other related accessories taken in a studio and subjected to post-production edits (e.g., highlights, exposure, shadows, clarity, contrast, vibrance, saturation). The catalog images may or may not include a person.

The Shop the Look service uses an object detector to detect the product depicted by the user selected wild image and uses a ranking system to suggest and rank the catalog images (which are stored in a catalog database) relative to the wild image. Suggestion and ranking in this context is a challenge. During learning, the ranking system is trained on synthetic catalog images (e.g., with a white background, neutral pose, and frontal illumination). But, during production, the ranking system is tasked with mapping wild images to catalog images. It is very difficult for such a system to achieve a good ranking because of the significant difference between the user selected wild images and the controlled images in the catalog database.

The technology disclosed improves the object detector and the ranking system of the Shop the Look service by training them over a greater and more diverse quantity of real-world product images produced by its automatic and unsupervised framework discussed above. In other use cases, the technology disclosed can be applied to additional or different services, such as the ones involving different products like cars, smartphones, and so on.

FIG. 1 shows one implementation of training 100 an object detector 104. One example of an object detector is a convolutional neural network (CNN)-based single shot multibox detector (SSD) described in W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed, “SSD: Single shot multibox detector,” arXiv:1512.02325, 2015, which is incorporated by reference for all purposes as if fully set forth herein. Object detector 104 is trained on controlled training data 102 that is mostly composed of synthetic images. The training includes forward pass that produces an output 106 and backpropagation of error that updates that the weights of the object detector 104 and improves its detection rate on controlled images.

FIG. 2 is a block diagram 200 that illustrates various aspects of the technology disclosed. FIG. 2 comprises two modules—detector training module 206 and training data generation module 210. Detector training module 206 implements the training cycle and training data generation module 210 implements the training data generation cycle. Different from FIG. 1, in FIG. 2, the object detector 104 is “in-training” and receives as input real-world training data 202. In one implementation, training data 202 is composed of real-world video sequences.

In the initial training cycle, because the objector detector 104 was trained on controlled training data 102, its detection rate on the real-world video frames is not very high. For example, in a video sequence depicting a fashion show, a small variation in the clothing shape may cause the object detector 104 to make missing detections, false detections, or non-accurate bounding boxes. This results in short and inaccurate tracks of video frames whose utility in the creation of comprehensive and scaled real-world training data is very limited.

To overcome this problem, the disclosed algorithm uses a combination of object detector 104 and object tracker 208 to produce long and smooth tracks of real-world video frames. Each frame of a video sequence is processed by the object detector 104 and the object tracker 208. Object detector 104 detects a target object in a frame and creates a bounding box around the detected target object. Object tracker 208 determines whether and where the target object detected by the object detector 104 appears in successive frames and creates bounding boxes around the tracked target object.

One example of the object tracker 208 is a fully convolutional siamese network described in L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” arXiv:1606.09549, 2016, which is incorporated by reference for all purposes as if fully set forth herein. For additional examples of the object tracker 208, reference can be made to W. Luo, J. Xing, X. Zhang, X. Zhao, and T.-K. Kim, “Multiple object tracking: A literature review,” arXiv:1409.7618, 2014; A. Milan, L. Leal-Taixe, I. Reid, S. Roth, and K. Schindler, “Mot16: A benchmark for multi-object tracking,” arXiv:1603.00831, 2016; and A. Sadeghian, A. Alahi, and S. Savarese, “Tracking the untrackable: Learning to track multiple cues with long-term dependencies,” arXiv:1701.01909, 2017, which are incorporated by reference for all purposes as if fully set forth herein.

When the tracked target object disappears, the object tracker 208 terminates the tracking and interleaves, in a new track, all the frames that contained the target object. The interleaved frames are then tagged with corresponding bounding boxes and stored as tagged real-world training data 212 on a track-by-track basis. In one implementation, the tracks are categorized based on the type of the target object contained in the constituent frames (e.g., dress, skirt, pant).

Accordingly, in one implementation, the training data generation cycle comprises the following steps—(1) providing real-world training data 202 (video sequences) to the object detector 104 and producing frames with bounding boxes around the detected objects, (2) providing the frames and the bounding boxes to the object tracker 208 and producing additional frames with bounding boxes around the tracked objects, and (3) storing pairs of additional frames and corresponding bounding boxes on a track-by-track basis as the tagged real-world training data 212.

The tagged real-world training data 212 is then used in the next training cycle as input to the object detector 104. Note that, in training data generation cycle, the input to the object detector 104 is the real-world training data 202 (video sequences). In contrast, in training cycle, the input to the object detector 104 is the tagged real-world training data 212. That is, the object detector 104 is re-trained using the most recent version of the tagged real-world training data 212 produced in the previous training data generation cycle to produce output 204.

After the current training cycle ends, the improved object detector 104 is used to execute the first step of the following training data generation cycle, as discussed above. This continues until the performance of the object detector 102 reaches a convergence point (or some other termination condition is achieved).

Thus, the diversity (comprehensiveness) and size (scale) of the tagged real-world training data 212 enhances with each training data generation cycle because the improved object detector 104 makes more frequent and accurate detections and in turn enables the object tracker 208 to produce longer, smoother, and more continuous tracks of real-world video frames tagged with corresponding bounding boxes.

Some implementations include removing noise from the tracks using human input before feeding them to the object detector 104 for re-training. Some implementations also include using a subset of frames from a track to re-train the object detector 104. Some implementations also include using human input to select the area to track in a video sequence and to determine when to stop tracking.

The enhanced tagged real-world training data 212 improves the performance of the object detector 104 in the subsequent training cycle, with the corresponding bounding boxes serving as ground truth. A control module 214 iterates the training cycle and the training data generation cycle.

In one implementation, the object detector 104 is re-trained using just those frames and corresponding bounding boxes that the object detector 104 was unable to detect in the previous training data generation cycle and that were instead tracked by the object tracker 208.

Upon convergence (or termination) in FIG. 2, the most recent version of the tagged real-world training data 212 is used to train a model that generates embeddings, i.e., an embedding model. The embedding model can be trained using a triplet network approach and/or by adding classification layers on top of it and using a classification approach.

Thus, the embedding model can be trained using one specific task (e.g., triplet or classification) or trained using a multi-tasks approach by combining multiple tasks. For example, one epoch over two, the embedding model can be trained using the triplet approach such that the underlying convolutional neural networks of the embedding model are part of the triplet network architecture and, one epoch over two, the same embedding model can be trained using a classification approach such that the underlying convolutional neural networks of the embedding model are part of the classification network.

FIG. 3 depicts one implementation of training 300 a triplet network 302 using tagged real-world training data 212. As previously discussed, upon convergence (or termination) in FIG. 2, the most recent version of the tagged real-world training data 212 is used to train the triplet network 302. In one implementation, the training includes providing three inputs to the triple network 302, i.e., two frames or patches from the same track and one frame or patch from another track (e.g., from a different video sequence). This way, the triplet network 302 learns to generate embeddings (e.g., output 304) such that embeddings of frames from the same track are similar or closer to each other than to embedding of the frame from the another track.

In other implementations, the most recent version of the tagged real-world training data 212 is used to train different or additional networks such as a classification network that includes multiple fully-connected layers and a terminal softmax classification layer. Such a classification network can be tasked with classifying tracks and patches of the tracks into categories (e.g., dress clothing, pant clothing, roundtable furniture). A cross-entropy loss function can be used to evaluate the performance of the classification network over the classification task and calculate gradients for backpropagation. The trained classification network produces embeddings.

In some implementations, the triple network 302 is also trained using catalog data (images) 402. Additional information about the triplet network 302 and training the triplet network 302 can be found in commonly owned U.S. patent application Ser. No. 15/295,926; Ser. No. 15/295,930; Ser. No. 15/373,897; and Ser. No. 15/619,299, which are incorporated by reference for all purposes as if fully set forth herein.

FIG. 4 illustrates one implementation of using 400 the trained triplet network 302 to embed catalog data (images) 402 in a catalog embedding space 406 as catalog embeddings 404. Additional information about using the trained triplet network 302 to generate the catalog embedding space 406 and about the catalog embedding space 406 can be found in commonly owned U.S. patent application Ser. No. 15/295,926; Ser. No. 15/295,930; Ser. No. 15/373,897; and Ser. No. 15/619,299, which are incorporated by reference for all purposes as if fully set forth herein.

FIG. 5 shows one implementation of using 500 the trained triplet network 302 to improve the “Shop the Look” service. At production, a user 502 selects a real image (wild image) 504. In response, the trained triplet network 302 produces a real-time image embedding 506 of the real image 504. Following this, a ranking module 508 compares the real-time image embedding 506 to embeddings of catalog images in the catalog embedding space 406 and produces a ranked set of catalog images 510 that are presented as suggestions 514 to the user 502 by a harvesting module 512. Additional information about exploring and exploiting an embedding space during production to produce image and/or screen suggestions in response to a user selection or user selected image and/or screen can be found in commonly owned U.S. patent application Ser. No. 15/295,926; Ser. No. 15/295,930; Ser. No. 15/373,897; and Ser. No. 15/619,299, which are incorporated by reference for all purposes as if fully set forth herein.

The following is a sample pseudo code of one implementation of the disclosed algorithm:

training_data = customer_training_data
validation_data = customer_validation_data
cloth_detector = build_cloth_detector(training_data)
tracker = build_tracker(cloth_detector)
cloth_detector_performance =
cloth_detector.evaluate(validation_data)
last_cloth_detector_performance = 0
while not has_converged(last_cloth_detector_performance,
cloth_detector_performance)
last_cloth_detector_performance = cloth_detector_performance
all_video_tracks = [ ]
For video in all_videos:
video_tracks = tracker .generate_all_tracks(video)
all_video_tracks.add_all(video_tracks)
additional_training_data = process_tracks(all_video_tracks)
training_data = [training_data, additional_training_data]
cloth_detector = build_cloth_detector(training_data)
tracker = build_tracker(cloth_detector)
cloth_detector_performance =
cloth_detector.evaluate(validation_data)
ranking_training_data =
process_tracks_for_ranking(all_video_tracks)
ranker = build_ranker(ranking_training_data)

FIG. 6 is a simplified block diagram 600 of a computer system that can be used to implement the technology disclosed and in particular the training data generation module 210. Computer system 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via bus subsystem 612. These peripheral devices can include a storage subsystem 624 including, for example, memory devices and a file storage subsystem, user interface input devices 622, user interface output devices 620, and a network interface subsystem 616. The input and output devices allow user interaction with computer system 610. Network interface subsystem 616 provides an interface to outside networks, including an interface to corresponding interface devices in other computer systems.

User interface input devices 622 can include a keyboard; pointing devices such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into the display; audio input devices such as voice recognition systems and microphones; and other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computer system 610.

User interface output devices 620 can include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem can include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem can also provide a non-visual display such as audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computer system 610 to the user or to another machine or computer system.

Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules and methods described herein. These software modules are generally executed by processor 614 alone or in combination with other processors.

Memory 626 used in the storage subsystem can include a number of memories including a main random access memory (RAM) 630 for storage of instructions and data during program execution and a read only memory (ROM) 632 in which fixed instructions are stored. A file storage subsystem 628 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations can be stored by file storage subsystem 628 in the storage subsystem 624, or in other machines accessible by the processor.

Bus subsystem 612 provides a mechanism for letting the various components and subsystems of computer system 610 communicate with each other as intended. Although bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem can use multiple busses.

Computer system 610 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 610 depicted in FIG. 6 is intended only as one example. Many other configurations of computer system 610 are possible having more or fewer components than the computer system depicted in FIG. 6.

Claims

1. A method of automated and unsupervised generation of real-world training data, the method including:

processing one or more real-world video sequences through a trained object detector to produce video frames of detected objects and corresponding bounding boxes;

providing the video frames of detected objects and the corresponding bounding boxes to an object tracker to produce additional video frames of tracked objects and corresponding bounding boxes;

storing pairs of the additional video frames of tracked objects and the corresponding bounding boxes as tagged real-world training data;

using the tagged real-world training data to re-train the object detector; and

iterating the processing, the providing, the storing, and the using until convergence to a final set of tagged real-world training data;

training an embedding model with the final set of tagged real-world training data using a triplet network approach including providing three inputs to the triple network, wherein two inputs are frames from a first track and one input is a frame from as second track and further wherein the embedding network learns to generate embeddings such that the first track frames are more similar to each other than a frame from the second track;

wherein the trained embedding model embeds frames in an embedding space.

2. The method of claim 1, further comprising removing noise from the tagged real-world training data prior to using to re-train the object detector.

3. The method of claim 1, further comprising using a subset of the tagged real-world training data to re-train the object detector.

4. A method of automated and unsupervised generation of real-world training data, the method including:

processing one or more real-world video sequences through a trained object detector to produce video frames of detected objects and corresponding bounding boxes;

providing the video frames of detected objects and the corresponding bounding boxes to an object tracker to produce additional video frames of tracked objects and corresponding bounding boxes;

storing pairs of the additional video frames of tracked objects and the corresponding bounding boxes as tagged real-world training data;

using the tagged real-world training data to re-train the object detector; and

iterating the processing, the providing, the storing, and the using until convergence to a final set of tagged real-world training data;

training an embedding model with the final set of tagged real-world training data using a classification network, wherein the embedding model includes multiple fully-connected layers and a terminal softmax classification layer, further wherein the trained classification network classifies tracks into categories to produce the embeddings of the frames.

5. The method of claim 1, further comprising removing noise from the tagged real-world training data prior to using to re-train the object detector.

6. The method of claim 1, further comprising using a subset of the tagged real-world training data to re-train the object detector.

7. A method of automated and unsupervised generation of real-world training data, the method including:

processing one or more real-world video sequences through a trained object detector to produce video frames of detected objects and corresponding bounding boxes;

providing the video frames of detected objects and the corresponding bounding boxes to an object tracker to produce additional video frames of tracked objects and corresponding bounding boxes;

storing pairs of the additional video frames of tracked objects and the corresponding bounding boxes as tagged real-world training data;

using the tagged real-world training data to re-train the object detector; and

iterating the processing, the providing, the storing, and the using until convergence to a final set of tagged real-world training data;

training an embedding model with the final set of tagged real-world training data;

8. The method of claim 7, wherein the embedding model uses a triplet network approach including providing three inputs to the triple network, wherein two inputs are frames from a first track and one input is a frame from as second track, and further wherein the embedding network learns to generate embeddings such that the first track frames are more similar to each other than a frame from the second track.

9. The method of claim 7, wherein the embedding model is a classification network, wherein the embedding model includes multiple fully-connected layers and a terminal softmax classification layer, further wherein the trained classification network classifies tracks into categories to produce the embeddings of the frames.

10. The method of claim 7, further comprising removing noise from the tagged real-world training data prior to using to re-train the object detector.

11. The method of claim 7, further comprising using a subset of the tagged real-world training data to re-train the object detector.