"As a visual discovery engine, Pinterest has many challenging problems that can be solved using machine learning techniques:"

"What interests shall we recommend to a new user?"

"How to generate an engaging home-feed?"

"How do pins relate to each other?"

"What interests does a pin belong to?"

"The critical moment happened in January 2015, when Pinterest acquired Kosei — a machine learning startup with expertise in recommender systems."

"Identifying Visual Similarities: Machine learning can not only determine the subject of an image, it can also identify visual patterns and match them to other photos."

"Categorizing and Curating: If a user pins a mid-century dining-room table, the platform can now offer suggestions of other objects from the same era."

"Predicting Engagement:"

"Pinterest pays more attention to an individual’s tastes and habits — what they’ve pinned and when — enabling the site to surface more personalized recommendations."

"Prioritizing Local Taste:"

"Its recommendation engine has learned to suggest popular content from users’ local region in their native language."

"Going Beyond Images:"

"Pinterest also looks at captions from previously pinned content and which items get pinned to the same virtual boards. That allows Pinterest to, say, link a particular dress to the pair of shoes frequently pinned alongside it, even if they look nothing alike."

"Pinterest set its sights on visual search in 2014. That year, the company acquired VisualGraph, an image-recognition startup, and established its computer vision team with a small group of engineers and began to show its work."

"Instant Ideas is a way to transform users’ home feed with similar ideas in just a tap."

"1 — Lens Architecture:"

"Lens combines Pinterest’s understanding of images and objects with its discovery technologies to offer Pinners a diverse set of results."

"To do this, Lens’ overall architecture is separated into two logical components."

"The first component is a query understanding layer where Pinterest derives information regarding the given input image."

"Here Pinterest computes visual features such as detecting objects, computing salient colors and detecting lighting and image quality conditions."

"The second component is Pinterest’s blender, as the results Lens returns come from multiple sources."

"Pinterest uses visual search technology to return visually similar results, object search technology to return scenes or projects with visually similar objects and image search which uses the derived annotations to return personalized text search results that are semantically (not visually) relevant to the input image."

"It’s the job of the blender to dynamically change blending ratios and result sources based on the information derived in the query understanding layer."

"Lens results aren’t strictly visually similar, they come from multiple sources, some of which are only semantically relevant to the input image. By giving Pinners results beyond visually similar, Lens is a new type of visual discovery tool that bridges real-world camera images to the Pinterest taste graph."

"Pinterest conducted a comprehensive set of experiments using a combination of benchmark datasets and A/B testing on two Pinterest applications, Related Pins and an experiment with similar looks."

"In particular, the experiment with similar looks allowed Pinterest to show visually similar Pin recommendations based on specific objects in a Pin’s image."

"Since an image can contain dozens of objects, Pinterest’s motivation was to make it as simple as possible to start a discovery experience from any of them."

"Object detection in visual search also enables new features, like object-to-object matching."

"Pinterest’s first challenge in building automatic object detection was collecting labeled bounding boxes for regions of interest in images as our training data."

"By aggregating this activity across the millions of images with the highest engagement, it learns which objects Pinners are interested in."

"Since Pinterest’s visual search engine can use any image as a query — including unseen content from the web and even your camera — detection must happen in real-time, in a fraction of a second."

"First, it identifies regions of an image that are likely to contain objects of interest by running a fully convolutional network over the input image to produce a feature map."

"For each location on the feature map, the network considers a fixed set of regions, varying in size and aspect ratio, and uses a binary softmax classifier to determine how likely each of these regions is to contain an object of interest."

"If a promising region is found, the network also outputs adjustments to this region so that it better frames the objects."

"These systems index global image representations to return images similar holistically to the given input image."

"Object search is a visual search system that treats objects as the unit. Given an input image, Pinterest finds the most visually similar objects in billions of images in a fraction of a second, map those objects to the original image and return scenes containing the similar objects."

"According to Pinterest’s CEO, Ben Silbermann, the company is doing 3 things with computer vision."

"They try to understand what are the aesthetic qualities of a product or a service, so they can do better recommendations in general."

"They want to be able to look inside an image that has multiple items, zoom in on part of it and computationally say, “Hey, it’s this type of object."

"Here’s where you can find something similar.” Then, eventually, they want to make the camera tool that you can query the world around you. Computer vision is the fundamental technology that powers all 3 of these things."

"To help people discover and do things that they love."