Pinterest’s Visual Lens: How computer vision explores your taste

Pinterest’s Visual Lens: How computer vision explores your taste

In the world of visual discovery engines, Pinterest has been at the forefront of using machine learning techniques to solve challenging problems. From recommending interests to generating engaging home feeds, Pinterest has leveraged machine learning to enhance user experience.

The critical moment for Pinterest came in January 2015 when it acquired Kosei, a machine learning startup specializing in recommender systems. This acquisition brought expertise in identifying visual similarities, categorizing and curating content, predicting user engagement, and prioritizing local tastes.

One of the key capabilities that machine learning brings to Pinterest is the ability to identify visual patterns and match them to other photos. This goes beyond simply determining the subject of an image and allows Pinterest to offer suggestions based on visual similarities. For example, if a user pins a mid-century dining-room table, the platform can suggest other objects from the same era.

Pinterest also pays close attention to a user's tastes and habits, considering what they have pinned and when. This allows the platform to surface more personalized recommendations, taking into account an individual's preferences.

Another interesting aspect of Pinterest's approach is its focus on local tastes. The recommendation engine has been trained to suggest popular content from a user's local region in their native language. This localization adds an extra layer of personalization to the user experience.

But Pinterest's use of computer vision doesn't stop at images. The platform also analyzes captions from previously pinned content and looks at which items get pinned to the same virtual boards. This allows Pinterest to link related items, such as a dress and a pair of shoes that are frequently pinned together, even if they don't look similar.

In 2014, Pinterest set its sights on visual search and acquired VisualGraph, an image-recognition startup. This marked the beginning of Pinterest's computer vision team, which has been working on developing new features and improving the platform's visual search capabilities.

One of the notable features that emerged from Pinterest's computer vision initiatives is Instant Ideas. This feature allows users to transform their home feed with similar ideas in just one tap. This seamless integration of visual search and recommendations enhances the user experience and makes it easier for users to discover new interests.

To achieve this level of functionality, Pinterest's visual search architecture is divided into two logical components. The first component focuses on understanding the query image, computing visual features, and detecting objects, colors, lighting, and image quality conditions. The second component, known as the blender, combines results from multiple sources, including visually similar results, scenes or projects with visually similar objects, and personalized text search results.

Pinterest conducted extensive experiments to fine-tune its computer vision capabilities. These experiments included benchmark datasets and A/B testing on Pinterest applications like Related Pins and similar looks. The goal was to show visually similar pin recommendations based on specific objects in an image, making it easy for users to start a discovery experience from any object.

Object detection plays a crucial role in Pinterest's visual search engine. By collecting labeled bounding boxes for regions of interest in images, Pinterest trains the system to identify objects that users are interested in. The real-time detection of objects in images, including unseen content from the web and users' cameras, is a vital aspect of Pinterest's visual search capabilities.

In addition to object detection, Pinterest also focuses on object search. This visual search system treats objects as the unit and finds visually similar objects in billions of images in a fraction of a second. This enables Pinterest to return scenes containing similar objects and enhances the visual discovery experience for users.

According to Pinterest's CEO, Ben Silbermann, the company is using computer vision to understand the aesthetic qualities of products and services, improve recommendations, and eventually create a camera tool for querying the world around us. Computer vision is the underlying technology that powers these three goals and enables Pinterest to help people discover and do things they love.

While Pinterest has made significant strides in utilizing computer vision for visual discovery, there are still areas for improvement. Here are three actionable pieces of advice for Pinterest:

1. Continuously refine the visual search capabilities: As computer vision technology evolves, Pinterest should stay at the forefront of advancements to provide users with the best visual search experience. This includes improving object detection accuracy, enhancing object-to-object matching, and expanding the range of searchable objects.

2. Foster user collaboration and contribution: To overcome the incentive misalignment between centralized platforms and users, Pinterest can incorporate mechanisms that reward users for creating and curating valuable information. By allowing users to share in the value generated from their work, Pinterest can create a more collaborative and mutually beneficial environment.

3. Embrace decentralized information networks: The rise of crypto mechanisms and tokenization presents an opportunity for Pinterest to explore decentralized information networks. By leveraging these technologies, Pinterest can enable coordination games that incentivize participants to cooperate and act together, ultimately leading to the production of valuable and legitimate information.

In conclusion, Pinterest's use of computer vision has revolutionized the way users discover and explore content on the platform. From identifying visual similarities to predicting user engagement, Pinterest has leveraged machine learning to enhance its recommendation engine and provide a personalized user experience. As the field of computer vision continues to evolve, Pinterest has the opportunity to further refine its visual search capabilities, foster user collaboration, and explore decentralized information networks to create a more dynamic and valuable platform for its users.