Pinterest’s Visual Lens: How computer vision explores your taste

Pinterest’s Visual Lens: How computer vision explores your taste

Visual discovery engine Pinterest has been utilizing machine learning techniques to solve various challenging problems. With the acquisition of machine learning startup Kosei in 2015, Pinterest gained expertise in recommender systems, allowing them to recommend interests to new users and generate engaging home-feeds.

One of the key applications of machine learning in Pinterest is the identification of visual similarities. Machine learning not only helps determine the subject of an image but also identifies visual patterns and matches them to other photos. This enables Pinterest to offer suggestions of other objects from the same era when a user pins a specific item, such as a mid-century dining-room table.

Another area where machine learning plays a crucial role is in predicting engagement. By analyzing a user's pinned content and their habits, Pinterest can surface more personalized recommendations. This personalized approach allows Pinterest to pay more attention to an individual's tastes and preferences.

Additionally, Pinterest's recommendation engine has learned to prioritize local taste. It suggests popular content from users' local regions in their native language, making the platform more relevant and appealing to users worldwide.

Pinterest's exploration of computer vision goes beyond just images. The platform also looks at captions from previously pinned content and the items pinned to the same virtual boards. This allows Pinterest to link different items together, even if they look nothing alike. For example, a particular dress can be linked to a pair of shoes frequently pinned alongside it, based on user preferences.

In 2014, Pinterest set its sights on visual search and acquired image-recognition startup VisualGraph. With the establishment of its computer vision team, Pinterest began showcasing its work, including the introduction of Instant Ideas. This feature transforms users' home feeds with similar ideas with just a tap.

To power visual search, Pinterest developed Lens, which combines image understanding and object recognition technologies. Lens provides Pinners with a diverse set of results by extracting information from input images and blending multiple sources to offer visually similar and semantically relevant recommendations. Lens acts as a bridge between real-world camera images and the Pinterest taste graph.

Pinterest conducted extensive experiments to refine its visual search capabilities. These experiments involved benchmark datasets and A/B testing on different Pinterest applications. The objective was to show visually similar Pin recommendations based on specific objects in an image. Object detection and search play a crucial role in these experiments, allowing users to start a discovery experience from any object in an image.

Pinterest's CEO, Ben Silbermann, highlighted three main areas where computer vision is being utilized. Firstly, understanding the aesthetic qualities of products and services to improve recommendations. Secondly, being able to zoom in on specific objects within an image and find similar items. Finally, the goal of creating a camera tool that can query the world around you. Computer vision is the underlying technology powering these initiatives.

Beyond Pinterest, businesses like Stitch Fix have also embraced the power of computer vision. Stitch Fix uses recommendation algorithms to personalize apparel, and they have experimented with DALL-E 2, a technology that generates images based on specific characteristics like color, fabric, and style. This allows Stitch Fix to visualize its products and match them with similar items in its inventory.

DALL-E 2 has also been used by individuals to create logos and by platforms like Unstock.ai to provide high-quality images and illustrations on demand. The potential of DALL-E 2 is immense, and as it becomes more accessible in terms of cost and speed, its applications will only continue to expand.

In conclusion, Pinterest's adoption of computer vision has revolutionized visual discovery on its platform. Machine learning techniques have enhanced recommendation systems, identified visual similarities, and personalized content for users. The introduction of Lens and visual search has further improved the user experience, allowing for seamless exploration and discovery. As computer vision technology continues to advance, we can expect more innovative applications across various industries, from fashion to graphic design and beyond.

Actionable Advice:

1. Leverage the power of visual discovery: Incorporate machine learning and computer vision techniques to enhance recommendation systems and provide personalized experiences for users.
2. Explore the potential of visual search: Implement visual search capabilities to allow users to discover similar items and explore related content based on specific objects within images.
3. Embrace computer vision in your business: Utilize computer vision technology to improve product visualization, generate high-quality images, and enhance user experiences.