"the “GLEe” model to articulate the product that establishes the company’s initial beachhead, how you might lead from there, along with how you will expand even further."

"A product vision enables teams to embrace a long-term mindset along with a somewhat flexible, step-by-step approach that helps the team navigate from where it is today to its hoped-for future."

"Get Big on _____________ Lead __________________ Expand into ____________"

"the product vision does not have to be “right.” The product vision is vague about the exact timing, and each step is actually a hypothesis. But it’s critical to share a product vision that helps investors and employees understand how big the opportunity may be, along with a rough plan of how you will eventually get there."

"The key is that all functions within a company are aware of the potential steps and agree on the level of investment for each stage at any moment in time. This helps the company to stay aligned"

"The key is for organizations to have shared language around the potential step-by-step product progression, along with a rough sense of the optimal level of investment at each stage."

"overcommunicate to keep everyone aligned."

"DHM model: Delight customers in Hard-to-copy, Margin-enhancing ways."

"What customers say doesn’t always match their behavior. You need A/B testing to measure behavior change."

"It’s helpful to understand how much customers value different features. We invested in things our members valued (broader DVD selection, lower prices, next-day DVD delivery). We invested less in features they didn’t value (new release DVDs, social, unique movie-finding tools)."

"We never established a more precise word-of-mouth factor, though I’ve heard that Amazon sometimes uses an 8X factor. A large WOM multiple encourages more investment in customer delight"

"The free trial reminder builds trust, which creates a more robust, world-class brand. Although Netflix loses $50 million, it builds a long-term advantage through its hard-to-copy brand."

"When you make product decisions, ask yourself if it’s a high or low-stakes decision. If the magnitude is high and it’s hard to reverse, consider the decision carefully. Give it plenty of time, and gather as much data as possible. If the stakes are low and the decision is easy to reverse, decide quickly, then move on. Most decisions by product leaders are low stakes (although they almost always believe the opposite). Postponing decisions creates ambiguity, which makes things worse. Be decisive."

"We continually A/B tested different positioning and branding approaches on the non-member homepage. We focused on two key metrics:"

"The headline provides an executive summary of the model. In cases where you develop an advertising campaign, it is the title of an ad campaign."

"Use clear, precise sixth-grade language. Consumers are busy and don’t have time to parse complicated ideas."

"1.) The positioning model describes the first three components of the brand definition: positioning, customer benefit, and personality."

"marketing defines the brand, and product brings the brand to life by building a great product."

"2.) The brand pyramid adds the two remaining elements: aspiration and emotion."

"1.) In simple terms, how do you describe your product or company?"

"2.) How does it benefit customers?"

"3.) How do you define its personality? (The question behind the question: how do you want your product to relate to customers?)"

"Be brief. Most teams begin with lots of complicated ideas. The key is to simplify and focus on a few, easy-to-communicate ideas."

"Defining the personality of your product describes how you want your brand to relate to customers."

"Why does the brand model incorporate emotion? The simple answer: it makes things memorable."

"Acknowledging emotion in the brand framework helps make products memorable."

"In the long-term, if you fully deliver the product and its emotional benefits to your customers, how might your product dent the universe? Think twenty years into the future."

"The “something bigger” was to deliver an entertainment experience that transported you to the magical place movies can take you — to escape reality."

"Think Big. Apple delivers revolutionary innovation, Nike enables customers to fulfill their full human potential, and SpaceX is working to save the human race by colonizing Mars. The “something bigger” is meant to inspire your team to build a great product and company over fifty years."

"The important thing is to take your best shot at defining the model, then explore the various ideas with consumers via focus groups, surveys, but most importantly, A/B tests."

"1.) Trials/Visitors. Historically, 2% of visitors to the non-member site started a free trial. 2.) Conversion from free to paid membership. At the end of a free trial, about 90% of customers converted to paid membership."

"Customers weren’t interested in sharing their movie tastes with friends, and when they did, their friends thought they had terrible taste!"

"“Today, brands are not the preserve of a marketing department. Brands are too important to be left to the marketing department — or any other ‘department,’ come to that. Organizational ghettoes do not create vibrant world-changing brands.” — Thomas Gad"

"Don’t let past investment inform future investment. Ask yourself, “Given what we know today, how much should we invest going forward?”"

"Small wins cloud judgment. The proxy metric kept going up and to the right, but Netflix failed to recognize early enough that it would never be big enough to matter."

"As a strategy, social makes sense. Friends promised to fulfill the holy troika: to delight customers in hard-to-copy, margin-enhancing ways."

"Friends failed to get big enough to improve retention, but the real question is, why did Netflix persist so long?"

"Why does Netflix fail so frequently? Two reasons: 1.) There are many biases that cloud human judgment, and 2.) it’s hard to invent the future."

"It’s hard to quit when the CEO is passionate about an idea."

"Netflix assumed the failure was in the execution, not in the idea."

"Paradoxically, small success makes it hard to kill features."

"Behold the idea and not the source."

"Temper your pride in ownership. As builders, companies love to build stuff. And nobody likes to kill projects. Passion and hope can cloud judgment."

"Establish clear objectives."

"Setting a goal guards against we’ll-figure-it-out-next-quarter youthful enthusiasm."

"Successful projects develop momentum quickly. Customers enthusiastically embrace good ideas despite early shortcomings."

"When engaged in a project, your humanity can cloud a disciplined evaluation of its merit. Take a moment to reevaluate, discounting both executive-level support and conventional wisdom. Remain objective by setting an objective goal, taking stock in your pride of ownership, and asking, “What should we invest in today, regardless of past investment decisions?”"

"Crisp execution and high-cadence experimentation are critical, but having a clear product strategy supercharges your efforts. Strategic thinking enables you to think ahead, to effectively “skip quarters,” and to build enduring value."

"I shifted my focus from satisfying customers to delighting them. I also learned about the balancing act of delight and margin, and what makes products hard-to-copy."

"Process power."

"An example: Netflix’s personalization technology. Because Netflix knows the movie tastes of 185M members, they can generate accurate forecasts of streaming hours for each potential title, and spend accordingly. They effectively “right-size” their investment in original content."

"Hamilton Helmer’s book, “7 Powers,” outlines seven hard-to-copy advantages."

"Product strategy answers the question, “How will your product delight customers, in hard to copy, margin-enhancing ways?”"

"Think about how your product delights customers, both now and in the future."

"Brand. Building trust with customers takes years of delivering value with a minimum of “trustbusters.”"

"Network effects."

"Economies of scale."

"Counter-positioning. This “power” — an offer to customers that is impossible for competitors to match — is rare."

"Unique technology."

"Switching costs. This hard-to-copy advantage exists when a customer invests so much in one product that it’s hard to switch to another."

"Captured resource. The clearest example of this power is a patent."

"The company was founded in 2001 as “Craigslist for college students” but didn’t get traction until it launched textbook rental in 2008."

"The GLEe Model"

"Our service provided overwhelming value to students who could rent a textbook for $50 instead of buying a new book for $200."

"Physical books offered advantages students valued that eTextbooks couldn’t match, most noticeably the ability to highlight text using a marker."

"The DHM Model"

"Enable students to buy and sell class notes."

"Create a “student graph.” We built a dataset of all courses on each campus. It included all the textbooks and content that were part of the course."

"Develop unique, personalization technology. We built this capability using the student graph data, above."

"Social (a Facebook-style newsfeed based on student graph data)"

"In sorting through the “DHM” ideas and experiments above, we teased out the following product strategies."

"we built a Facebook-style feed but got no traction."

"Dan Rosensweig, our CEO, was confident in our ability to raise capital and prioritized “GEM” as follows:"

"Growth, measured by year-over-year growth in student accounts and revenues."

"The Rolling 4-Quarter Product Roadmap"

"The brand, reinforced by the large orange boxes on campuses, did experience viral growth, however."

"We began by providing step-by-step solutions to all the answers in textbooks. Later, we expanded into other homework help areas — writing help, tutoring, flashcards — to create a monthly subscription service called “Chegg Study.”"

"The roadmap provided a relatively high-level view that demonstrated how the product strategies might play out over time."

"I outlined how Netflix hoped to “Get Big,” “Lead,” and “Expand” during its startup phase."

"The intent of the “GLEe model” is to encourage product leaders to think long-term and to think big."

"I’ve outlined four strategies that I think are most important."

"Personalization."

"Original content."

"A better watching experience."

"Interactive storytelling."

"The GEM Model"

"Here’s how the company might force-rank growth, engagement, and monetization today:"

"The SMT Lockup"

"But because retention is so hard to move, the product team needs a more sensitive proxy metric. I speculate that the “percentage of members who watch at least 40 hours/month” is a reasonable proxy"

"How to keep strategy “front and center” and accelerate both results & learning."

"My peers describe my communication style as “Lather, Rinse, Repeat,” which is a fair assessment."

"At Netflix, I began with monthly product strategy meetings for each swimlane, then evolved to a Quarterly Product Strategy meeting. Over time, this meeting became a mechanism for the Netflix culture."

"An essential part of the Netflix culture is to enable teams to become highly aligned and loosely coupled. “Highly aligned” means each group understands the overall product strategy and how they contribute to the company’s success."

"“Loosely coupled” means teams occasionally check in with each other but avoid the trap of ‘tight coupling’ — consulting multiple teams for every decision they make."

"Another Netflix principle is “context, not control.” The intent is to provide context through strategy so that focused teams make decisions without rules or heavy-handed process."

"Use slides, but don’t polish them. Slides are a good conversation starter, but you don’t want “death by Powerpoint.” The goal is to deliver crisp articulation of your strategy along with test results that inspire debate."

"Here is a rough outline for your first quarterly product strategy meeting:"

"Articulation of the high-level strategy by the head of the product team. Key team members can also share insights relevant to everyone in the room."

"The strategy for each swimlane, presented by each product leader."

"A wrap-up at the end of the session. This time provides an opportunity for general discussion, to debate unresolved issues, and to frame which information should be shared broadly outside the room."

"I have participants complete a Net Promoter Score survey to understand what went well and what could be better. The intent is to make each meeting better than the one before."

"Prioritizing growth, engagement, and monetization to keep organizations aligned."

"One of the biggest causes is differing opinions on how to prioritize growth, engagement, and monetization."

"The GEM model forces cross-functional teams to prioritize these factors and helps build a metrics-focused organization."

"GEM Prioritization"

"Monetization: As measured by Lifetime Value (LTV) and gross margin."

"Engagement: As measured by monthly retention. (Think of this as a proxy for product quality.)"

"Our priority was monetization to answer the question, “How can Netflix deliver a higher-margin business?”"

"At different times in a company’s life, the priorities change. It’s an excellent habit to reassess the priority of these three factors every six months or so."

"Thinking about the overall needs of your company, how do you prioritize growth, engagement, and monetization? Which metric will you use to measure each?"

"The GLEe model helps product leaders to form a product vision to address these criticisms."

"The model encourages teams to think big, think long-term, and describes a phased, step-by-step approach to build a product that “dents the universe.”"

"The GLEe model encourages teams to ask, “What’s next?” and forces long-term thinking. To invent the future, you have to be optimistic, and long-term thinking enables an “anything is possible” attitude. By long-term, I mean 3–5 years, and in this timeframe, most things are possible. But the GLEe model goes far beyond this to encourage product leaders to think 10–15 years ahead."

"By 2012, Netflix finally established a hard to copy network effect through its device ecosystem."

"The GLEe model also reinforces that you don’t have to do everything at once."

"As Netflix expanded globally, personalization helped connect members with movies they loved, regardless of nationality."

"Finding a high-level product strategy that supports multiple phases of a company’s growth is rare. But if you accomplish this, it builds a huge, hard to copy advantage."

"1) What is the initial product that enables the company and product to “Get Big” over the first 3–5 years of its life?"

"2) What will you “Lead?” Three to five years in the future, what is the next wave your product or company will ride — the equivalent of internet video for Netflix?"

"3) Once your product establishes a leadership position, how might it “Expand” even further?"

"It’s especially hard to guess the timing of projects, given this uncertainty."

"When I share a roadmap, I express confidence about the next quarter’s projects but highlight that the content and timing of the subsequent quarters are highly speculative."

"The roadmap articulates the focus and organization of the product team."

"The roadmap is an artifact — an expression — of your product strategy."

"The roadmap tells a story about how the overall strategy for the product team might play out over time."

"One note on organizing product teams: I favor structures where each product leader has a clear proxy metric that communicates what success looks like in their area."

"The percentage of new members who rate at least 50 movies in their first six weeks with the service."

"The theory was that if members were willing to rate lots of movies, it meant they valued the results of their rating: personalized movie choices."

"Given an in-depth knowledge of members’ movie tastes, and data about each movie, matching algorithms would connect members with personalized movie choices."

"Over time, Netflix moved the proxy metrics, and in the long-term, proved that a highly personalized experience improved retention."

"It took more than a decade to prove that personalization improved retention. A steady improvement in the proxy metrics, however, provided a strong signal that we would eventually succeed."

"Each product leader worked with a dedicated team of engineers, designers, and data analysts, and could articulate their high-level engagement metric, along with the key strategies, proxy metrics, and projects for their area."

"The key to the Strategy/Metric/Tactic lockup is to identify a high-level product strategy, assign a proxy metric to measure the strategy’s effectiveness, then brainstorm a set of projects that will move the metric."

"I often ask a team to list all the projects they believe are important, then sort the projects into “buckets” of ideas."

"The label for each of these buckets often indicates the implied strategy."

"reduce text and explanations"

"provide step-by-step guides for new members"

"eliminate unused features or functions"

"In contrast, personalization efforts at Netflix are an example of a top-down approach. We decided that the high-level hypothesis — creating a highly personalized experience will improve retention — then identified a proxy metric and brainstormed projects to move the metric."

"Look at the list of projects you believe are essential, then sort them to identify high-level themes."

"How to assign metrics and tactics to each product strategy."

"At Netflix, the high-level engagement metric — that defined the product team’s ability to make the product better — was monthly retention."

"Why do you need a proxy metric?"

"Improving retention is hard — the progress is almost glacial. You need more sensitive, lower-level proxy metrics to see if your strategies are working."

"Each of our product strategies had a clear proxy metric to determine if the strategy worked or not. And there were typically two to three projects — think of these as tactics — that brought the strategy to life."

"Over the years, we learned to double down in areas where we moved our proxy metrics and demonstrated retention improvements. We also learned to cut our losses when projects didn’t deliver."

"The next step is to tease out high-level hypotheses that combine delight, hard to copy advantage, and margin. Achieving two or three of these objectives with a single strategy is at the heart of an effective product strategy."

"Easy. Netflix wasn’t simple at launch but became simpler over time."

"A surprising example in 2018: the decision to remove movie reviews. Now that members can quickly hit “play” or “quit” at any time, they no longer need reviews."

"Social. Netflix experimented with “Friends” for six years."

"Unique movie-finding tools. In 2005, the vision was that personalized previews would begin to play on each member’s homepage, delivering both delight and margin."

"Price and Plans."

"Ads/Used DVD sales."

"Next-day DVD delivery."

"Streaming."

"Today, there’s hard to copy advantage in the technology that Netflix employs to encrypt and deliver video."

"Entertainment."

"Open APIs. In 2006, when Facebook, LinkedIn, and others opened their Application Programming Interfaces to enable partners to innovate on their platforms, Netflix followed suit."

"Device ecosystem."

"By 2012, Netflix had critical mass with hardware partners; they created a hard to copy network effect that also delights customers who enjoy watching “anytime, anywhere.”"

"Exclusive DVD content."

"Interactive stories."

"High-quality video & sound. One of the early lessons at Netflix was the importance of staying focused on making the core product better."

"When I join a company as a product leader, advisor, or board member, I brainstorm ideas to answer three questions:"

"1) How will the product delight customers?"

"2) What will make the product hard to copy?"

"3) What are the business model experiments required to build a profitable business?"

"What’s the list of “delighters” for your product?"

"Brand."

"Today, nearly all TVs, DVD/Blu-Ray players, game systems, set-top boxes, and mobile devices are pre-wired to stream Netflix."

"Counter-positioning."

"This “power” — an offer to customers that is impossible for competitors to match — is rare. In 2004, however, Netflix advertised “No Late Fees.”"

"They effectively “right-size” their investment in original content."

"Switching costs."

"To a small degree, Netflix customers don’t turn to Amazon or Hulu because it’s too much work to recreate profiles for each family member."

"Captured resource."

"How will your product generate margin?"

"You’ll need to experiment to evaluate different prices and business models over the life of your product. You’ll never be “done.”"

"Exercise 1: Take a moment to jot down how your product delights customers today, then add a few ideas about how you might delight them even more in the future."

"Exercise 2: Using the eight hard to copy powers above as a springboard, list ways your product might create a hard-to-copy advantage in the future."

"Exercise 3: For your product, list a few price and business model experiments you might explore over the next 1–3 years."

"At this point, you should have a list of hypotheses to delight your customers, create hard to copy advantage, and experiment with price/plan combinations and business models."

"Fast forward to 2005 when I joined Netflix. I shifted my focus from satisfying customers to delighting them. I also learned about the balancing act of delight and margin, and what makes products hard-to-copy. And I learned to articulate a product strategy — a set of hypotheses for how to delight customers in hard-to-copy, margin-enhancing ways."

"We had failed to build long-term, enduring value."

"The Good. I learned to accelerate progress by thinking strategically."

"Strategic thinking enables you to think ahead, to effectively “skip quarters,” and to build enduring value."

"To decrease these customer service calls, Netflix tested a free trial reminder. On day 28 of the free trial, Netflix sent texts and emails to new customers: “Your one-month free trial is about to end. Click here if you’d like to cancel.”"

"Over two decades, Netflix improved its subscribers’ monthly cancel rate from 10% to 2%. They did this by balancing delight against margin (profit) while building a durable, hard-to-copy advantage. I call it the DHM model: Delight customers in Hard-to-copy, Margin-enhancing ways."

"Rather than buying up a surplus of newly released DVDs to handle that short-term spike in demand, we decided to optimize for the long-term. As a result, during the first few months after a DVD launch, some customers had to wait a week or two to get the new release."

"If the new release wasn’t available when the customer requested it, we shipped the next movie on the customer’s movie list, then sent the new release when it became available."

"Will faster new release DVD delivery improve retention, and, if so, by how much?"

"If this experience improves retention, is it worth spending more money on DVD inventory?"

"Yes, faster new release delivery improved retention, but by much less than we expected."

"Here’s how we evaluated the balance between delight and margin:"

"The value of this perfect new release experience was $1 million: 5,000 saved customers x $100 (the lifetime value of a customer) X 2 for the WOM factor."

"But the cost to deliver that perfect new release experience was $5 million in additional DVD inventory. Given the potential gain of $1 million against a $5M spend, we chose not to roll the feature out to all members."

"It’s helpful to understand how much customers value different features. We invested in things our members valued"

"The result? A drop in paid conversion from 90% to 85%, resulting in a $50M loss. (The 5% decrease in conversion translated to a total loss of $60M, but the $10 million in “I forgot to cancel” customer service calls dropped to zero.)"

"A/B tests can’t measure all of these factors, which is why judgment and strategic frameworks like the DHM model (Delight customers in Hard-to-copy, Margin-enhancing ways) are critical."

"As a product leader, your job is to delight customers in hard-to-copy, margin-enhancing ways."

"as Netflix slowly learned, no one wants to reveal all the movies they have viewed. (“Did you really watch Adam Sandler’s The Ridiculous Six?”) But, even more critical, movie tastes are amazingly unique. Given this inherent diversity, reality eventually came into focus: Your friends have sucky movie taste."

"Why does Netflix fail so frequently? Two reasons:"

"1.) There are many biases that cloud human judgment,"

"2.) it’s hard to invent the future."

"Netflix members would delight in getting movie ideas from friends, they would build a hard-to-copy network effect, and movie suggestions would be cheaper as friends recommended long-tail movies to one another."

"But in 2010—I was at Chegg by that time—Netflix killed Friends because they recognized they needed at least 20 percent of members to engage."

"The focus on addressing challenges like this distracted Netflix from a more careful evaluation of the social strategy itself."

"Netflix needed to do a better job of evaluating the idea on its own merit."

"Temper your pride in ownership."

"Take off your blinders."

"Ignore sunk cost."

"Be open to better ideas."

"Kill projects properly."

"Consumer science—the process of forming and evaluating hypotheses through existing data, qualitative research, surveys, and A/B tests—requires finely tuned intuition."

"Here’s my definition of a brand:"

"3.) The evolution of Netflix shows how the non-member homepage — the “store window” of the site— evolved in tandem with the brand over twenty years."

"3.) How do you define its personality?"

"If you try this positioning model on your own, here are some tips:"

"Use clear, precise sixth-grade language."

"Be brief."

"In answering the “personality” question, ask yourself, “If someone met my company or product at a cocktail party, how would I like folks to describe him or her?”"

"In the long-term, what’s the one word you’d like to own for your product or company?"

"Product attributes:"

"What are the product features that deliver benefits to customers?"

"Product benefits:"

"how does your product improve customers’ lives?"

"Emotional benefits:"

"How does your product make customers feel?"

"Something bigger:"

"In the long-term, if you fully deliver the product and its emotional benefits to your customers, how might your product dent the universe?"

"Headline:"

"The headline provides an executive summary of the model."

"Product attributes. What are your product's features or components that consumers buy or use to enjoy the benefit of your product?"

"Product benefits. These are the same benefits from the first positioning model."

"Emotional benefits. If you built a world-class product that delivers the benefits you describe but ultimately exceeds customers’ expectations, how would customers feel?"

"Something bigger. Many teams struggle with this part of the exercise."

"The headline."

"In 2011, Netflix was also experimenting with Facebook Connect to sign-up and get movie recommendations from friends. This social approach eventually failed — twice. Customers weren’t interested in sharing their movie tastes with friends, and when they did, their friends thought they had terrible taste!"

"I hope your marketing and product teams will work together to apply both the positioning and brand pyramid models. The models will help you define a “true north” for your product and build a product that brings your brand to life."

"“David Cancel, CEO of Drift and one-time VP Product for Hubspot, called roadmaps a no-win scenario."

"""Either I’m going to disappoint you by giving you exactly what we thought six months ahead of time was the best solution when it’s not, or by changing course and having lied to you.”"

"An outcome-based roadmap describes the path to a successful product in terms of the problems to be solved to achieve the desired results."

"So if your roadmap is about problems to solve — which is the focus of outcomes-based roadmaps— it can be the steady light guiding you toward success.”"

"The challenge of a roadmap is it creates an expectation that the team will deliver the projects on a specific schedule. When I present a roadmap, I give the following “No Commitment” speech:"

"The roadmap helps the organization to understand how projects fit together and evolve."

"Outcome-based roadmaps provide a different approach to navigating the implied commitment of a roadmap but still let you tell a story about how things might play out over time."

"Project-based roadmaps present their hypotheses — product strategies — while outcome-based roadmaps express “the problem to be solved,” communicated using customer language."

"Outcome-based roadmaps reiterate the proxy metrics from my “SMT” model and position them as “leading indicators.” For both, the assumption is that the metrics are meaningful, but you don’t know exactly which projects will move them."

"Outcome-based roadmaps avoid a specific quarter-by-quarter plan and instead use the phrases, “Now, Next, & Consider.”"

"The result reinforces the importance of moving the “leading indicators,” irrespective of the timing or exact composition of each project. Most importantly, there’s no implied commitment to a timeline."

"Roadmaps are an expression of your product strategy. Whether you create a project or outcome-based roadmap, you need to do the heavy-lifting of figuring out your product strategy first."

"Roadmaps are a story-telling tool."

"Whichever you choose, make sure you do the hard work up-front to articulate your product strategies (problems to be solved), your metrics (leading indicators), then fill in the blanks of your roadmap with the projects you believe will move your metrics."

"NPS is like a compass — it’s directional. NPS doesn’t present a complete story."

"The fact that only 31% found NPS helpful doesn’t surprise me."

"I realized that NPS takes on almost mythical status as a “single source of truth” within some organizations because it presents a single number. It appears to be quantitative data, but it’s not. It’s a numerical representation of qualitative data."

"I execute surveys to capture who the customer is and how to think about them — by demographics, competitive product usage, entertainment preferences, etc. NPS is one tool in this survey toolkit."

"At Netflix, we collected NPS data but rarely looked at it. We had a much better proxy metric for product quality— the percentage of members who canceled each month. Monthly customer retention is a much better proxy metric for product quality than NPS. It measures what customers do, not what they say."

"At Chegg, we executed an NPS survey in 2010, and the NPS was 60, which is very good for a startup."

"Another fun fact: Netflix used NPS when the service launched in South America, and the score was very high. So they were surprised when they had poor retention during their first few months. It turns out that Brazilians are very generous “graders” (while folks in Australia and Germany are stingy with their 9’s and 10’s)."

"NPS provides an easy-to-understand signal to me, and the “What’s good?/What could be better?” verbatims are incredibly helpful."

"Another note: all of the data sources— with the possible exception of NPS— are gameable. If I aggressively ask readers for more “hearts,” for instance, I get more of them."

"the “Project v. Outcome-based Roadmaps” essay tops most of the data points I listed, and the qualitative from the NPS survey gives me a strong sense of “Why?”"

"The importance of stories and examples to bring ideas to life."

"And if there’s a small sample, NPS is really noisy — both high and low. So when I get a high or a low NPS score, I have to nicely remind myself, “It’s just a number,” as I dig into the qualitative to form hypotheses for my next experiment."

"“Axiomatically, if you have high NPS, you love NPS.”"

"2007: Netflix streaming launch"

"2007: Netflix Prize"

"The “Netflix Prize” offered $1 million to any team that could improve the predictive power of Netflix’s collaborative filtering algorithm by 10%"

"All ratings are not created equal."

"The more algorithms, the better."

"Contestants discovered that the ratings members provided for recent movies provided more predictive power than older ratings."

"2009: The Next Big Netflix Prize"

"2010: Testing the New “Netflix Prize” Algorithm"

"Netflix launched the Netflix Prize to provide better movie choices for members, which the team hoped would translate into retention improvement. But when Netflix executed the new algorithm in a large-scale A/B test, there was no measurable retention difference. It was a disappointing result."

"2010: Popularity Matters"

"Netflix published all of its learnings from the Netflix Prize, and other companies studied the results."

"2011: Netflix’s Movie Genome Project"

"Netflix’s new movie genome algorithm is called “Category Interest.” Now, for the first time, Netflix can suggest a movie and give context for why a member might like it."

"2011: How the Personalization Algorithms Work"

"Netflix creates a forced rank list of movies for each user — from the content most likely to please to the least. Then this list is filtered, sliced, and diced according to attributes of the movies, TV shows, and the member’s tastes."

"Netflix’s personalization approach has three components:"

"A forced-rank list of titles for each member"

"An understanding of the most relevant filters for each member so the algorithms can present a subset of movies and TV shows from the list above, and"

"The ability to understand the most relevant rows for each member, depending on the platform, time of day, and lots of explicit/implicit movie taste data."

"2011: Netflix Proves Personalization Improved Retention"

"2012: Profiles Re-invented"

"2013: “House of Cards” Original Content Launch"

"By this time in Netflix’s history, it’s clear that personalization delights customers in hard-to-copy, margin-enhancing ways."

"there’s another aspect of personalization that improves the company’s margin: the ability for Netflix to “right-size” their content spend."

"Based on its knowledge of member tastes, Netflix predicts that 100 million members will watch Stranger Things and invests $500 million in the series."

"Netflix has a huge advantage in its ability to right-size its original content investment, fueled by its ability to forecast how many members will watch a specific movie, documentary, or TV show."

"2013: Netflix wins a Technical Emmy"

"2015: Does It Matter if You’re French?"

"Like the demographics test in 2006, members’ tastes are so idiosyncratic that language and geography don’t help predict a member’s movie preferences."

"As before, the most efficient way to seed a member’s taste profile is to ask for a few TV shows or movies they love."

"2016: Netflix Tests a Personalized Interface"

"execute an A/B test of the five-star system against a thumbs up/down system. The result: the simpler thumbs system collected twice as many ratings."

"2017: From Stars to Thumbs"

"2017: What Happens to The Five-Star System?"

"2017: Percentage Match"

"So, with the loss of star ratings, and the insight that star ratings do not equal movie enjoyment, Netflix changed its system. They switched to a “percentage match” that indicated how much you would enjoy a movie, irrespective of its quality."

"2018: Personalized Movie Art"

"2021: Do you feel lucky?"

"Here’s the long-term personalization vision: twenty years from now, Netflix will eliminate both the “Play Something” button and its personalized merchandising system, and that one special movie you’re in the mood to watch at that particular moment will automatically begin to play."

"Today, more than 80% of the TV shows and movies that Netflix members watch are merchandised by Netflix’s personalization algorithms."

"Part 1: From startup to 2006"

"In twenty years, Netflix has gone from members choosing 2% of the movies the merchandising system suggests to 80% today."

"Today most members look at forty choices before they hit the “play” button. Twenty years from now, Netflix hopes to play that one choice that’s “just right” with no browsing required."

"2001: The Five-Star Rating System"

"2002: Multiple Algorithms"

"Dynamic store."

"Metasims."

"Search."

"2004: Profiles"

"Given the low adoption, Netflix announced its plan to kill the feature but capitulated in the face of member backlash. A small set of users cared deeply about the feature— they were afraid that losing Profiles would ruin their marriages!"

"2004: Netflix launches “Friends.”"

"The hypothesis: if you create a network of friends within Netflix, they’ll suggest great movie ideas to each other and won’t quit the service because they don’t want to leave their friends. At launch, 2% of Netflix members connected with at least one friend, but this metric never moved beyond 5%."

"Two insights on social in the context of movies:"

"Your friends have bad taste"

"You don’t want your friends to know all the movies you’re watching."

"2006: The Netflix Personalization Strategy"

"Gather explicit taste data, including movie and TV show ratings, genre ratings, and demographic data."

"Explore implicit taste data"

"Create algorithms and presentation layer tactics to connect members with movies they’ll love."

"Use the explicit/implicit taste data, along with lots of data about movies and TV shows (ratings, genres, synopsis, lead actors, directors, etc.), to create algorithms that connect members with titles."

"Improve average movie ratings for each member by connecting them with better movies and TV shows."

"The high-level hypothesis: personalization would improve retention by making it easy for members to find movies they’ll love."

"Netflix had a more sensitive, short-term proxy metric: The percentage of members who rated at least 50 movies during their first two months with the service."

"2006: The Ratings Wizard"

"The original petri dish for personalization was an area on the site with a “Recommendations” tab. But testing revealed that members preferred less prescriptive language."

"Members “binge-rated” while they waited for their DVDs to arrive. The Ratings Wizard was critical in moving the “percentage of members who rate at least 50 movies in their first two months” proxy metric."

"2006: Demographic data"

"How did Netflix measure predictive power? The proxy metric for the personalization algorithms was RMSE (Root Mean Squared-Error) — a calculation that measures the delta between the algorithm’s predicted rating and a member’s actual rating."

"Unfortunately, age and gender data did not improve predictions— it did not improve RMSE. Movie tastes are hard to predict as they are very idiosyncratic. Knowing my age and gender doesn’t help predict my movie tastes. It’s much more helpful to know just a few movies or TV shows I like."

"Netflix asks you for a few titles you like to kickstart the personalization system. That’s all they need to seed the system."

"2006: Collaborative Filtering in the QUACL"

"The QUACL is the Queue Add Confirmation Layer. Once a member added a title to their Queue, a confirmation layer would pop up suggesting similar titles."

"2006: The $1M Netflix Prize"

"Become the voice of the customer."

"Product skills"

"Technica"

"Management"

"Creative."

"Business"

"Design."

"Marketing"

"Consumer science. Develop consumer insight through the scientific method to discover what works."

"my requirement for technical skills is, “Don’t let your eyes glaze over when your tech partners talk ‘techy.’”"

"Leadership"

"Strategic thinking"

"Management."

"Proactive, results-oriented"

"Culture."

"Technical skill."

"Find the right role"

"Build trust with your engineering partners"

"Spend lots of time with your product."

"Be a consistent source of “ideas that matter."

"Be a great communicator."

"Write a lot. It doesn’t matter if you write articles, stories or code — they are essentially the same thing. The creative process takes lots of time and discipline, so the more you practice, the better."