How Radical Candor Can Revolutionize Your Team’s Performance by Jason Rosoff by Kazuki

How Radical Candor Can Revolutionize Your Team’s Performance by Jason Rosoff

Kazuki

Nov 1, 2024 10:41 PM

Summary

Introduction to Jason Rosoff (00:09): Jason is the CEO of Radical Candor and a founding team member of Khan Academy, where he significantly scaled user engagement from 100,000 to over 10 million users monthly.
Concept of Radical Candor (01:02): Radical Candor is about caring for colleagues while challenging them directly, blending care with constructive feedback to enhance performance and relationships in leadership.
Impact of the Radical Candor Book (03:36): The book, co-founded by Jason and Kim Scott, has sold over one million copies worldwide and emphasizes the importance of effective communication in the workplace.
Hiring Philosophy and Growth at Khan Academy (30:28): Jason reflects on the importance of hiring the right people to establish a strong organizational culture, sharing that mission-driven individuals are attracted to the nonprofit approach.
Feedback Mechanisms (20:40): Jason highlights the "get it, give it, gauge it" framework, which emphasizes soliciting feedback, delivering it (both praise and criticism), and assessing its impact for effective team dynamics.
Metrics for Success (48:43): At Khan Academy, they measured user growth, engagement, learning outcomes, and employee satisfaction to ensure educational effectiveness while retaining talent.
Long-Term Impact and Legacy (1:12:00): Jason articulates his vision of being like water, consistently making small, impactful changes over time, rather than seeking dramatic, immediate transformations in the education landscape.

Transcript

Hi, welcome back to Grasp Talk, and tonight, we are very excited to have Jason Rosoff with us. Jason is the CEO of Radical Candor, where he helps leaders and their teams build better relationships for exceptional results. He was a founding team member of Khan Academy, a non-profit educational platform offering free online courses, lessons, and practice for students of all ages. At Khan Academy, he started as its first designer and product manager, and later serving as

chief people officer and chief product officer. And during his seven years at Khan Academy, he helped scale the product from 100,000 users a month to over 10 million users. And today, we'd like to ask him about his journey and insights on leadership and team building. And please join me in welcoming Jason Rosoff. Thank you for joining, Jason, today. Thank you so much. I'm excited to have this conversation with you.

I'm really looking forward to it. Thank you. So, first of all, I know Radical Candor is a book written by Kim Scott. Yeah. And, but, you know, could you explain what is the concept of Radical Candor with us and our audience? Sure. So Kim is my business partner. She and I founded Radical Candor, the company together about seven years ago, about a year after the book was published. And one of the reasons why I decided to do this with her after Khan Academy, it's hard

to follow Khan, it's hard to figure out what your next act is after something like Khan Academy, which has had such a profound impact on so many people. One of the reasons why I decided to do this with her was because of how the book and the concept helped me better understand what I appreciated about the leaders who I admired and what I was doing when I was at my best as a leader and a people manager.

And so Radical Candor at its core is a really simple idea. It just means caring about the people that you work with while being willing to challenge them directly to continue to do really well or to do better. That's it. And it's when we combine both of those things simultaneously that the magic really happens, when we combine care and challenge. And the way that I like to think about this is, in the broadest sense, is that I look

back at my career, the best relationships in my career were all defined by people who really did demonstrate it to me that they cared about me and my success, but pushed me to be the best version of myself, pushed me to be better at my work. And that doesn't always mean correcting mistakes. Sometimes it was really about encouraging me, pushing me by encouraging me to lean into things that made me a little uncomfortable actually produced exceptional results.

So I think that at its core describes what Radical Candor is and why it was so helpful to me, because I just didn't have words, before Radical Candor, I didn't have a way to describe succinctly what it was that people did that made me appreciate their leadership and what it was that I was trying to do as a leader. I see. And also that's really impressive that the book is New York Times and Wall Street Journal

bestseller for multiple years, and also translated into 20 languages with more than half a million copies sold worldwide. That's pretty impressive. Yeah. We just actually passed one million this past year. So we hit a new milestone. Yeah. One million copies sold worldwide. Yeah. Incredibly impressive. I mean, Kim has this amazing talent for reducing ideas down to their sort of essence, the core, the nugget at the middle that is most valuable.

And she's a fantastic storyteller. So I think the combination of those two things is what made the book so popular. I see. And how did your image came? So I was on my transition out of Khan Academy, obviously, if you're chief people on chief product officer, it's not like you quit, right? There's a period of time where you're like transitioning responsibility to other people and things like that. And I made a commitment to myself to meet as many people who I thought were doing interesting

things, especially in the space of leadership. That was kind of my focus at the time. I was considering becoming a coach, actually, like full-time executive coach. And I read Radical Candor and I thought it was amazing and I was like, oh, there's no way. Like, who do I know that would know Kim or like how would I even get in touch with her? So it was sort of like on my list, but kind of fell off. And then a friend of mine just mentioned that, hey, like a friend, he was raising money

for a startup and he's like, hey, one of my investors introduced me to this person, Kim Scott. It seems like she would know something about this. And I was like, that's great. But like, what do I want? Before you make the introduction, like, let me think about what I would want to say. You know, what do I want to have a conversation with Kim about so I can make good use of her time? And she had just written this blog post because while she was writing Radical Candor, she

actually started a software company. So if you read the first edition of the book, it's like very present in the first edition of the book. It slightly changed in the second edition because like, you know, the time had changed and she was able to kind of like slightly update the story. But in the first edition of the book, it's really about there's all these references to this company that she started. This is not the company that she and I have been working on.

And really what they're trying to do is build software tools to help people put Radical Candor into practice. And so they had built a handful of prototypes, all of which had failed. And it had become fairly clear that, in Kim's words, that Radical Candor is about putting the phone down and like looking each other in the eye or like being really present and having a conversation with another human being.

And if that's what it is at its core, taking your phone out is a sort of value subtracting round trip. It's actually a distraction, not a help. And she wrote this blog post about shutting down that company and how painful it was and how challenging it was in that moment to realize that this was a really, really difficult thing to do, to teach people social skills, essentially, like the skills that are fundamentally relational

in nature. And that blog post just resonated with me so much because at Khan Academy, I think one of the biggest learnings that I had was that context matters. Like content is incredibly important. And what Sal was amazing at, Sal, who's the founder of Khan Academy, was amazing at was creating great content. But in order to learn, context matters. An example of this is when we were doing user research and we were talking to teachers who

were working in schools in Silicon Valley, where you wouldn't think things like food scarcity is a problem, right? You're like richest area of the country, one of the richest areas in the world. But these teachers were telling us stories about how kids were coming to school hungry, and that was really hard to manage, right? So you're not in a state to learn if you haven't eaten that day. And it seems so trivial.

But what it made me realize was that technology can't gather, you know, it doesn't have perfect context. It can't gather all that information. And so as we were doing all these things like building recommendation engines to figure out like what concepts followed which, like what other concept and like how to help people plan their learning journeys. Like one of the things that I realized was that that was only a tiny fraction of the

context that we needed to understand in order to really help that person learn. And Kim's blog post basically said the same thing, which is like, I didn't fully understand or appreciate the context in which this tool was going to be used and what people really needed. And as a result, I think we failed to be able to build something. And in her defense, I think, you know, this is 10 years ago, eight or nine years ago now.

I actually think that the technology required to build what she wanted to build didn't really exist. Like they needed to be like a real tech company that was like building, you know, AI or machine learning or whatever you want to call it, way back when. You know, it might have been a different case if radical candor is being published today with generative AI being as, you know, as, as cheap as it is, she might have had a different

kind of success. But anyway, that was the crux of our conversation was just a commiseration of how different it is to teach people things with technology as an intermediary, because the technology on its own, often fails to have enough context to actually be helpful to the to the person on the other side of it. And it was really through that conversation and just talking about how challenging it was that we built a bond.

And then we both were at a point where we're like, you know what, maybe we just want to do some stuff that doesn't scale. That isn't like maybe we want to try some things that are just smaller and simpler and are much more likely to where we're more in control of the kind of impact that we're going to have even if. Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp

Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp

Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Glasp Talk � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason

Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp

Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp Talk � Jason Rosoff � Glasp How do you navigate them to apply the things they learned into their daily life? Do you have some tips or advice? Do you follow up with them? How do we, in the moment, translate the concept into behaviors or actions, things that people can do in order to achieve better results? And then the second question that you're asking is,

how do we help sustain that learning over time? How do we help ensure that people are applying that learning after maybe an initial session? And the answer to that is, I think we have good answers on the first and we have okay answers on the second. On the first thing, one of the things that Kim did really well and the place where we've probably spent the most time iterating over the last few years is getting very practical or sort of tactical about what radical candor looks like

in the real world. So, in addition to this care personally and challenge directly frame, she also has a couple of other frameworks and some that we've co-developed over time. So, one of those frameworks is the order of operations, is what she calls it in the book. And this is the idea that if you want to build a team on which it is easier to give and receive feedback in order to help one another grow, there are some things you have to do in a particular

order. And so, the order of operations are get it, give it, engage it. And so, the idea here is that somewhat counterintuitively, if you want to build an environment that is rich in feedback, you actually have to build the most important habit to build is a habit of soliciting or asking for feedback on a regular basis. There's a great quotation, and maybe I can share the specifics with you if you all do show notes, that basically says that the act of asking for feedback, one,

makes it easier for us to receive. So, just because like even if the feedback is delivered the same way, if we asked for it versus it was delivered to us unsolicited in the exact same words, we react differently. At like a biological level, we react differently. So, this is like they've done fMRI studies of people's brains and things like that to see like what's happening inside us when people give us feedback.

And two, the act of asking for feedback is one of vulnerability and builds trust. So, I think this is one of the first major lessons for a lot of people who walk into a radical candor session is they assume that the most important thing they can do is sort of tweak the way that they deliver their feedback if they want to get better results. Like, I'm going to learn like a little change that I need to make and how I say my feedback to other people, and that's what's going to help me succeed.

And we're like, nope, actually, that's not where to start. The place to start is by asking for feedback. Then we tell people you need to give feedback, and that is means both praise and criticism. So, this is the other thing is that people think the only valid form, many people live in a world where the only valid form of feedback is criticism. That's the only thing that's going to help you grow. And so, that's the only kind of feedback that matters.

It turns out that that's wrong on every level. So, like the most valuable developmental feedback that people get is praise. It is helping people identify. Another way to say it is like positive target identification. It's basically helping people see or understand or perceive what they do that actually works. And the example that I like to give is there was a study that has nothing to do with work. It was done with bowling, the American sport.

I think your audience is going to be familiar with it, even if it's global. But anyway, bowling, it started with this sort of bowling coach, and they did this study where they had the coach review tapes of people bowling. And then the coach, in one condition, the coach would criticize, like, oh, you put your foot in the wrong position. Your foot was turned in. You need to turn your foot out, for example. Or where you release the ball was too high or too low.

So, they were giving tips, right? They were saying, here's the correction you need to make. So, it wasn't just like, oh, you did it wrong. It was, you did that wrong, and here's the correct thing to do. And then in other situations, what they did was they had that same coach review the tape and say, oh, here's one thing that you did right. You should focus on doing that thing correctly again. And the improvement was significantly more in the group that received praise than it was in the group that received

criticism. And it wasn't just about good feelings, because that's the thing that people assume. It's like, oh, it feels good to receive praise. And so, that's why it worked. What was actually happening was if you're a novice and you're getting coached in something, if you're new to something and you're getting coached, it's very hard to know how to do something you've never done. So, let's say the foot position thing. Let's say you always make that mistake.

You always put your foot in the wrong position. And then someone says, put your foot in a different position. Well, it's actually kind of hard to do that, because you've never done that before. But if they're like, where you're releasing the ball is perfect. Like, I really want you to stay, like, really focus on releasing the ball in that position. Well, that's something you've done before.

It's easier to redo a thing that you've done before than it is to do something you've never done before. And so, once you focus on a couple of those good behaviors, and those become rote, right, those become sort of innate, you know how to do them, then it's easier for you to know how to do them, then it's easier to layer on top of that correction to the mistakes that you're making. Right? Because, like, part is working really well. That feels really strong. Now, let's layer on a correction to the mistake.

So, anyway, that example, it seems really trivial, but it turns out that that result has been shown in all kinds of feedback contexts, which is that coaching people on their strengths or the things that they do really well is a much more effective way to rapidly increase performance than focusing on the things that people do wrong. Now, obviously, we need to stop mistakes from happening. And criticism is very important, but the ratio of praise to criticism should be greater than one.

And in almost every organization that we walk into, when we ask people, hey, you know, how much feedback do you give? And it's usually not a lot. So, that's one problem. And then the second question is, like, how much criticism and how much praise? And it's almost always the case that they're like, oh, about equal or maybe a little bit more criticism. Like, we don't really praise each other here. That's a really common thing that we hear.

And while there's no, like, perfect ratio of praise to criticism, like, the research on that is that it's multiple. So, it's, like, three times, five times, seven times the amount of positive reinforcement to criticism is actually ideal. So, get it, give it both praise and criticism. And while we're not encouraging people, we're not saying it has to be two to one. We're saying, be conscious of the fact that you want to focus on the good stuff and make sure that you're talking about what's working, because that's actually going to help you and your team get

better results faster. And then gauge it is the last step. And gauge it is really about measuring how your feedback is landing. So, one of the things that I, this is one of the other misapprehensions that people have, which is, if I give feedback and I do it well, well, in air quotes, my job is done. Like, I've done the thing. And turns out that also is not true. Like, we need to be concerned about the impact or effect that our feedback is having on the people that

we're giving it to. Whether it's praise or criticism, we need to pay attention to how the other person is responding and be willing to adjust our approach. And there's lots of good reasons for this. First of all, we often give feedback the way we like to receive it. So, you know, I like pretty direct feedback. It's helpful to me when people just say, like, hey, here's how, like, here's how I think about that thing. This is what was good about it.

This was, this is what was, could be better about it. That, people getting straight to the point is really helpful to me. But there are people on my team who do not feel the same way, who would rather a more curious approach, like where I ask questions instead of sharing my perspective first. And we need to be willing to make those kinds of adjustments. Because our goal as colleagues or as leaders is to make sure that we are actually communicating with the other person.

Like, it's not enough to say the thing. We want to make sure the other person is understanding what it is that we're trying to say to them, which means there's no one size fits all approach to radical candor. In the book, Kim says, radical candor is measured not at the speaker's mouth, but at the listener's ear. And I think that's a really simple way of thinking about it. It's like, we don't get to claim we've been radically candid.

We can only know after we've solicited some feedback from the other person about how our communication actually landed with them. I see. Yeah, that reminded me of quotes or saying that, you know, people wanted to be valued, appreciated, trusted, understood, and so on. Yeah. Yeah. Yeah. Totally designated. And thanks. And, okay, then let's move on to your past career and Khan Academy, if that's okay. And I'm very curious.

You joined Khan Academy in 2010, and Khan Academy started 2006, right? And you were a founding member. And I'm curious, how did you start working at Khan Academy? Did you know Sal Khan before? Or, yeah. Not really. So, from 2006 to 2010, Khan Academy was basically a YouTube channel. There was a little bit of software development there that was, like, used with a very small number of people. But in 2009, Sal was trying to take some of the software stuff that he was doing and bring it to more people.

And so he started an open source project where people could contribute. And that's actually how I got introduced to Khan Academy, was as a contributor to the open source project. Yeah. In 2010 was the first time that Sal got any sort of funding to hire anybody. So, the first bit of funding he got was from Ann Doerr, who is a philanthropist in Silicon Valley, who sort of heard, maybe saw Sal in an interview or something like that and had a conversation with him.

And she asked him, like, how are you funding Khan Academy? He was like, out of my savings. Like, I'm funded. It's like self-funded. I'm, like, spending my own money to run it. And it was at a point where he had a family and his family. And they were like, can we continue to do this? Like, how is this going to happen? And so she was his first major donor. And she went back and wrote him a check for, like, $100,000 or something like that and said, you shouldn't be funding this out of your savings.

And then shortly after that, he got, you know, maybe six months after that or something, he got funding from the Gates Foundation, a million-dollar grant. And it was on the back of that million-dollar grant that he was able to start considering, like, building a team. And so with that money, he hired one of his best friends from college, Shantanu, who was, like, the chief operations officer, like, COO-type person.

And then he hired Ben Kamens as head of engineering and me. And Ben and I were working together at a company called Fog Creek Software in New York City at the time that this happened. So we were actually, we had been working together for, like, five years at that point. And we had been contributing, working together to contribute to this open-source project.

And yeah, he sort of reached out to us and said, you know, I'm looking to hire, like, I'm looking to hire a team. You know, Ben had been volunteering longer than I had. And he's like, you know, do you know anybody who might be able to do some product stuff? And Ben was like, well, Jason's actually been in the background helping me with, like, the product definition design and things like that. And we met for the first time over the phone. I was in New York and he was in Silicon Valley.

And then Ben and I flew out for, like, a week to meet him and Shantanu. And yeah, the rest is history. I think, like, he was, we collaborated together over the course of that week. He was pretty excited with the direction we were able to take things. And he sort of offered us the job on the spot. Now, the job was for significantly less money than we were currently making. And it required relocating to California. So it was a little complicated.

Like, from a family perspective of choice, my wife and I bought a house in 2005. And we're like, what do we, like, do we sell the house? Like, what if Khan Academy collapses? Like, you know, maybe, like, it might not go anywhere. You know, the classic sort of startup story of, like, do we uproot our entire life for this? But I was lucky enough to have a partner that was, you know, excited to, for an adventure. And it turned out to be an incredibly, like, a good decision, not only because Khan Academy

turned out to be very successful, but also because, you know, we loved living in California, it was like a great change of life for us. And allowed my partner, Jillian, to change careers, too, because she was in sales up until that point. And she actually, she's now a therapist, a marriage and family therapist. So she went back to school and started her new career in California. But that was, like, part of the deal.

So I said, well, if we move, you know, I get to do something new, but you also get to do something new if you want. I see. And I'm curious about early days hiring in early days. Because, you know, like, in early days, if you hire someone, and that affects the culture a lot, right? The percentage, the ratio is higher. And I wonder how did, how did you find the right person early days? Because, you know, those people set the culture, right? So, yeah.

Yeah, I think, I mean, the truth is, we didn't always find the right person. So what I will say is that one of the big advantages of Khan Academy, I think there's a similar advantage at Radical Candor, is that there's a, there's like a philosophical underpinning to the reason for the business to exist. And that really is quite helpful at weeding out, like, a big chunk of people who you wouldn't want work, you know what I'm saying? Like, I feel like the idea of,

like, caring deeply about educating people, you know, for free all over the world. And at its core is this sort of, there's this moral stance that, like, people deserve, like, it's a human right, high quality education is a human right. And so it tended to attract people who already believed those things, which helps a lot in terms of, like, maintaining a culture that is really mission driven.

The fact that Khan Academy was a not-for-profit also helped because they're, like, you knew when you joined that there was never going to be stock options, you know what I mean? This is not the kind of thing where you're going to, you know, you're going to get rich. So that filtered out a whole bunch of other people, especially in Silicon Valley, who would not have been great fits for the organization. But we made some mistakes along the way.

I'm not going to say that we didn't, you know, especially because, like, we, there were some things that we had to get good at that we weren't necessarily very good at, like, fundraising. You know, Sal is, he's actually really incredible as a fundraiser. And I think he's talked about this publicly, but Sal is terrified of flying. And so it was, like, it was pain, literally painful for him to, like, go fly to do this fundraising.

So one of the, we tried a few times to find the right person to do fundraising for the organization. And that was really hard. Like, one, we didn't have a great perspective on how to do it. So, you know, Ben and I kind of had it on the product and engineering side of things. We had a philosophical approach really rooted in what we had learned from Joel Spolsky and, you know, Spolsky and the team at Fog Creek for, like, how to hire and what we were looking for and how to approach software development, things like that.

But that was an area of the business where we didn't. So we had some false starts there, right? We, like, hired some people and they weren't quite the right fit. They were, you know, they were maybe used to fundraising for a different kind of organization, different kind of nonprofit organization than the one that we were trying to build and things like that. So that's an example.

We also made some bad hires on the product team, you know, people who weren't exactly the right fit. But the benefit, I think, in some ways is that it was apparent pretty quickly that they, you know, that they weren't the right fit. And I think that has a lot to do with the ethos of the organization being really clear to all the members of the team. And I think the same thing is true with Radical Candor. Like, you know, there have definitely been some people that we worked with over the years that we thought were going to be great

and turned out not to be great, both from a fit perspective, like cultural fit or contribution perspective, or just from like a working style perspective and things like that. Radical Candor is a completely distributed organization. We have no offices anywhere in the world. And that takes a special kind of person to do to do that. I think you need a different kind of mentality to be effective in an organization like that.

So anyway, I guess the long and the short of it is I don't think... The goal is not to avoid mistakes. Like I'm sure the goal... Yes, try to avoid mistakes wherever you can. I think the goal is like responding to mistakes with action as opposed to sort of inaction or paralysis. And I would say I have gotten a lot better at that over the course of my career. I have felt really empowered to do things, to like take action at Radical Candor.

Khan Academy struggled a little bit more, especially in the early days of reacting quickly to problems in hiring, but it got much better at it as time went by. And I feel like that's really where the magic happens. I think you have to be willing when you're growing quickly to take a chance on people, and you have to be equally willing to pull the plug if that experiment didn't work out.

One of the things that we've been trying at Radical Candor is more of like long-term consulting to hire. So we might hire somebody full-time as a consultant for three months or something like that. And it's not just a trial period to see if they'll fit in the organization. For us, it's also for them, right? They get the opportunity to experience this. And we pay... When we do this, we do try to pay people really well.

So it's not cheap, but it's less expensive and easier to break up at the end of it if it's not working out than if you hire somebody out right from the beginning. And I think more and more organizations are trying that. And a quick question, by the way. So when you interview people and you have a gut feeling, and you see their skills, performance, and communication skills and so on, but also you have gut feeling about that person sometimes.

So do you advise trusting in that gut feeling? Yeah, I'm curious about your opinion on gut feeling. Sometimes everything looks great, but my gut feeling says no in that situation. Have you went through that situation? Yeah, absolutely. And here's the thing about my gut is she's unreliable. I think what I've learned is that when I get one of those feelings, it's usually an indication... It's helpful to look underneath that feeling.

So usually the gut feeling is like, I feel uncomfortable for some reason, right? Like there's some reason I feel uncomfortable. And instead of making a decision based on that gut feeling, instead of what I've learned to do over time is to try to understand the reason beneath the gut feeling. What is it that I am uncomfortable with about this situation? Can I name it? Can I get more specific than that? That is probably the biggest lesson that I've taken away.

The second thing that I'll say is that I don't think you should make hiring decisions based on gut feelings. I also don't think a single person should make a hiring decision completely on their own. I think that hiring promotion and firing decisions should have at minimum two people's perspectives in there because there's always a risk that the source of that gut feeling is bias or prejudice against that person for some reason that I might not even be fully aware of.

And so one thing that we instituted at Khan Academy to help with this was we had a rubric, meaning we had a definition of what we were looking for in people. And we asked people to evaluate against that rubric. So that was number one to help reduce the amount of bias. But we left like at the spot about the rubric, it was like a blank spot that said like, what else did you notice about this? And the approach that we took was the results of the interviews that came before you were not available to you until you had completed your own interview.

So you couldn't look at what other people had said about a candidate until after you completed your interview with that person. And what we noticed was that those gut feelings were often reinforced when other people had the same observation, kind of blinded, right? Without knowing that the person before them had had a similar feeling. And that was a really helpful way to turn the gut feeling into data that you could actually respond to in a way that was fair

to the person who was applying for the job. I think we experienced the same things as you mentioned just recently. So yeah, that's why we are noting. So yeah, you as a white person, we should ask. Yeah, but by the way, so you're working as a CPO at Khan Academy, but so turn out, yeah, turn like in chief product people officer, then chief product officer, how was that like transitions? And why? There was no transition.

I did it wrong and I did both jobs at the same time, which I will, I do not, I strongly discourage. Yeah. I couldn't be, if I learned one thing, it's that I am not the kind of person that can do two, two big jobs simultaneously without like it harming me, like my health. Like I was effective, but I was sacrificing my sanity in order to, to, to, to do both of those jobs. Here's how it happened though. I, because like this, that the, how it happened, I think is sort of interesting, which was as we were growing,

the, the team that grew the fastest was the, the sort of product design engineering organization. That was the team that was growing the fastest inside the company. We had all these great hiring processes in place, these rubrics. We also had like leveling. So we'd like, how basically a, another set of rubrics for how to consider someone for a promotion. We built up all of this, these sort of like mechanisms for hiring and managing people that were working pretty well

to, to, to both attract and retain the kind of talent that we were hoping to have, not just in product engineering and design, but across the organization. And I said, the reason why this is working is because we've treated this as a design problem. We basically said, Hey, like an organization is designed to serve the people that it hires and to ensure the people that it hires can serve the organization's mission and successfully execute the tasks and duties that

correspond to the role of the person's hired into it. That is a design problem. And so I was like, I feel like we've shown a successful pattern here. I want to bring some of that to the rest of the organization. I want to treat people operations as a design challenge. And I, you know, to Sal's credit, he was like, okay, go for it. And I was like, Oh, I felt like, you know, the, I don't know if this expression is a,

is a global bond, but I feel like the felt like the dog that caught the car, you know, sometimes dogs chase cars. But like, they don't catch them. And that's good because if like a dog were to catch the car, like now it would be terrifying, right. Cause it'd be dragged along by the, by the car, but I was the dog that caught the car. And it was really, really, really, really challenging to do both those things, but also incredibly rewarding.

Like I got to build a great people operations team, super proud of that, of the work that we did together. We got to build like a more robust recruiting function and things like that side of the organization that was really good for the organization's ability to grow as, and to like evaluate and retain talent. Thank you. Not sure if this is a light question because like current academy is NPO, but so what kind of metrics or like KPIs, if you have, so what are you following at current academy?

Yeah. If you can tell at the point. Yeah. For my, in which role? CPO maybe. Yes. Productive. Yeah. So we had, we had a handful of metrics, like the standard ones that every, you know, software company has like user growth, engagement retention. But we also had learning goals. So we had some metrics that corresponded specifically to how much people were learning. And we had, we had ways to evaluate how much learning growth was happening on the platform.

And those were like, those were quite important. Now it turns out there's a virtuous cycle between engagement and learning growth to a point. One of the things that was cool about working at Khan Academy is like, you know, we were a part of a, I want to say a generation of apps like Duolingo and things like that where we were very friendly with each other, you know, like the, the explosion of the like Coursera type organizations.

And so all of these product teams were like talking to each other about how you can measure these things and like, what does it actually look like to be successful? And. And we always landed on the side of we'd rather be more conservative around engagement and not try to push engagement so hard. And stick a little bit closer to learning-first. Learning as a leading metric or at least equal to engagement. And at some point, I remember, and I might be misremembering this story,

so I apologize to my friends at Duolingo. But at some point, they went the other direction and they focused far more on engagement. And they were like, the learning will follow. They were seeing the same pattern that we did, which is as engagement goes up, learning goes up. But they made a decision where it's like they made the app easier, which increased engagement dramatically.

And they actually saw a negative response, which is like learning, total learning went down. It stopped increasing as engagement went up. And that to me was like, I remember talking to my team, my product team and being like, someone's going to do it. Someone's going to do the engagement thing. And I don't necessarily want it to be us. I don't feel strongly enough that it's worth the experiment to juice engagement at the cost of learning. I said, someone's going to do it and we'll learn something from it.

If I'm remembering, again, this is 10 years ago, if I'm remembering the story correctly, Duolingo tried it and they wound up having to revert the change that they made to actually keep the learning and engagement tied more closely together, even though that was going to slow down their platform growth potentially. So yeah, those are the KPIs. And we also had some internal KPIs around employee satisfaction,

like how well people understood their roles and careers and things like that on the product team, because we knew that our biggest risk was losing talent to the next big startup, right? And that did happen a bunch of times. So we had the product related metrics and then we had employee retention satisfaction metrics that we paid very close attention to because retaining talent was a big challenge. I see.

And also you have seen the growth from 100,000 users a month to tens of millions of users at the end. And have you found any deflection point or turning point that, oh, really something went viral or any turning point? That's sort of interesting. There were like a couple of occasions where I think that was true. But the interesting thing about Khan Academy is like interest in Khan Academy from the things that would typically go viral, which is like a story about Khan Academy or something, those

didn't typically get to our users, right? Because our users were like primarily school age kids who didn't care about news stories about Khan Academy or that Sal wrote a book or any of that stuff. So those things, those like viral moments, they attracted donors. That was really helpful, right? So they helped fuel the money into the company. They didn't help fuel usage.

The thing that helped fuel usage was building great content and making it more accessible on the internet. One of the big learnings for me about Khan Academy was like we had to learn to treat Google as an interface to Khan Academy. Because most of the people who came to Khan Academy for the first time, the way they got there was by typing a homework question into Google. And so we had to learn how to make sure that our interface spoke equally well to students and to robots so that it was easy for people to discover the content

that we had. So great content, number one, discoverability of content, number two, were the things that actually led to growth. And there were some moments where we got significantly better at discoverability that had real growth implications. But I would say most of it was just sort of like hard fought year over year, like incremental improvements, making things better.

And then the law of big numbers being on your side, which is like 1% growth on 20 million users is still a huge number of new users, right? So if you're growing 10% year over year, which is small for a lot of platforms, that's still huge for education, right? Because there's 2 million more students a year getting access to high quality, free educational content. I see. And sorry, this may be like a stupid question. But you know, out of my curiosity, do you think Khan Academy was going to be successful if they are for profit?

Hmm. I don't think that's a stupid question at all. Like, I I think it would have made it very hard if we had to monetize early, it would have been it would have made it very hard to there would have been a lot of things that like led us away from the mission. That's what I think. If the mission is to provide a free world class education for anyone, anywhere free of charge, if that, like, you know, I'm saying a world class education free

of charge, like, I think being a for profit company would have challenged us a lot. Now, I don't think it would have been impossible, necessarily. But I think it would have been incredibly challenging. What I will say is that even as a not for profit, one of the things that we realized over time is we had to, if we wanted independence from our funders, we had to find ways to generate revenue, we just had to find the right audience to charge.

You know what I'm saying? We had to find the right customer to charge. And, and what, over time, the way that we accomplished that was, we charged partners like the College Board who makes the SAT, we charged them to build free SAT prep on Khan Academy. So they paid us to make a free product for their, for their audience, because one of their objectives as an organization was to make it so that there wasn't such an advantage for people with wealth in taking the SAT. So that was one way that we made money.

And the other way that we made money was by licensing to school districts who have a budget, like, you know, you can buy a textbook, or you can license, you know, Khan Academy and get an interactive textbook, essentially, licensing school districts. And then also, there's a virtuous cycle there, because the school districts could apply for grants to get Khan Academy as a license, but essentially, like, the, the, it gave us more independence from from the grant grantors,

because the relationship was us to the school district, not us to the, to the grantor. And that helped to align some things that and generate some independence. And both in sort of in terms of like, decision independence, but also, it's a lot of work to like raise money every year, you know, I'm saying raise large sums of money every year. It's a lot better if you have like recurring contracts with school districts, that's a that's a much more straightforward way

of being able to, you know, to meet your budget, to meet your budget every year, less likely to go wrong, in some ways, with that. Yeah, totally. Yeah. I understand. Yep. And also, you touched on the AI, and education in the beginning, right? And, you know, nowadays, you know, people are using Chachibiri, anthropic growth, Gemini, for many things, you know, and then some teachers, professors, you know, students are using, like AI or large language models to cheat in testing, you know,

in test or homework, etc. And how do you think education or learning change with AI? From your experience, you know, what you have seen from Khan Academy, and from your, the current job in some? Yeah, so I, I have some, I have sympathy for the college professors who are like, oh, they're like, they're using it's possible to cheat the system. And that makes AI bad. For education, I have sympathy for that point of view.

And I feel like our systems shouldn't be so easy to cheat. Like, if we really want to evaluate a student's learning, like they're, you know, there's a good question in my mind, whether writing a paper on your own and turning it in asynchronously for a grade is really a great evaluation of a student's ability, like maybe that should be alive. You know, maybe, maybe more classes should wind up with something like a thesis defense, as opposed to, you know, a paper on your own.

Jason Rosoff � Glasp Talk � https://www.glasp.com.au Jason Rosoff � Glasp Talk � https://www.glasp.com.au Jason Rosoff � Glasp Talk � https://www.glasp.com.au Jason Rosoff � Glasp Talk � https://www.glasp.com.au

Legacy or impact do you want to leave behind for the future? She would often come to be very frustrated that we weren't able to make like big sweeping changes to the platform, or it felt like that the impact that we were having was incremental. And, and I said, like, I, I really do appreciate the frustration. Like, I understand where you're coming from. And, like, education, the world of education, it's like this big rock, it's ossified, you know what I'm saying? It's like, creaky and old and it's very hard to move. So I was like, I tend to think of it as like this huge boulder that we're trying to change.

And I was like, and it's granite, you know what I'm saying? It's like very, very hard stone. And I said, like, the frustration that you're feeling, I understand, but you're like, you know, you're, you're not even a chisel, you're like a point, like a needle. And you're, you're, what you're trying to do is like run as hard as you can against that rock over and over again. And what's going to happen is the rock is going to dull you. You know what I'm saying? Like, you're not going to win that fight. Like, you're just going to wind up being a dull needle.

And I said, like, the way that I tend to think about the impact that I want to have on this problem, and this is how I think about most problem, most problems that I think are worth solving, I think, wind up being similar to this. Because I said, instead, I want you to be water dripping on the top of that stone. I want you to be implacable. I want you to be tireless. I want you to be constant. Because the only thing that's changing the shape of that rock is water. That's it. That's the only thing that has a chance of changing the shape of that rock. Nothing else, you know what I'm saying? Like, you could get it, you could try to get a jackhammer against that thing. But as one person, that jackhammer is not significant enough. The other thing that could potentially change the shape of that rock is huge collective action, right? Millions of people with jackhammers chipping away at that rock.

Right? And you destroy value. You don't just create a new thing. You've, like, destroyed the core, like, the thing at the center of it. So I said, as a person, as an individual, I think the better strategy is, like, be the water, right? Drip, drip, drip, drip on that stone and change it over time. Have patience. Because what you're doing is making a difference. It might not seem like a huge difference. But every single, you know what I'm saying? If you look at a rock that has water running over it, where the water runs over it, life grows, right? Like, plants grow from the water running over it.

And it's, like, it's hard to appreciate that when you're the water. Because it's sort of the stuff is growing behind you. But it's happening. Like, you're making a difference. And she found that really helpful. And I think, and I hope that maybe some folks in your audience found that helpful also. Because I think that is the legacy that I want to have. I want people who come after me to say that, you know, Jason helped to gently carve a path for me. Or for, like, helped to, like, patiently change the landscape of the company that we work for.

Or the problem that we're working on in a way that made it better for everyone along the path that he worked. That's what I really hope. Like, a little better every day. I know it seems really trite to have a similar, take a similar approach, right? But they do, they take that little better every day approach. Because I have seen it have massive, massive impact over time. You know, Khan Academy, it seemed like this huge thing that was changing all the time. But honestly, it was those little improvements that we made along the way that wound up making a cumulatively huge difference over the course of 70 years while I was there.

Yes. Yeah, it's really deep. And yeah, thank you so much for sharing the story. And yes. Yeah, so thank you so much for joining today. Yeah, we really enjoyed the conversation and learned from your experience and insights. Thank you so much. Absolutely. My pleasure.

Share This Audio Transcript & Summary 📚

00:0001:15:12