Unconfuse Me with Bill Gates
EPISODE 02: Sal Khan
Date aired: August 10, 2023
SAL KHAN: I started getting a few goosebumps. And then I said, “Ask ChatGPT why that’s the answer.”
BILL GATES: Hmmm.
SAL KHAN: It explained it!
BILL GATES: Oh yeah, it’s so good at that!
[music]
BILL GATES: I had great teachers that I’ve learned from. I had a librarian at my elementary school. I have a great tennis coach. And, you know, the best way to get ‘unconfused’ about something is to find somebody who really, deeply understands it. I call that, ‘getting unconfused.’
[music]
BILL GATES: Welcome to Unconfuse Me. I’m Bill Gates.
[music]
BILL GATES: The role of technology in education isn’t always obvious, but the potential and the possibilities are inspiring. When I would talk to people about loving to learn on the internet, and it was just at the point where putting up videos was kind of getting mainstream, in about 2010, people said, "Wow, you’ve got to use Khan Academy." My guest today is a teacher. He’s CEO of Khan Academy, Sal Khan. Khan Academy is an amazing tool. I was so impressed that Sal was able to teach such a broad range of subjects himself in the early days. Khan Academy’s got 140 million users. A lot of students have benefited. I’m excited to have him here today. Welcome, Sal.
SAL KHAN: Thanks for having me.
BILL GATES: Well, to start, I found out when you were younger, you were a singer in a heavy metal band and a cartoonist for the school paper. Which is the better skill, drawing or singing?
SAL KHAN: Depends what you’re trying to do, but I’ll say drawing.
BILL GATES: Yeah, well, in your lectures, you do have to draw stuff.
SAL KHAN: Oh, yeah.
BILL GATES: You erase a bunch of times if you don’t like what you did.
SAL KHAN: Every now and then, if you watch enough of the videos, I try to flex my art a little bit.
Sal can draw a cartoon. Sal can do a little bit of shading and cross stitch, and all of that. When I was a kid, I would just draw for hours. Now I realize it does give you a certain, one, you never get bored. And it does build a certain curiosity and creativity, just drawing all day.
BILL GATES: Well, Sal, remind us how you got into all of this. I think you were at a hedge fund at one point.
SAL KHAN: You go back to 2004, my original background was in tech, but I go to business school, I go into finance. I was an analyst at a hedge fund in Boston. I had just gotten married, family visiting me from New Orleans, which is where I was born and raised. I just came out of a conversation with my 12-year-old cousin, Nadia, who was having trouble with math. I said, "Hey, Nadia, when you go back to New Orleans, I’m happy to tutor you." She agreed. I started tutoring her after work every day. And long story short, that made a difference. It went from her being a weak math student, or perceiving herself to be a weak math student, to being a very strong one. I started tutoring her brothers. Word spreads in my family, free tutoring is going on. Before I know it, I’m tutoring ten, fifteen cousins. By 2009, there were almost 100,000 folks who were using it back then. That’s when I took the plunge, set it up as a not-for-profit, admission-free, world-class education for anyone, anywhere.
BILL GATES: And then it spread.
SAL KHAN: A lot of the vision of Khan Academy in the early days, yes, there was an accessibility piece of it. We can make videos! We can share it with everyone. But the pedagogical aspect wasn’t just access. It was that people could finally leverage technology to learn at their own pace. If you only got a 70% or 80% on some concept, the class doesn’t need to move on to the next one and build on that gap. The student should be allowed to work on that. If you just let people have a strong foundation, the next few layers in math come pretty easy. With a weak foundation, no matter how bright or hardworking you are, they’re very, very difficult. So that was the thesis. I think over the last ten years on this journey, we’ve got a ton of efficacy studies in different contexts, different countries. They all kind of say the same story, that if students engage, let’s call it 30 to 60 minutes a
week, even that, what I would call fairly low level of engagement, they’re growing pretty dramatically, 20%, 30% more than expected, or in some cases more.
BILL GATES: Amazing.
SAL KHAN: And so I think the question over the next couple of years is: How do we get more students to engage at that level?
BILL GATES: A lot of our conversations have been about how your tools are a miracle for the 1015% most motivated. But then, how do we draw in that other 85%? And I do think in the last year or two, as you’ve engaged with teachers and districts, we’re starting to see that it can be a tool basically for all students.
SAL KHAN: That’s the goal. But to your point, I actually think 80, 90% of – hopefully, 100% of students, if they’re able to engage in the right way, especially if you catch them early – they can have that foundation. I think the main issue is just a lot of kids, frankly, just check out by the time they’re in middle school.
BILL GATES: In learning, you’ve got the classroom environment, you can use that time in a certain way, and you’ve got whatever time you get with a student where they’re kind of by themselves and assume they have a device. How do you see the time in the classroom? Is the availability of your tools shifting what people do in the classroom? And do they use it real time in the classroom?
SAL KHAN: When we were starting ten years ago, twelve years ago, it was only more affluent school districts that would have one-to-one laptops. The rest of them had to share. The good thing is over the last ten years, because of E-Rate, a lot of that’s been addressed. It’s unusual now to see a school where you don’t have one-to-one laptops in, say, a math class. All the students will take out the laptops and do twenty minutes while the teacher walks around. They’ll take ten kids aside, do a focused intervention with them, do a worked example with them. The other twenty keep working, then they’ll take another ten aside. But when they do that consistently, it makes a huge difference. And then for the students who either need gap-filling or the students who are ready to race ahead, they can do so. That’s what they really like about Khan Academy. Not only do you get that practice and feedback in the moment, but the kids who want to race ahead can and the kids who need to do some gap-filling can too.
BILL GATES: And the dashboard that helps the teacher see the status of all the students, that’s a
Khan Academy thing that the teacher account gives them a really clear indication of what’s going on.
SAL KHAN: Yes. We all remember growing up in a classroom, that the only time of real measurement is at the end of the unit, and then you get the test, the test is graded, you get a 90%. I get an 80%. All right. Let’s move on. No one even attempts to improve that. What we realize is that now with tools like Khan Academy, you can give real-time information. You can give real-time feedback to the student, but you can also give real-time information to the teacher so that they can intervene, and they can do things. I hope in the future that that practice is the assessment, that you don’t even have to take separate assessments. But yes, the idea is ‘let’s give the teachers real-time information; let’s not wait until it’s too late.’
BILL GATES: Let’s talk about AI. You and I have both been lucky enough over the last six months or so to have engagement with the mix of both Microsoft and OpenAI and have early access. And I remember some of the best examples of how to get the AI to do fun things, where, when you came up and saw me and we were brainstorming about this, you were the one who said, "Hey, if you tell it to write a speech like various politicians, including Trump and others," that it’s stunning how it captures the voice of different people. Give us how you first were using the AI, and then this is super timely because you’ve just recently come out with the Khanmigo, if I’m saying that right.
SAL KHAN: That’s right, very good. The OpenAI folks reached out, and they said, "Hey, we’re a couple of weeks away from having the first training of our model." They wanted to reach out to Khan Academy for two reasons. One was they said we want to make this really good at AP Biology, and I only found out later – I don’t know if this is true – they told me you gave them the challenge.
BILL GATES: That’s right. In June, they kept showing me this thing, and I was like, "Yeah, it’s kind of an idiot savant. I don’t think it’s practical. Why don’t you see if it can do the AP Biology exam. And I’m not going to pay any attention until you can get a 5." And I thought, "Okay, that’ll give me three years to work on HIV and malaria." And then it was so bizarre because Sam Altman and Greg Brockman in late August said, "Hey, we want to come show you this thing." It was early September when there were like thirty people at my house. And I’ve said it’s the most stunning demo I’ve ever seen in my life. I mean, right up there with seeing the Xerox PARC graphics user interface that set the agenda for Microsoft for about fifteen years. This demo was so surprising to me, the emergent depth that as they scaled up the training set, its fluency, and you have to say understanding, that computers could not read in the sense humans do, and it couldn’t write in the sense humans do. And now, with lots of footnotes about hallucination and things like that, but I’m still personally in a state of shock at wow, it is so good, and okay, therefore, let’s see where we can put it to good use.
SAL KHAN: Yes, 100%. And so thank you for that challenge to them. I think that helped. They came to us because we have a large library of AP Biology questions, etc. They’re like, "Hey, can we use that to either evaluate or train?" And at the time I was like, "Well, what’s in it for Khan Academy?" You know, I’m in it. And they’re like, "Well, maybe you get access to the model." And I, too, was skeptical. I had seen GPT-3 at that point, and it was cool, but I don’t see how we were going to apply it. Two weeks later, they showed us the AP Bio question and said, "So Sal, what’s the answer to this?" And I read it. I was like, "Okay, I think it’s C." And it said, “the answer is C," and I was like, "Oh, that’s interesting." I started getting a few goosebumps. And then I said, "Well, ask it why that’s the answer." It explained it.
BILL GATES: Oh, yes, it’s so good at that!
SAL KHAN: I think what folks need to realize – because everyone had that moment with ChatGPT, but this was like that, but more, because GPT-4 is even better.
BILL GATES: Way better.
SAL KHAN: And I said, "Can you say why the other answers aren’t correct?" Did it. Then I said – I’m almost shaking – I said, "Can you write ten more questions like this?" Bam, bam, bam. The first ten I saw were all pretty good. I’m like, "Yep, yep, yep, yep. That’s all legit." And then the implications for Khan Academy started to go through my mind. And then we did start to get into some of its hallucinations and some of its math errors in those early days. But that weekend, they gave me and our chief learning officer access on Slack, and I couldn’t sleep, I was having these ‘in the rabbit hole’ conversations with it. And then we had a hackathon for our team; we got about 40 people on our team under NDA, and we said “just come up with stuff.” We were having the debates that everyone was having around, well, the information is not 100% airtight. The math isn’t great right now. The costs are not trivial. It can introduce bias. What’s the safety? The use of people’s information, etc.? But then we were starting to get it to work well as something that helps you answer questions while you’re watching a video, to work well as a tutor. Every ten minutes we thought about it like, wait, it could also do this. It could also do that. It can also do that. And so we said, "Well, what if we could make it so you could talk to any historical character? What if you could make it so it gets into a debate with you? What if you could make it so it doesn’t write your paper, but it writes it with you? What if it could do lesson plans for teachers?" It could be the end of static curricula. The imagination kept going. By December, January we had our team in full, rapid prototyping mode. Just recently we launched Khanmigo. So far we’re starting to titrate it out to the world, giving people access to it. But the feedback is very promising. What we’re hearing overwhelmingly from social media and the press is that they’re really happy that we’ve engaged in this and that we’re taking a safe approach where parents and teachers can monitor it. We have a moderation filter. OpenAI has also gone through great pains to make sure that things stay appropriate. You and I have talked extensively about the math issue, and we’ve done some things that I think make it quite robust on top of the things that the model does. And the costs are coming down dramatically.
BILL GATES: I mean, that’s really impressive.
SAL KHAN: Even ChatGPT isn’t bad, and then GPT-4 is dramatically better. It makes mistakes, to be clear. They both make mistakes. But one of the things we realized is when you do math, especially if I’m tutoring you, let’s say you do some work, I don’t just immediately say, "Correct, incorrect." I say, "Well, let me see this." Okay. Let me see what he did. Okay. Okay. Okay. “Yeah, yeah. Good job.” And one of the hacks – I don’t think it’s actually a hack, I think it’s a principle we’re doing – which is we weren’t getting good results when we just asked it to decide, when it acts as a tutor, whether a student is right or wrong. But as soon as we said, you know what, construct your thought, and these thoughts are private to you, write that down first, and then evaluate the student’s response to your thoughts. And then say something publicly to the student, then the math improved dramatically. So it is funny. As soon as one day you think like, "Oh, this is so not like a human being,” and then the next year it’s like, wow, that’s kind of how we operate. We kind of need that thought before we can talk.
BILL GATES: The way I think of it is, it’s like a human that’s not very good with context, that the math context we know is a special context of ‘check your answer.’ It’s also sometimes, even in conversation, if you get it into a mode where it’s telling jokes, with humans, we have kind of a look, or the new question is quite different. It thinks it’s supposed to just keep telling jokes, and you almost have to do a reset to get it out of this “Hey, everything is a joke”-type mode. So it’s like kind of a very naive person in terms of all these different contexts we’re in. My favorite one is where it tries to do Sudoku, which it can’t do. And you point out, "Hey, that’s not a good solution.” And it says, "Oh, I must have mistyped,” because it’s like, where is the typewriter? You have a typewriter?
SAL KHAN: [laughs] I like how it gets defensive.
BILL GATES: It sees how humans deal with being accused, “My dog ate my homework,” or something like that. It’s kind of human, but in a naïve, untrained way. So if we think about the next ten years, I’m sort of back to my optimism that I had when the Gates Foundation first got into education, that both the absolute level of learning and the gap with lower-income, minority students, that with these new tools can both close the gap and raise up the overall level of achievement.
SAL KHAN: After the pandemic, tens of billions of dollars came in from the federal government to address learning loss. Everyone looked at these decades-old studies around high-dosage tutoring and said, "Oh, we can kind of afford to do tutoring now." And so they hired a bunch of tutoring companies and this and that. And it’s not the tutoring companies’ fault. But what you’re seeing now is just another example that we often see in the school system like, "Wow, this is a good idea, but we’re not really seeing it move the dial." The retrospective people were saying, "Well, in order for it to really work, it’s got to be connected to what the kids are doing in school, and ideally it should be happening while they’re doing it in school." That’s almost impossible logistically to do and expensive if you wanted to do it with live tutors. But Khanmigo, that is, the tutor that we’re putting in Khan Academy, it, in the moment, can act like a fairly good human tutor. There’s moments with it that I think would pass the Turing test, where you’d think that there’s a good human on the other side of the chat. And I think that has a good chance of driving engagement because not only can it tutor, you can ask it why you need to learn this. It says, “Well, what do you care about?” And you say, "Well, I care about this or that." It’s like, "Well, if you care about this, then you definitely need to learn this, and here’s some other things to think about." It honestly answers it better than most humans would. That is going to drive a whole other level of engagement, a whole other level of, "Hey, a student is stuck in algebra because they have some gaps from seventh grade." That’s very hard for one teacher with 30 kids to diagnose. But now to be able to intelligently, with traditional tutoring interactions, say, "Hey, are you having trouble with the negative number here? Let’s review that a little bit." I think that can drive a ton of engagement. I think honestly, just feeling more connected can drive engagement. I’m actually really excited about the next phase of Khanmigo. It’s going to be Khanmigo facilitating interactions inside of a classroom.
BILL GATES: With a group?
SAL KHAN: With groups. So imagine a teacher just says, "Hey, Khanmigo, take my class of 30, put them into ten groups of three, and I want every group to do this problem. And then I want the students to talk to each other about it. And you’re the facilitator." We’ve run those types of experiments and they’re surprisingly fun where you can do that. It’s not just in math. You could do that in science, where it pushes the students. How does entropy in chemistry connect to entropy in information theory or computer science? And all of these types of things, and the humanities? I do think all of these are correlated, too. When students get confidence in one part of their academics, it drives all the other ones. But now a teacher can say, "Hey, Khanmigo, ask all my students to give their reflection on this issue." And immediately it can tell the teacher, "Hey" – and this isn’t science fiction. I think this is like six months away – it can say, "This is what everyone did. Here’s some exemplars. Some of your kids tend to be confused here. This student seems a little bit disengaged." I think we’re going to see it next year; while the teacher is lecturing or whatever they’re doing, if you’re confused by something they just said, you just ask the bot in real time. It’s like whispering to your friend without disrupting the classroom, but then Khanmigo will tell the teacher, "Hey, three kids just asked me this question. I answered it for you. They’re good now. But you might want to double click on that a little bit." We have another nonprofit called Schoolhouse.World, which is about peer-to-peer tutoring. We’re already using AI to evaluate the transcripts and give the tutors feedback on what they can do better. I could see it being a little bit of a real time, like, "Hey, you haven’t called on this student, or here’s a worked example for you." It’s going to be a wild couple of years, but hopefully positive.
BILL GATES: And one thing I always underestimated is how valuable it is for most students to have kind of a social experience. How do you create a conceptually rich but socially engaging way of talking about these math things?
SAL KHAN: No, 100%. Some of my best memories are sitting with friends and working on a particularly hard problem, and somebody has an epiphany; the more that we can try to replicate those things. The model we use is a tutor for every student, and it could be a teaching assistant or an army of teaching assistants for every teacher. I think every teacher on the planet would love to have one, two, three amazing teaching assistants in that classroom that say, "Hey, let’s facilitate these breakouts. Let’s do focused interventions with different students. Let’s make them explain the math to each other."
One challenge we’ve always had is that when we bring new technology into the
classroom, if we don’t do it well, the teacher feels like, "Oh, you’re trying to denigrate my creativity or freedom or you’re suggesting I’m not capable on my own." And yet we all know teachers are heroic, one of the most important, hardest jobs in the world. So do you feel like we’re doing better at drawing both their input and showing that, okay, they are 100% at the center of the best solution?
SAL KHAN: Yes. I think not only is this about making sure that everyone understands their critical role, but it’s also one of the reasons why you have such a high attrition in the profession. We can talk about pay and compensation, all of that, but I think the biggest thing is it can be a very lonely and a very tiring job. Most of us with ‘desk jobs,’ we don’t have to be ‘on’ continuously.
BILL GATES: Yes. Nobody’s trying to disrupt.
SAL KHAN: No one’s trying to disrupt, or chewing gum, or looking at you. Even when we’re presenting, we’re usually presenting to people who at least pretend to be interested, right?
BILL GATES: [laughs] They’re told to be. Yes.
SAL KHAN: Exactly. They got the memo. Well, in a classroom, the kids aren’t always giving you their most attentive look. And that is so draining, energy-wise. We imagine the classroom of the future, and a lot of teachers are already doing this, is to make the students do most of the things most of the time, and that the teacher is the architect, is the conductor of what’s going on. I don’t think anyone in an orchestra would think that the conductor is not an important role. The conductor is making sure that the orchestra is happening together and that everything is fitting together, that it’s controlled chaos. Then I think everyone benefits. The kids have more fun, they learn more, and the teachers have more fun. Instead of one teacher with 30, you should have two teachers with 60, or three teachers with 90 so that they also feel less lonely, which you can start to do if you take the lecture out of the classroom.
What do you think is going to happen in AI, in education and work?
BILL GATES: Well, in the education piece, I actually think reading and writing where the software tools have not been that great. Very few students get feedback on an essay, that this could be clearer, you really skipped this piece and the reasoning. I do think the AI will be like a great high school teacher who really marks your essay and you go back and think, okay, I need to step up there. The hardest question to answer is the job market. Now, we’re not going to ever have too many teachers because in the worst case, we shrink class sizes and give more attention. The demand is pretty infinite there. I do a lot of my work in countries where the shortage of doctors and teachers is so acute that the idea of, okay, the AI is too good, we don’t really run into that. But the job market, I think that one, it’s very hard to predict how it’ll shift things. I don’t think students should feel like,
“Okay, I need to learn less. I don’t need to multiply because computers are so good at multiplication.” The new areas of job demand and how we shape this is going to be fascinating to see.
SAL KHAN: If you were to advise a parent or a student who’s, let’s say, a teenager who is figuring out what to do with their life, given everything we talked about AI, job market, what would you tell them?
It’s great if you learn enough that you find something that’s particularly fascinating
to you. I have an orientation towards the sciences. I still think helping these AI things be better, we need that. The climate work we need is very, very scientifically intense. Also anything in medicine. I have one child who wants to do political policy, including health care. I have one who’s going to be a doctor. I have one who wants to be the CEO of the hospital. All of them can contribute in very different ways. What they pick kind of matches the differences in what they enjoy, what they’re really good at. I would love it if some really smart people would go into politics. I know that’s not attractive, but I think we need a new generation of science-literate and reaching across the divide we have. God knows, young people better come in and help us out there.
SAL KHAN: So, Bill, what teachers really pop out in your mind that really affected you?
BILL GATES: Well, early on in math, I was kind of lazy, and a teacher in eighth grade said, "How come you’re so lazy? You could be really good at this." I said, "But we’re not doing anything interesting." He would give me books to read and help really push me quite a bit. It made a huge difference that he thought I was wasting my time. It changed my whole view of education. I kind of had this view that the less effort you put in, the cooler you were.
SAL KHAN: [laughs] That’s right, very good.
BILL GATES: Do you have a particular teacher that helped motivate you?
SAL KHAN: I have several, probably three or four that pop in my mind. There’s second grade, Ms. Krauss and Ms. Roussel. It was kind of this enrichment gifted program. They were the first teachers that really said, "Hey, what are you interested in?" And that was the first time I experienced personalized education. Jefferson Parish School System in Louisiana, I’ve got to give credit. It was a really great experience. Then I remember Ms. Ellis in fifth grade. She ran her fifth grade social studies class like what I now know is a college seminar. She would just peel an orange and just ask us questions. I would look forward to it. Also Dr. Harris Antonia. When I was in high school, I took courses at the University of New Orleans, which was the local college there. He was my differential equations teacher. When he found out that I didn’t have a computer at home, he’s like, "Well, you need a computer." He got me a research job at the University of New Orleans when I was 15 years old. That’s the first time I could program. I really got to work with real computers. If I didn’t have that experience, I don’t know what would have happened after that.
BILL GATES: If you had one magic wish to master a subject without any studying, which would you choose?
SAL KHAN: I would say it would either be quantum physics or piano. [laughs]
BILL GATES: I’m embarrassed that even though I know all these computer languages, and I took Latin and Greek, I don’t speak any real languages other than English. Maybe if it was easy to learn and I could just use my wish on it, I might choose Chinese, because I do think helping China and the US get along is going to be pretty important.
[music]
So I said, before you came, if you had a record that you wanted to bring, to please
share it with me.
SAL KHAN: What we’re going to play is Bob Marley’s “Redemption Song,” which is, as you mentioned, in my high school rock star days, “Redemption Song” was one of the first songs that I learned on guitar. It always spoke to me. It’s about bondage of many kinds, both physical and mental bondage, and kind of breaking free of it. So I’ll play it.
BILL GATES: Cool.
[music –"Redemption Song" by Bob Marley]
BILL GATES: So great. This is just a classic.
[music]
BILL GATES: So, if you do a web search on Sal Khan, you might get some of this guy. Do you ever get confused with Salman Khan?
SAL KHAN: I do. In fact, in the early days of Khan Academy, I got letters from some of his fans saying I’ve always been in love with you, I didn’t know you could do math, and all that. And it’s funny because, you know, us South Asians like a good kind of cheesy thing. So when I went to India in 2015, we had a live, on national television, interview between the two of us.
BILL GATES: Oh, you’re kidding.
SAL KHAN: Just by virtue of us having the same name. But I do watch more Bollywood movies than folks suspect.
BILL GATES: [laughs] Uh-oh.
SAL KHAN: My wife grew up in Pakistan, so people think that she made me watch the Bollywood movies, but it’s actually the other way around. I’m the one that made her watch the Bollywood. So I was very aware of him. It was kind of a fun thing to be in the same room and share the same name.
BILL GATES: Do you watch them in Hindi or English?
SAL KHAN: I can understand Hindi, which I have no business understanding. My family, our mother tongue is Bengali.
BILL GATES: Oh, Bengali.
SAL KHAN: Yeah.
BILL GATES: Oh, wow. And your wife?
SAL KHAN: My wife’s mother tongue, they’re originally Gujarati, but she spoke Urdu growing up in Pakistan. I learned Hindi mainly hearing my mom gossip on the phone when I was growing up.
Bollywood movies, that’s my India now. Like a little bit Hindi and Urdu spoken is almost the same thing. So I know enough to get by.
English
中文 
