Why Sal Khan't: On Learning by Making but Teaching by Telling

When I’m understimulated by a teacher, my mind wanders. I’ll start thinking about something else entirely. When my mind comes back, I’ll have missed a couple minutes of content. I can overcome this with willpower, but doing so is exhausting.

Playing videos at 1.5-2x fixes the problem entirely. My attention is held without any effort.

If a concept is new or difficult enough, I just slow the video back down or pause it whenever I need to think things through.

> There has always been value in somebody showing you how to apply a math technique and practicing using it to solve problems.

Only if your goal is to teach people how to solve that particular problem. But it’s not the best way to teach mathematics.

For my money, the best way is problem based. I heard of one math teacher who had a video playing as the students walked in. The video was a weirdly shaped bucket being slowly filled by a hose. Students came in, saw him sitting up the back, got fidgety then someone asked “how long will this take??”. At that, he shot up and said “good question”! And they spent the rest of the class trying to estimate how long it would take for the bucket to be filled. Complete with diversions into flow rate, volume, and so on. He made the students ask the question. Then got the students to try and answer it themselves, helping them work through their own ideas on the board.

That’s beautiful teaching.

I looked at Khanmigo fairly early on. I tried it out and thought: this is ChatGPT with a system prompt. And not a good system prompt either. In addition to what felt like rote math it had some had some experiences, like you could "talk" to Plato or some other figure. Each of these was also pretty awful; simplistic prompts that spent all their time on guardrails and none on the experience itself. (What it the student thinks they are ACTUALLY talking to Plato?! Avoiding this seemed very important to the makers.)

In the more core experience they were super focused on not giving kids answers. I suppose built on a fear of subverting the teachers or classroom environment. But again it became a fixation that took all attention away from teaching itself.

But I figure you put something out in the world and see what happens. Yet I came back to it a year later and it was exactly the same. Nothing had changed. It baffled me then, and still baffles me... I had done enough to know the whole experience was easy to manipulate and yet I couldn't see any evidence of effort. It felt very much like "we tried nothing and it didn't work". Maybe I was missing something, I don't know.

I think there's a few different problems here:

1. I think the author is correct to highlight that underlying motivation, which is so important to teaching. We can focus on a high-minded Constructivist ideal of intrinsic motivation, but motivating students is also what grades are for, and classrooms, and learning with peers, having someone pay attention to your work, and so on. I do think that the AI chatbot is not capable of providing that. But the AI can still ENGAGE with that motivation. I have a feeling Khanmigo never did. I suspect some of their privacy controls also kept it from ever creating a good model of the learner.

2. The entire experience was very focused on answers, on solving problems. And yet it focused on that so reluctantly. The obsession with not giving away the answers had the unfortunate result in not engaging with anything but answers. It felt like it was always teasing at something it would withhold. But questions are cheap, especially rote questions, they shouldn't be treated as precious.

3. This may be unfair, but I infer the makers of Khanmigo did not build it with love and craft, which is unfortunate. I don't know enough to say why. Though there's a weird confidence to the product and to all of Khan Academy that I feel does not bring the necessary humility.

4. The chat experience was asymmetric in the wrong direction. In each turn the AI can and will spew out paragraphs of text, and the student types a couple words. Chatbots DO NOT have to work like this, they operate wonderfully with large chunks of user input. But you have to have the modality to receive it, the openness of experience to elicit that response, and the respect to make use of that response. Speech can be good, but it's a real challenge in a classroom environment (though if you are investing in Khanmigo, you can also invest in the other hardware to make it possible).

5. In some fairness, individual tutors don't represent any pedagogy at all. It's usually just someone making it up as they go. Motivation is a huge aspect, studying with someone sitting next to you, watching you study, will do wonders for keeping you focused! But also a tutor will have a natural theory of mind, hopefully the ability to see a student get the wrong answer and get inside the student's head to model their understanding and find a path to a correct understanding. That's a very challenging task, not beyond GPT-4 entirely but still quite challenging, especially if you are optimizing cost or time to response.

6. A good experience will do a TON of prework. I don't know if Khanmigo does that, if they've reshaped and expanded each educational module into a thorough playbook for the AI. That's where you have a chance to model things like wrong answers (there are actually SEVERAL books enumerating wrong answers in math), and use that to plan out responses.

If Khanmigo is not successful, I think that's on Khanmigo. Still teaching is very challenging. And maybe all of Khan Academy fails in this way: it does not seem to acknowledge the challenge of what they're trying to attempt. They seem far too confident in what they do, and not nearly creative enough.

What entrepreneurs in the US have made huge money from EdTech exits? I can’t think of any. So this critique doesn’t seem fair to me.

(India is another story. If you are referring to ByJu, which was absolutely a predatory enterprise, I agree with you)

Oh ok, got you. While it entirely differs on how different people receive ideas, for me, I just try to keep the end goal in mind (what are we ultimately moving towards) and just think of those repeated words as just small steps rather than just annoying repeating word.