Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

intextbooks.science.uu.nl

Post reply on HN
Print view View original

71–80 of 126 posts

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#71

Post by fn-mote »

fragmede wrote: post #45
Buttons840 wrote: post #41

Earlier quoted context omitted.

Again, I really think this is a viewpoint we've talked ourselves into to help us feel better about how cumbersome creating the cards are. I'm willing to grant that there is some value in choosing what to put in the cards, but most of the awkwardness around making cards is UI related. Nobody creates cards on their phone, or while they're walking (AI could do both of these) - people create cards sitting at their comput…

> Nobody creates cards on their phone, or while they're walking Wait, when are you doing it then? No wonder you think it sucks! Adopt some modern tools, yo. Use Anki, or vibe code your own app.

> Use Anki

Anki is great for studying, but the card creation experience sucks. To be specific: I found creating any custom card type immediately dropped me into the bowels of CSS. It felt like writing HTML by hand.

Is there any facility for re-using shared pieces?

I felt like it needed a static site generator type tool to move up a layer of abstraction and reduce the copying of chunks into my card. Is there one? Please mention if so.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#72

Post by gravypod »

fn-mote wrote: post #70
chasd00 wrote: post #38

Earlier quoted context omitted.

I work in consulting and one of my projects is piloting an AI use case for a department within one of my clients. On a discovery call someone casually brought up that they bought a reMarkable notebook themselves and were wondering if it could be integrated into the use case. It really got me thinking. Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back t…

> Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad What does “bridge a student’s writing” mean?? If this is a real argument it needs to be clearer. What’s the functional difference between a Remarkable and an iPad? The former is less responsive, costs less, and has better battery life, right? I really don’t see how that’s significa…

I assume that the reMarkable here would mainly just be used to capture the writing. It could then be examined by an LLM asynchronously.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#73

Post by SgtBastard »

johnfn wrote: post #60
klustregrif wrote: post #59

A lot of pessimism in the comments, but I am just happy that we are seeing some work towards bridging the 2 Sigma gap for regular education vs. elite private tutoring. I can't imagine that people assume it's the physical presence of the tutor that is making the difference, it has to come down to the personalisation and expertise which is exactly what AI can provide in a form. And yea it might not be "there" yet. But…

Tell me if I am oversimplifying, but I never understood the noise about the two sigma problem. Like, of course if you have a private tutor to immediately answer any question that pops into your head at the immediate moment you get confused, you are going to learn vastly more efficiently than in a large classroom where once you get confused you are likely to stay confused. To say nothing of how the pace will likely ei…

The “problem” is exactly as you’ve framed it: individual 1-1 tutoring unambiguously improves education outcomes by a significant (2 StDev’s in Blooms study) vs any other understood method but delivering this universally is (was?) infeasible.

Bloom looked at other methods used in concert to achieve similar improvements to “solve” the problem that could be delivered at scale.

LLMs may provide a new path to the 2-Sigma improvements without the same delivery problems.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#74

Post by naasking »

Buttons840 wrote: post #25
baq wrote: post #11

I'm on record saying that a system like this with some extra hardware (i.e. a way for the LLM to have live understanding of the student's paper notebook or handout which are being written in with a plain old pencil) combines the best of both worlds - individual tutoring with approximately zero screen time which scales linearly with the number of students. The role of the teacher or professor then becomes a manager of…

I would add that somewhere in there should be a spaced repetition algorithm. Spaced repetition is very effective, but it's really really clunky to use. My unpopular opinion is that we all have Stockholm syndrome when it comes to creating "cards", and people talk about how valuable creating cards is; but I think it stucks, it takes a lot of time. If AI is already teaching me math (let's say), it would be nice to tell…

> Spaced repetition is very effective, but it's really really clunky to use

Spaced repetition is just reviewing the same material periodically. It doesn't have to be a complicated system.

We already know how to learn and educate: spaced repetition (periodic review), and retrieval practice (frequent testing). This is how school used to be fore centuries; it's not sexy but it's effective.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#75

Post by [deleted] »

fn-mote wrote: post #70
chasd00 wrote: post #38

Earlier quoted context omitted.

I work in consulting and one of my projects is piloting an AI use case for a department within one of my clients. On a discovery call someone casually brought up that they bought a reMarkable notebook themselves and were wondering if it could be integrated into the use case. It really got me thinking. Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back t…

> Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad What does “bridge a student’s writing” mean?? If this is a real argument it needs to be clearer. What’s the functional difference between a Remarkable and an iPad? The former is less responsive, costs less, and has better battery life, right? I really don’t see how that’s significa…

[deleted]

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#76

Post by rightbyte »

p1necone wrote: post #68
radioactivist wrote: post #40

I am somewhat skeptical of this. First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement Second, trying to incorporate past grades into their modelling is not a substit…

It feels to me that the venn diagram between "students that fully engaged with the material" and "students that learned well from the material" is going to basically be a circle for any teaching method.

Yeah you'd need to compare this method with some classic method as a control. Otherwise "engagement" is just an indirect measure of how much the students studied.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#77

Post by bobajeff »

Even if the research is flawed I'm happy they are trying this. They are taking advantage of LLMs to have less rigid tests and also give feedback.

I think there is more potential applications possible with combining LLMs with reference/text books. Like how about an assistant that points you to the correct books/chapter/paragraph for the concept you need to understand better for a project you are working on? Or clarify any confusion you are having?

Like a human tutor but infinitely patient and non-judgy + search engine.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#78

Post by delis-thumbs-7e »

I currently study Multivariate Calculus by using very new and nodern method: I read the text book, while solving the examples of the general for,ulas, or try to come up with my own. Then I do a s$ht-ton of exercises. I only use LLM’s to quickly clarify confusing topics or notation, but not really much else. I cancelled my Claude subscription. Now I use just Mistral and local Vibethinker-3B, but they work just fine.

Earlier I used Claude by giving it the course material and asking it to generate me exercises (our cpurse work went way over my head) and yeah i learned to differentiate a gradient or Jacobian, but it was very shallow - I knew the formulas, but not what they meant or how to apple them correctly. After I just filled glaring holes I had in Univariate Calculus by readong and doing, I actually started to understand something.

Lon story short, in my experience Learning with LLM’s is ok with very unfamiliar material that is not too complex (there’s obvious problems of LLM’s themselves being pretty ghastly with maths sometimes), but at least it os not better than the traditional method of just putting your nose on the grimd stone.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#79

Post by dash2 »

radioactivist wrote: post #40

I am somewhat skeptical of this. First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement Second, trying to incorporate past grades into their modelling is not a substit…

Yeah, calling this an "effect size" is just nonsense, and it is alarming that educational software can get away with such poor statistical practice. I'm hoping this was just a student project.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#80

Post by chasd00 »

fn-mote wrote: post #70
chasd00 wrote: post #38

Earlier quoted context omitted.

I work in consulting and one of my projects is piloting an AI use case for a department within one of my clients. On a discovery call someone casually brought up that they bought a reMarkable notebook themselves and were wondering if it could be integrated into the use case. It really got me thinking. Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back t…

> Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad What does “bridge a student’s writing” mean?? If this is a real argument it needs to be clearer. What’s the functional difference between a Remarkable and an iPad? The former is less responsive, costs less, and has better battery life, right? I really don’t see how that’s significa…

I feel like an eink b/w device will work better because of the display and lack of distractions like on a full blown iPad. Seems like sink would be similar to a sheet of paper but digital so it could be sent to an LLM or some Other API. Just my two cents though, I don’t have a super strong opinion about it.
Post reply on HN