Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are the ultimate demoware

blog.charliemeyer.co

Post reply on HN
Print view View original

41–50 of 128 posts

Re: LLMs are the ultimate demoware

#41

Post by voidhorse »

simonw wrote: post #29
getnormality wrote: post #21

Earlier quoted context omitted.

It's nice that you think it's clear and responsive, but I think it [1] needs to be validated by an expert in both the material and education. Or we need some way to show that people have actually learned the topic. People sometimes prefer explanations that are intuitive and familiar but not accurate. Meanwhile, there are math education resources like iXL that maybe cost a little money but the lessons and practice pro…

What makes you think https://www.mathacademy.com/faq hadn't been evaluated by experts? That appears to be their whole thing, and they've been in business for longer than LLMs have been around.

I think parent was clearly referring to LLM use, and not math academy.

Re: LLMs are the ultimate demoware

#43

Post by badsectoracula »

tptacek wrote: post #13
empath75 wrote: post #10

Earlier quoted context omitted.

It seems very hard to maintain the belief that LLMs are useless in the face of the fact that millions of people are using them. It's very much "nobody goes there anymore, it's too crowded"

I think you'd be crazy to say LLMs are blockchain-style hype when it comes to software development but I don't begrudge anybody who believes they're not currently workable for the kinds of problems they work on; I think reasonable people can disagree about how ready for prime time they are for production software development. But for math tutoring? If you claim LLM math tutoring is demoware, you're very clearly telli…

I wouldn't trust the LLM's raw output to be correct, but math is provable and if there was a filter between the LLM's output (which would be in some more rigid/structured format, not free form text) and whatever the user sees that tries to prove the LLM's output is correct (and try again if it goes wrong[0]), then i can see LLMs being perfectly fine for that.

In fact i'd say in general anything that LLMs produce that can be "statically checked" in some way, can be fine to rely on. You most likely need more than a chat interface though, but i think in general it is plausible for such solutions to exit.

[0] hopefully it wont end up always failing, ending up in an infinite loop :-P

Re: LLMs are the ultimate demoware

#44

Post by turnsout »

dingnuts wrote: post #15
turnsout wrote: post #8

This is such a weak take to read while I have Claude Code running in the background creating a new database migration for a feature we're building

If Claude Code can do it what do we need you for?

Someone has to come up with the idea for app, have a vision for what it needs to be, and continually push it forward without going off-course.

Re: LLMs are the ultimate demoware

#45

Post by getnormality »

simonw wrote: post #29
getnormality wrote: post #21

Earlier quoted context omitted.

It's nice that you think it's clear and responsive, but I think it [1] needs to be validated by an expert in both the material and education. Or we need some way to show that people have actually learned the topic. People sometimes prefer explanations that are intuitive and familiar but not accurate. Meanwhile, there are math education resources like iXL that maybe cost a little money but the lessons and practice pro…

What makes you think https://www.mathacademy.com/faq hadn't been evaluated by experts? That appears to be their whole thing, and they've been in business for longer than LLMs have been around.

I think before that question is useful to ask, we have to know if that FAQ even says anything about LLM-based tutoring. After a few minutes of research, I can't find any evidence that Math Academy offers LLM-based tutoring.

Re: LLMs are the ultimate demoware

#46

Post by AnotherGoodName »

getnormality wrote: post #21
tptacek wrote: post #7

It's wild to me that, of all the things to call LLMs out for, this piece has chosen to include math tutoring. I've been doing Math Academy for a bit over 6 months now, going from (essentially) Algebra II through Calc II (integration by parts, arc lengths, Taylor expansions) and LLMs have been a huge part of what has made that effective: * Clear explanation of concepts that respond to questions and reformulate when th…

It's nice that you think it's clear and responsive, but I think it [1] needs to be validated by an expert in both the material and education. Or we need some way to show that people have actually learned the topic. People sometimes prefer explanations that are intuitive and familiar but not accurate. Meanwhile, there are math education resources like iXL that maybe cost a little money but the lessons and practice pro…

>I think it needs to be validated by an expert in both the material and education

LLMs getting it wrong is terrible when it matters but i also don't think it's a huge problem when it comes to acting as an additional resource to learning. Here the parent is using a lesson plan that costs money and using LLM for a little more explanation. It's similar to using web search on a topic and sometimes you get a hit, sometimes you don't.

Asking LLMs for numeric examples of complex maths sometimes fails. It's easy to spot and no great loss. When it works though it's extremely helpful to follow through.

Re: LLMs are the ultimate demoware

#47

Post by voidhorse »

getnormality wrote: post #38
fourside wrote: post #30

Earlier quoted context omitted.

Not sure the condescending tone is really necessary. I’d agree with you if the parent comment was saying they asked an LLM to create a math curriculum and problems for them. But they’re using an established app created by a math major and then using LLMs to ask questions. It’s easier to validate the responses you get back in those cases.

I think students are not a reliable source of information about the effectiveness of LLM tutoring. There is no 100% nice way to say this, but I did my best. You're free to disagree, but I think the tone criticism is off-base.

I agree with you completely. People mistake the impression of learning for learning itself super easily. This is why we have examinations and other tests of mastery, after all. I think using LLMs for generating exams or supplementary material is great, but using them to develop accurate understanding that would actually turn into long term retention seems dubious to me.

Re: LLMs are the ultimate demoware

#48

Post by bgwalter »

simonw wrote: post #19
dingnuts wrote: post #15

Earlier quoted context omitted.

If Claude Code can do it what do we need you for?

Someone needs to know what a "database migration" is in order to ask Claude to build one. I think a lot of people are massively underestimating how much knowledge and skill is needed in software engineering beyond typing code into a text editor.

You know that Linus was at one point "typing code into an editor" to review other people's patches because he found it easier to catch mistakes that way.

If your field of software engineering is so simple that you can survive on code snippets stolen from other people, great. Please do not generalize that.

Re: LLMs are the ultimate demoware

#49

Post by baklavaEmperor »

getnormality wrote: post #21
tptacek wrote: post #7

It's wild to me that, of all the things to call LLMs out for, this piece has chosen to include math tutoring. I've been doing Math Academy for a bit over 6 months now, going from (essentially) Algebra II through Calc II (integration by parts, arc lengths, Taylor expansions) and LLMs have been a huge part of what has made that effective: * Clear explanation of concepts that respond to questions and reformulate when th…

It's nice that you think it's clear and responsive, but I think it [1] needs to be validated by an expert in both the material and education. Or we need some way to show that people have actually learned the topic. People sometimes prefer explanations that are intuitive and familiar but not accurate. Meanwhile, there are math education resources like iXL that maybe cost a little money but the lessons and practice pro…

It’s interesting how people insist math requires expert validation when it’s literally the most self validating subject there is. The instinct to gatekeep even something as mechanistically checkable as algebra says more about insecurity in education than it does about rigor.

Re: LLMs are the ultimate demoware

#50

Post by simonw »

getnormality wrote: post #45
simonw wrote: post #29

Earlier quoted context omitted.

What makes you think https://www.mathacademy.com/faq hadn't been evaluated by experts? That appears to be their whole thing, and they've been in business for longer than LLMs have been around.

I think before that question is useful to ask, we have to know if that FAQ even says anything about LLM-based tutoring. After a few minutes of research, I can't find any evidence that Math Academy offers LLM-based tutoring.

This was linked from the homepage: https://www.mathacademy.com/how-our-ai-works

But more importantly if tptacek says they use LLMs and is a user of the platform that's good enough for me.

Post reply on HN