Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

211–220 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#211

Earlier quoted context omitted.

Interesting. Ages?

This was probably middle school.

What do you think about it? I think that the ability to study by yourself is probably the most valuable thing one could get from school and most kids usually don't. Your setup would suggest that demanding that a kids know how to study by themselves is going mainstream...?

Your answer - middle school - I find it extremely young for this, which is extra interesting.

Re: AI boosted homework scores, then exam scores dropped: study

#212
post #163

All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates. We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional mat…

What you described about exams is like how I approach most certification exams: go through a test bank of questions until I get 80% then take the real test.

Re: AI boosted homework scores, then exam scores dropped: study

#213

Earlier quoted context omitted.

> Homework should be optional material for self study At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams. But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I k…

I've given this advice many times, and the ones who followed it did well. I only added "leave your laptop in your dorm room" for modern times. The general pattern at Caltech was 2 hours of study for every hour of lecture. Which was quite a shock to me.

I'll add one more thing to this, given that my experience as a math major at SIU was that each hour of lecture was expected to carry four hours of studying: read and summarize the course's texts (additionally, for math courses, attempt the problems) before lecture.

I found that (1) I didn't need to take anywhere near as many notes during lecture and (2) I could ask way, way, way more relevant questions.

This also helped tremendously when it came to studying for the actuarial exams.

Re: AI boosted homework scores, then exam scores dropped: study

#214
post #157

Earlier quoted context omitted.

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generat…

well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure." they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for. there can be many questions of pedagogy. one of them is, what do our exams…

The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal, students who studies more have higher exam scores.

Re: AI boosted homework scores, then exam scores dropped: study

#215

Earlier quoted context omitted.

The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs. The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...

I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All…

> I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place?

Citation needed? I have no clue where you got this from. I hadn't even heard of it as a conjecture, let alone as something anyone accepted, let alone as generally accepted...

Re: AI boosted homework scores, then exam scores dropped: study

#216
post #163

All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates. We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional mat…

> Homework should be optional material for self study At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams. But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I k…

Naval Nuclear Power School had (maybe still does) this model 20 years ago as well. Pass rate for my class cohort was 28% over the entire 2 year pipeline.

We didn't call them retch sessions, and they were taught by the instructors though. We also were encouraged to peer tutor and since we were all restricted to one building the homework was always group work allowed.

Also had badges to track time spent in the building for required study hours, though some people gave up and just slept at their desks when they started sliding down the grade scale and the hours racked up.

Generally I think I did 30 hours of studying/homework (went up and down depending on what was being taught, but was around that) a week (for 12-15 hours of actual lecturing), with some of my friends putting in 50% more. Generally the only day we weren't there was Saturdays. Most of the day Sunday was usually spent in class preparing for the next week.

Re: AI boosted homework scores, then exam scores dropped: study

#217

Earlier quoted context omitted.

> well most university exams are designed to measure how much you study. Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.

at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, y…

Because "know more" isn't actionable. Knowing more is achieved by studying but not necessarily more time spent studying, but well spent effort. Staring at the page for hours and saying "I don't understand" doesn't help. Solve exercise problems, explain the material to fellow students, discuss it with them, make mind maps, bullet point summaries, work through derivations step by step, etc. There are many techniques.

At the end of the day though what matters is what you know. Furthermore, if it's a serious subject, it shouldn't matter whether you learned it from this teacher or from another school and teacher, as long as your knowledge is correct. Knowing the idiosyncracies of this particular teacher should not factor into the grade. A serious subject can be learned on one continent and examined on another. Bullshit courses are all about learning pet peeves and hobby horses of a particular teacher.

Re: AI boosted homework scores, then exam scores dropped: study

#218

Earlier quoted context omitted.

The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs. The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...

I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All…

How do you expect students to learn anything when:

1. They don't do any homework.

2. All the in-class time is split between the teacher babysitting and playing social worker to problem students, and lecturing, with little to no opportunity to actually practice what they've learned?

I understand that some students don't have home environments that are conductive to doing homework well. I understand that some students are enrolled in five hours a day of extracurricular university-application-padding activities. I understand that some students have incredibly poor screen discipline and impulse control.

But I don't understand that anyone has magically figured out how to teach complicated things to students, and have it stick without them spending a lot of time practicing what they are learning.

As anyone who has tried to do something hard knows, the first step to being good at something is to spend a lot of time being pretty shit at it.

A student who has written and received feedback on 500,000 written words is going to be way better at writing than that same student who wrote 50,000, just like someone who has put 5,000 hours of focused practice into playing the piano is going to be better than my dumb ass, who has only put 100 hours in.

(If you found the solution to get good at stuff without practicing it, I'd love to get good at piano without putting any homework in on it.)

Re: AI boosted homework scores, then exam scores dropped: study

#219

Earlier quoted context omitted.

> well most university exams are designed to measure how much you study. Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.

at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, y…

You become a person who knows more, by studying more about the things you need to know.

Re: AI boosted homework scores, then exam scores dropped: study

#220

Earlier quoted context omitted.

> well most university exams are designed to measure how much you study. Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.

at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, y…

I would usually say something along the lines of “everything we covered is on the table” or “everything we covered since the last exam is on the table” depending on the nature of the test. That’s the same message as “know more” but I think it sounds politer.
Post reply on HN