"You have a 27% 'AI' issue in here" • Hacker News Board

Re: "You have a 27% 'AI' issue in here"

#191

Post by 9dev » Wed, Oct 04, 2023, 9:05 PM UTC

fnordpiglet wrote: post #107

Miraste wrote: post #21

Earlier quoted context omitted.

>if they cheat they are only cheating themselves anyways. I've heard people say this a lot. It's a very old-school liberal arts perspective that doesn't fit with the modern world. In practice, for most students, education is a third-place benefit of university behind 1. the certificate and 2. networking.

I find the “liberal arts” thing weird. First, as a scientist with an engineering degree, I definitely feel if you cheat through school you’re only cheating yourself. In the hard sciences a lack of knowledge and ability is pretty easy to suss out. The degree gets you the interview, the network might open a door or two, but the education is how you get the job. And once you’ve got it, if you didn’t learn anything, you…

> The degree gets you the interview, the network might open a door or two, but the education is how you get the job.

What a great way to respond. I couldn’t agree more.

Re: "You have a 27% 'AI' issue in here"

#192

Post by buildsjets » Wed, Oct 04, 2023, 9:05 PM UTC

gweinberg wrote: post #131

wccrawford wrote: post #16

When I was in college, they were going to put carpet in the science labs, and a few of us thought it was a really stupid, dangerous idea. We started a petition and asked for signatures at the on-campus cafe. After a few days, the administrators asked us into their office for a chat. They asked us stupid questions and tried to talk us out of our petition. When it didn't work, they eventually caved and didn't do the ca…

Who thought carpet was a good idea? Obviously not the professors, or anybody who new anything about chemistry labs.

This decision was not optimized for the operation of the chemistry lab. It was optimized to game the university's procurement process. It was a brilliant idea, if your objective was to exhaust your allocated facilities maintenance budget on something that you knew would have a very limited lifespan. That means you would have to request an even larger budget in just a year or two, in order to replace the destroyed carpet. Then your department gets both a new floor and a larger budget.

Re: "You have a 27% 'AI' issue in here"

#193

Post by 9dev » Wed, Oct 04, 2023, 9:12 PM UTC

lozenge wrote: post #144

giarc wrote: post #18

If you submit a word document, is there some way to detect if the words were typed out, how long the document was open etc? Basically trying to detect if the student pasted the text? Obviously students would get around this simply by typing it out word for word, but could work here.

If you use Google Docs, the edit history would pretty clearly the progress and different drafts. I guess eventually somebody will prompt ChatGPT in the right way to create realistic drafts and edits as well.

Watch me Place my tablet with ChatGPT next to my laptop and type the essay off of that into Google Docs on the laptop. Whatever contrived solution you come up with, students will find ingenious solutions around them.

Re: "You have a 27% 'AI' issue in here"

#194

Post by em-bee » Wed, Oct 04, 2023, 9:15 PM UTC

floren wrote: post #82

MisterBastahrd wrote: post #79

Earlier quoted context omitted.

The main reason I am not an Eagle Scout is because I took two classes at a winter camp for merit badges, did everything from signup and turned in every bit of work, and then at the end of the class they told me they had no record of me being in it. I did all the work for every other part of that award, but I flat out refused to redo those badges again. I've never jumped through hoops to make up for the failures of la…

Well, you're an Eagle Scout in everything but name, and honestly that might be better -- you get the personal satisfaction, but you don't do the thing all of us find extremely tedious about Eagle Scouts, which is constantly mentioning (in your 40s) how you're an Eagle Scout.

doing work and not having it recognized but in fact rejected because someone somewhere lost their notes is not the path to personal satisfaction.

a lot of merit badge work is based on trust. it should not be to hard to ask a few questions about the work done to be able to judge if it was really done.

most of this should be in the control of the scout masters. if they can't trust their scouts to be honest about this and work out a compromise then their troop has a real problem

Re: "You have a 27% 'AI' issue in here"

#195

Post by OkayPhysicist » Wed, Oct 04, 2023, 9:15 PM UTC

ineptech wrote: post #19

HDThoreaun wrote: post #4

Earlier quoted context omitted.

I doubt oral examinations come back in general. The undergrad model relys on massive classes, I don’t see how you can do oral exams in those 300 person classes without massive cheating as the first students tell everyone else what to expect.

Is there any value to those lectures, nowadays? I can see a PhD lecture on the fundamentals of virtually any academic topic, for free and at my leisure, on youtube and similar sites. Waking up at 8am to attend an in-person Geology 117 lecture with 300 other freshmen is probably as outdated as a fax machine. Perhaps the next step is, education that isn't based on personal instruction isn't rare enough to be worth payi…

> Is there any value to those lectures, nowadays?

Absolutely. Even in the largest lectures, people can ask clarifying questions, live, from the subject matter expert. Likewise, the professor can look at the students' body language and realize that they've lost a significant portion of the class, and stop, slow down, and maybe prompt the class for questions.

There is an art to a good lecture, and it's definitely harder the larger the class, but good lectures cannot be replaced by recordings.

Re: "You have a 27% 'AI' issue in here"

#196

Post by OkayPhysicist » Wed, Oct 04, 2023, 9:21 PM UTC

Tijdreiziger wrote: post #138

specproc wrote: post #27

Earlier quoted context omitted.

I'm assuming this is America? 300 person classes is insane, particular how much debt's involved. I've studied in the UK and EU, and I don't think I've ever been in a class larger than 40.

I have a Dutch CS degree, we had about 200 new freshmen every year. 40 seems excessively low for popular degrees.

You can have multiple sections of the same course. At the university I went to, there were about ~150 CS students per year. The very largest classes were 50 students each, with 3 sections of the same course, sometimes taught by the same professor, sometimes taught by different professors. As you progressed through the program, the number of "every CS major takes this course" courses reduced, and most classes were 20-something students.

Re: "You have a 27% 'AI' issue in here"

#197

Post by 2ap » Wed, Oct 04, 2023, 9:21 PM UTC

dan-robertson wrote: post #173

2ap wrote: post #109

I am an educator at a UK university. The essay is rapidly ceasing to be an appropriate way to assess students knowledge and critical thinking. We regularly organise in person face to face practical exams for our entire several hundred strong year group of undergraduates. It is possible to do assessment properly if the will is there.

When I was a student at a uk university, essays were, I think, written in examinations where one can’t easily use AI, as were most of the other things one would need for a degree. There was a small coursework component of my course and I think others. But I’m still a bit surprised — surely if you are getting a coursework essay from a history or English student or similar, the AI will be a worse writer and won’t have…

I agree - out of the box, a LLM will write an extremely poor essay. But that is not how to use the LLM. WHat you do, is give the topic to the LLM, and say, "with this topic, what are 20 titles that I could have for essays." Then, you pick the best title, and you then say, "I am writing an essay with the title , what should the central argument for the essay be?" You then ask for a list of bullet points for the central theme of each paragraph to elaborate on that argument, then you ask it to write each paragraph individually, then you put the pieces together, and ask it to proof read it's own work and finally you add in some references of your own. Within 1/2 an hour, you have a passable essay. And if you then iterate on it with the LLM you can fairly soon have a half decent one, especially if it is a well trodden area (presumably with lots of training data that the LLM draws on).

I tried to get students to critique things, but even then you can put in the text to critique to a LLM with a long enough context, and the LLM will kick off with a passable critique, if you iterate with it enough.

So, even though they'll never get a top mark, they will still be able to get through the assessment. So I don't set essays any more.

Re: "You have a 27% 'AI' issue in here"

#198

Post by OkayPhysicist » Wed, Oct 04, 2023, 9:23 PM UTC

petemir wrote: post #14

HDThoreaun wrote: post #4

Earlier quoted context omitted.

I doubt oral examinations come back in general. The undergrad model relys on massive classes, I don’t see how you can do oral exams in those 300 person classes without massive cheating as the first students tell everyone else what to expect.

Furthermore oral examinations do not scale. We had to do them during covid times and it was gruesome, spending full days in meetings and losing a full week with only 70-80 students.

It scales by having more, smaller classes, with more teachers.

Re: "You have a 27% 'AI' issue in here"

#199

Post by devilbunny » Wed, Oct 04, 2023, 9:28 PM UTC

amalcon wrote: post #141

slaymaker1907 wrote: post #121

Earlier quoted context omitted.

You're right about oral examinations, but written _proctored_ exams would also solve a lot of these problems, particularly if you require it to be handwritten. Having take home essays be handwritten would also help deter using AI by making it much more inconvenient to use compared to just doing the assignment in the first place.

I went to high school (secondary school) back when it was pretty normal to forbid students from typing assignments because "not everyone has a computer". I happen to be personally responsible, via my terrible handwriting, for at least two teachers dropping that policy. At least one other just gave me a specific exception in the name of "accessibility".

A student a year ahead of me had atrocious handwriting. They allowed him to type up his AP tests after the answer period and included the typewritten sheets along with his handwritten booklet (so the graders could confirm he hadn’t altered his answers).

Re: "You have a 27% 'AI' issue in here"

#200

Post by fluoridation » Wed, Oct 04, 2023, 9:28 PM UTC

godelski wrote: post #103

garciasn wrote: post #81

Earlier quoted context omitted.

The council-level Eagle Board of Reviewers likely didn't know the individual scouts and were simply applying their individual knowledge about the content reviewed compared to the whole and made, in this case, poor assumptions about the quality of writing they should expect. As it relates to academia today, this is now happening at a larger scale with folks lacking more trust, not having to think critically about the…

I've started referring to this phenomena as "Goodhart's Hell." We're living in a world with quickly increasing complexity. Variance in samples is much larger because our reach in what we sample is larger (we talk to people far from our homes, with different cultures, interpretations of words, language patterns, behaviors, ethics, etc). But we've responded to this only by simplifying our evaluations. We've created mor…

In part it's due to requirements of transparency. If some stranger is going to be evaluating you, a simple way to ensure that you're not being evaluated unfairly is to have the evaluation criteria be very simple, almost to the point of being mechanistic. If you're part of a small enough community it's likely that your evaluator, their overseer, and your peers all know you, so it's very difficult for unfairness to creep in unnoticed. If your community is gigantic everyone starts to be strangers to everyone else, so people can't intuitively know whether someone is being treated unfairly or not. When everyone knows each other it's much more difficult to game the system, because people can tell when you've been lazying around.