Why did the A-level algorithm say no? • Hacker News Board

Re: Why did the A-level algorithm say no?

#121

Post by luckylion » Mon, Aug 17, 2020, 7:24 AM UTC

Certhas wrote: post #102

mjburgess wrote: post #28

Earlier quoted context omitted.

> it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented . And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teach…

It's not obvious that predicted grades are categorically worse than the system they put in place though, is it? In some aggregate statistics we can see them perform worse but exams and grading aren't about averages at school level but individual achievement. A prediction systems that doesn't take students prior achievements into account is just obviously defective here. And that matters especially if it locks out a w…

> A prediction systems that doesn't take students prior achievements into account is just obviously defective here.

Doesn't it? What else would how students have been ranked in ability mean?

> And finally, so what if 2020 had had outlier grade average? It's the outlier year.

"Hey, we'll give everyone 40% better grades, because it was a tough year" won't do good. You'll get lots of people into university who will fail miserably, because the grades don't magically improve their skills. Unless, and that's a bad idea as well, you just tighten the criteria so the same number as before can go to uni. Grade inflation won't help anyone either.

Re: Why did the A-level algorithm say no?

#122

Post by lordnacho » Mon, Aug 17, 2020, 7:25 AM UTC

I've been listening to talk radio here in the UK, and it's a constant stream of disappointed kids/teachers/parents phoning in. Some of the stories are horrendous, people expecting AAA getting CCD and similar. It seems unlikely that you'd ever have three teachers predicting you to get top marks, but you end up with that on results day.

But about the algorithm, it seems to be an impossible task. You cannot satisfy the requirement of the gross statistics being roughly the same, ie little grade inflation on average, while also identifying talented individuals.

The outline of the algorithm seems to be that you basically take the gross stats for earlier years from a school, and then you take the teacher's guesses about who will do well, and make up a distribution of grades around that. This is probably what most people would do if forced into such an exercise. And you can add wrinkles to it like a bit of grade inflation, adjustment for overconfident predictions, and so on.

But you can't get that piece of information that you really need, which is a mix of signal and noise about the true distribution of the kid's abilities. If you have a talented year in a low performing school, you'll never know. And vice versa.

The year before I finished the IB, two kids got the top 45/45. That's normally achieved by something like 1/400 kids worldwide, and it was a class of maybe 30 kids. That would never have been awarded without examinations.

In addition the other presumed goal, of having the grades as close to the predictions as possible, give the algo designer an incentive well known from machine learning: guess safely. Squish the distribution on the edges, that way you're not off much. This is bad for people who shoud be getting As, because many will get Bs. And it's good on the other end, though I don't know where the lump ends (C or D?).

Re: Why did the A-level algorithm say no?

#123

Post by JoeSmithson » Mon, Aug 17, 2020, 7:26 AM UTC

handelaar wrote: post #9

jansan wrote: post #2

So is "controversial" the same as unfair or bad? Journalists often use "controversial" to discredit something that is not in accordance to their ideology. Controversial politicians (which politician isn't controversial?), controversial university professors, controversial speeches, controversial thoughts...

In this case, yes it is. Low-performing students at private schools are having their estimated Cs and Ds bumped up to As. Stellar students at schools in poor areas who got As in their mock exams and have teacher-predicted As are being awarded Ds instead.

> "Low-performing students at private schools are having their estimated Cs and Ds bumped up to As"

Do you have an example of this happening? It seems basically impossible from my understanding of the process.

-----

Edit: I think I understand how it can happen. A school with a cohort that is small, but large enough to get the algorithm, and a previously exemplary results, has an unusually bad pupil. I can see this happening in a subject like Further Maths where only good students take it

Re: Why did the A-level algorithm say no?

#124

Post by DanBC » Mon, Aug 17, 2020, 7:29 AM UTC

jdietrich wrote: post #83

corin_ wrote: post #31

Earlier quoted context omitted.

I completely agree that the system put in place is poor and that it causing an accidental bias is very feasible. But if you're going to put this system in place, with months of planning, you should make damn sure that it isn't massively favouring the children who are already advantaged in life. Which it statistically is, whether intentional or not. They screwed it up so spectacularly that the education secretary had…

Independent schools got higher grades because of a quirk in the algorithm. If there weren't enough students taking a particular course at a particular centre for the rank/previous results method to be reasonably accurate, the algorithm defaulted to partially or wholly relying on predicted grades. Independent schools generally have fewer students and a wider range of courses and predicted grades were inflated across t…

You could have looked at past performance of those schools.

https://twitter.com/queenofswords6/status/129447148835198566...

> A levels...short thread

> If anyone doubts who the winners are this year, here's a random selection of independent schools and the % of A*-A grades this year.

> 2019 figures follow in brackets.

  Mayfield Girls'     66% (55%)
  Sherfield School    70% (37%)
  Leweston School     51% (37%)
  Farlington School   62% (48%)
  The Marist School   55% (52%)
  Godolphin School    50% (29.2%)
  St Mary's Cambridge 54% (51%)
  St George's Ascot   63% (35%)
  Rye St Anthony      48.1% (18.3%)

Re: Why did the A-level algorithm say no?

#125

Post by CaptainZapp » Mon, Aug 17, 2020, 7:29 AM UTC

throwaway_pdp09 wrote: post #113

nelaboras wrote: post #106

An obvious point not discussed is that the exams system was already biased to begin with. Previous years were already unfair as great kids at bad schools have much fewer chances than less great kids for which the parents can pay a great school. England especially is awful at achieving a fair system (that gives everyone a fair chance at good grades) because schools are so stratified - by design of funding rules, etc.

You're stating the obvious, a very well-known obvious. Perhaps you can suggest any system that is not intrinsically unfair (I'd be interested as I went both to schools in sink areas, and an expensive private school). (this was in the UK BTW) > ...is awful at achieving a fair system [...] because schools are so stratified - by design of funding rules etc. I'd say it was more down to the culture of parents in the poor…

I'm not surprised that you're downvoted and may be into oblivion. But you actually make a very valid point.

That said I also believe that the current UK government is far beyond any benefit of a doubt in terms of competence and maliciousnes.

Re: Why did the A-level algorithm say no?

#126

Post by DanBC » Mon, Aug 17, 2020, 7:32 AM UTC

mjburgess wrote: post #32

corin_ wrote: post #31

Earlier quoted context omitted.

I completely agree that the system put in place is poor and that it causing an accidental bias is very feasible. But if you're going to put this system in place, with months of planning, you should make damn sure that it isn't massively favouring the children who are already advantaged in life. Which it statistically is, whether intentional or not. They screwed it up so spectacularly that the education secretary had…

It doesnt "favour" any children. Those children were already going to get the highest results. There is no incompetence here. The predicted grades are a good estimate of what the results would have been. The issue is that all teachers significantly over-predicted their students grades, but less so at high-performing schools. (Why? because the grades are more consistent at those schools, so there's less room for error…

> but less so at high-performing schools.

The bit you haven't understood is that teacher predicted grades were reassessed by the algorithm unless the class sizes were 5 children or fewer. This meant that the predicted grades that were not adjusted by the algo were overwhelmingly at independent fee-paying schools.

Re: Why did the A-level algorithm say no?

#127

Post by throwaway_pdp09 » Mon, Aug 17, 2020, 7:36 AM UTC

CaptainZapp wrote: post #125

throwaway_pdp09 wrote: post #113

Earlier quoted context omitted.

You're stating the obvious, a very well-known obvious. Perhaps you can suggest any system that is not intrinsically unfair (I'd be interested as I went both to schools in sink areas, and an expensive private school). (this was in the UK BTW) > ...is awful at achieving a fair system [...] because schools are so stratified - by design of funding rules etc. I'd say it was more down to the culture of parents in the poor…

I'm not surprised that you're downvoted and may be into oblivion. But you actually make a very valid point. That said I also believe that the current UK government is far beyond any benefit of a doubt in terms of competence and maliciousnes.

I'd prefer constructive comebacks from people rather than downvotes. I can't learn from downvotes.

But OMG the competence of the UK govt. It's like they literally can't do... anything without screwing it up. It is bizarre. Then they try to paper over it with words.

Re: Why did the A-level algorithm say no?

#128

Post by CaptainZapp » Mon, Aug 17, 2020, 8:13 AM UTC

throwaway_pdp09 wrote: post #127

CaptainZapp wrote: post #125

Earlier quoted context omitted.

I'm not surprised that you're downvoted and may be into oblivion. But you actually make a very valid point. That said I also believe that the current UK government is far beyond any benefit of a doubt in terms of competence and maliciousnes.

I'd prefer constructive comebacks from people rather than downvotes. I can't learn from downvotes. But OMG the competence of the UK govt. It's like they literally can't do... anything without screwing it up. It is bizarre. Then they try to paper over it with words.

I think it's mostly due to this statement:

"I'd say it was more down to the culture of parents in the poor areas mostly not giving a shit about their kid's education"

Which sounds nasty, but rings true for a big part of England. It's also an uncomfortable truth, which basically always leads to downvotes.

Once a year (safe for this year, obviously) I'm in Blackpool at the Rebellion Punk Festival[1]. What you see in Blackpool and how people live there is very, very different from the impression you get when just visiting London.

People wouldn't believe how poor and destitute some areas in the UK are and the priority of a lot of people in such areas is not really education. Young women want as many kids as possible, since government benefits beat the shitty work prospects they have in such areas and the future for young people in general in those places are so bleak that it often leads to an understandable "why bother?" attitude.

And yes, this attitude is inherited, or actually worse. When there is a rare young person trying to be an achiever you can almost be sure he's mocked and bullied by his peers.

[1] http://www.rebellionfestivals.com/

Re: Why did the A-level algorithm say no?

#129

Post by kbut » Mon, Aug 17, 2020, 8:46 AM UTC

harry8 wrote: post #110

nelaboras wrote: post #107

Earlier quoted context omitted.

You are under the mistaken assumption that somehow exams are more fair or get a better view of what students can do. It's well established that exams measure a few limited congnitive constructs and its not the smartest or most hard working students that perform best, but rather those who are prepared by teachers for those exact constructs and the precise way of answering them. A brilliant student at a public school m…

Exams, being far, far from perfect /ARE/ more fair. Why? because it is the same exam for everyone. The person marking the exam has no information about the person who wrote the paper, not their name, not what they look like, not how rich they are, not how pretty they are, not what school they attend. Whether or not you can achieve better outcomes or get a better indication of capability with other means, that total a…

Not what school they attend: Except they dont mark their own schools and i think they keep the schools papers together in bundles. Plus markers have been to open days and continuous development days with the other teachers. plus they are having meetings daily to discuss how to mark classes of answers (where teaching methods used come up). i think the net of this is that they often do know what school they are marking (especially for the outliers - best and worst schools). For some subjects e.g. computer science then the pupils will answer in the langiages they were taught, so if its a python answer you know its one of the schools that teaches python)

There is kind of just a problem overall with comparing private schools to state schools. At the very least this process may have worked better if they had two data sets, one for private one for atate schools.

Beyond that though a stastical solution was never going to work, you cannot generate accurate discrete/quantized data, only representative random data.

private schools imho teach you to pass the exam, they are more cut throat about focussing on this than state schools. and the teachers seem to fight harder for their kids - the difference in personal statements from private school teachers compared to state schools on uni applications would be an interesting study.

I am not saying i have answers, just intereted in thw discussion. Overall i am always left with a few basic ideas to keep things fair (though i sometimes think some of these may be dumb and i am not suggesting do all of them):

-Zero charitable status or governemnt support for private education.

-cap the number of students a uni can take from one school (e.g. dont take the top 100 students, take the top 1 from 100 schools)

-ban people who were not state educated from governemnt positions. This is contravertial but imho you effectively live a different life when you go to privare schools (class sizes, support and oppourtunity are in comparable)

Re: Why did the A-level algorithm say no?

#130

Post by polytely » Mon, Aug 17, 2020, 9:04 AM UTC

naruciakk wrote: post #37

Couldn't the UK just have a regular exam, just with the social distancing measures implemented? The exams were held in many European countries with these measures and nothing bad happened and the children has received a fair examination for the University

My brother here in the Netherlands just got his average grade (of the test throughout the year that count toward the final grade) for each subject as final grade, which seems like it's fair enough.