Why did the A-level algorithm say no? • Hacker News Board

Re: Why did the A-level algorithm say no?

#21

Post by macspoofing » Sun, Aug 16, 2020, 8:21 PM UTC

weego wrote: post #14

macspoofing wrote: post #12

Earlier quoted context omitted.

>If what is written here is true ... So is it? The guy is mining theoretical examples, but I wonder if in practice anyone has been hurt by this. And if they have (let's say a non-failing student was given a failing grade), why not handle those cases within a manual appeals process? After all, the exams weren't written, so no matter what mechanism you choose their will be students that get higher markers than they oth…

Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn, and it's not by a point or two, we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now. Oxford have…

>Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn

Yes. I understand there will be cases like this, but what numbers are we talking about?

And you were never going to prevent circumstances like those.

>we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now.

That's not irrelevant. Schools don't flip their standards from year or another. A bad school this year, will be a bad school next year. And many students are hurt by mark inflation. It isn't fair that one school will give out B, while another A+ for the same academic performance - and that certainly happens. There was a case in Canada some time ago where it came out that Universities were scoring high-schools based on performance of their alumni and normalizing the marks of new applicants based on those metrics. It was a small controversy, but what else are they going to do? The alternative is to assume that 'A' from one school is the same as 'A' from another - which is grossly unfair.

>Oxford have decided to ignore results and honor previously given offers because the process is so untrustworthy.

Good. Everyone understands that this is a weird time and everyone is trying their best. This is why this does not worry me one bit. I fully expect all schools that rely on these marks to be a little bit more understanding.

Re: Why did the A-level algorithm say no?

#22

Post by oh_sigh » Sun, Aug 16, 2020, 8:21 PM UTC

weego wrote: post #13

chongli wrote: post #7

Earlier quoted context omitted.

It's controversial in the sense that a bunch of people are protesting about it. Whether it's actually unfair or bad is unknown. Fairness is a complicated thing. A lot of it is based on feeling and relative status, rather than some objective measure of well-being. One person might find it deeply unfair that another person has a lot of money, relatively speaking. But take that money away and redistribute it equally bet…

It's very clearly unfair, I'm not sure what your example has to do with anything. No students performance should be a sum based on how previous students have done at their school. That's literally reinforcing the postcode divide in education quality that were supposed to turn a blind eye to. My home town school has historically refused to even put children in for exams if they didn't get at least a C grade in their m…

My high school had one person 3 years prior to my senior year get into an Ivy, and then before that it was no one in recent memory. In my senior class, 6 of us got into Ivies. I can only imagine what my life would be like if we only got the "fair" allotment of zero Ivy spots in a system like this.

Re: Why did the A-level algorithm say no?

#23

Post by corin_ » Sun, Aug 16, 2020, 8:24 PM UTC

Two things worth noting that the article doesn't go into...

1) "The year-on-year rise in proportion of students achieving A or A+ grades was much higher at independent schools than state comprehensives"

https://www.theguardian.com/education/2020/aug/13/england-a-...

Essentially, students lucky enough to be in private education were more likely to see their grades go up / stay the same, compared to students receiving free education seeing grades going down.

Has been explained away as a byproduct of a sensible algorithm, but considering the government's track record and private school backgrounds (85% of the UK's Prime Ministers have been educated at Eton, arguably the country's most exclusive private school) it's hard not to be sceptical that this wasn't intentional and class-motivated.

2) "Appears that Ofqual's algorithm caused today's A-level chaos. Ofqual chair Roger Taylor, also chairs the Centre for Data Ethics & Innovation (CDEI). Cummings' fave AI consultants - Faculty, have some juicy contracts with CDEI. And Faculty's COO Richard Sargeant is on CDEI board"

https://twitter.com/milesking10/status/1293886007771893762?s...

(Ofqual is the department that oversees exams and responsible for this situation, they report to Gavin Williams the government's Education Secretary. Cummings is a controversial special adverse to the Prime Minister, and Faculty is a firm he used when he was leading one of the two primary pro-Brexit campaigns before the referendum. Subjective statement: Cummings is a vile, evil man.)

Re: Why did the A-level algorithm say no?

#24

Post by ngcazz » Sun, Aug 16, 2020, 8:26 PM UTC

macspoofing wrote: post #15

A big pile of 'meh'. They are trying to make a statistical guess based on priors because the pandemic prevented the exams from being written. They are going to get things wrong. If they have an appeals process for things that are completely out line (e.g. a passing student getting a failing grade), then that's the best you can hope for given the circumstances. In the big picture, the good students are going to get go…

Even if you weren't completely mistaken in your assumptions, shouldn't students have the right to get the exam grades they actually deserve? I was at best average in secondary school and I had to kick it up a few notch for the national exams (Portuguese A-level equivalent). Under your "meh" style, I'd have lost at least a year before I'd be able to enroll in uni, because whatever!, despite getting scores above the 95th percentile in my exams.

Solidarity clearly means nothing here.

Re: Why did the A-level algorithm say no?

#25

Post by pjc50 » Sun, Aug 16, 2020, 8:28 PM UTC

macspoofing wrote: post #21

weego wrote: post #14

Earlier quoted context omitted.

Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn, and it's not by a point or two, we're talking straight A students getting demotd to Ds across the board because their school has a history of bad results, which are irrelevant to any given student right now. Oxford have…

>Of course children have been hurt by this is practice, there are numerous cases of children getting down graded from their predicted results so badly they're having University offer places withdrawn Yes. I understand there will be cases like this, but what numbers are we talking about? And you were never going to prevent circumstances like those. >we're talking straight A students getting demotd to Ds across the boa…

Marks aren't awarded by the school per se, they're awarded by one of the agencies such as AQA. This results in a lot of angry and upset teachers as the marks just received are a surprise to them too.

The Scottish government SQA has already backed down on its "adjustment" and agreed to use estimated marks from teachers and mock exams.

Re: Why did the A-level algorithm say no?

#26

Post by dynamite-ready » Sun, Aug 16, 2020, 8:29 PM UTC

jansan wrote: post #2

So is "controversial" the same as unfair or bad? Journalists often use "controversial" to discredit something that is not in accordance to their ideology. Controversial politicians (which politician isn't controversial?), controversial university professors, controversial speeches, controversial thoughts...

I wanted to use the word 'contentious', but I felt that would bend Hacker News rules. 'controversial' certainly feels like the right word in this case.

The scheme is flawed, as it clearly penalises some students, and benefits others. And many are rightfully outraged by that.

So yes. 'Controversial', I think.

Re: Why did the A-level algorithm say no?

#27

Post by macspoofing » Sun, Aug 16, 2020, 8:32 PM UTC

alexanderskates wrote: post #19

macspoofing wrote: post #15

A big pile of 'meh'. They are trying to make a statistical guess based on priors because the pandemic prevented the exams from being written. They are going to get things wrong. If they have an appeals process for things that are completely out line (e.g. a passing student getting a failing grade), then that's the best you can hope for given the circumstances. In the big picture, the good students are going to get go…

The problem is that the prior appears to be placed over the school, rather than the individual -- ie if your school had a low proportion of high achievers in previous years, students are finding their grades marked down, almost regardless of their performance. This results in particularly high grade disparity between independent and state schools. So it is not so much the case that good students get good marks, bad s…

Because mark inflation is a thing, and schools are different. There are schools that will give out A+ and others a B, for the same academic performance. School reputations don't change from year to year. So a school that inflates marks one year, will more than likely inflate them the next year. Are you suggesting that this should not be taken into consideration? That you should simply trust the relative weight of grade (relative to all the other schools) at face-value?

>"bad" schools get bad marks.

Why the quotes? There are schools that are at the bottom of academic rankings. That's a fact of reality.

>students are finding their grades marked down, almost regardless of their performance.

What numbers are we talking about here? No matter what algorithm or heuristic they chose, some unfairness was going to happen and my argument is that these cases are overstated. I'm sure they exist because we're talking about hundreds of thousands of students all in different circumstances. Having said that, in cases of egregious outcome, an appeals process would make sense. Also I'm sure many universities will take the pandemic into consideration and the fact that this was a best attempt at replicating standardizing test result without a test actually being taken.

Re: Why did the A-level algorithm say no?

#28

Post by mjburgess » Sun, Aug 16, 2020, 8:33 PM UTC

corin_ wrote: post #23

Two things worth noting that the article doesn't go into... 1) "The year-on-year rise in proportion of students achieving A or A+ grades was much higher at independent schools than state comprehensives" https://www.theguardian.com/education/2020/aug/13/england-a-... Essentially, students lucky enough to be in private education were more likely to see their grades go up / stay the same, compared to students receiving…

> it's hard not to be sceptical that this wasn't intentional and class-motivated.

It's incredibly easy "not to be sceptical".

The teacher's predicted grades are, on average, unprecedented. And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school).

Teachers shouldn't have been asked to predict grades. That's the heart of the problem. Not some harebrained in-public conspiracy.

This whole thing is dumb. And the media are deliberately failing to explain basic facts about the education system to provoke this kind of nonesense outrage.

Re: Why did the A-level algorithm say no?

#29

Post by pjc50 » Sun, Aug 16, 2020, 8:38 PM UTC

macspoofing wrote: post #27

alexanderskates wrote: post #19

Earlier quoted context omitted.

The problem is that the prior appears to be placed over the school, rather than the individual -- ie if your school had a low proportion of high achievers in previous years, students are finding their grades marked down, almost regardless of their performance. This results in particularly high grade disparity between independent and state schools. So it is not so much the case that good students get good marks, bad s…

Because mark inflation is a thing, and schools are different. There are schools that will give out A+ and others a B, for the same academic performance. School reputations don't change from year to year. So a school that inflates marks one year, will more than likely inflate them the next year. Are you suggesting that this should not be taken into consideration? That you should simply trust the relative weight of gra…

Per other thread, A levels are marked outside the school by the qualifications body.

Re: Why did the A-level algorithm say no?

#30

Post by marksomnian » Sun, Aug 16, 2020, 8:40 PM UTC

Jabbles wrote: post #3

Do teachers recommend a grade, or a score? If they only recommend grades, how did the algorithm pick which students to change the grades for?

As I understood it:

1. Teachers submit both an estimated grade and a relative rank ordering for each student

2. Ofqual calculate a distribution based on the predicted grades

3. They compare it to grade distributions at that school from previous years

4. If need be, they shift the distribution up or down to be roughly in line with that school's prior results

5. They then use the rank ordering to assign students grades from that distribution (e.g. in a class with 20 students, each student is worth 5% of the distribution - so if it predicted 10% A* (highest grade), 15% A, etc., the top 2 students would get an A*, the next 3 an A and so on)