Why did the A-level algorithm say no? • Hacker News Board

Re: Why did the A-level algorithm say no?

#51

Post by krisoft » Sun, Aug 16, 2020, 9:03 PM UTC

This was so foreseeable when they anounced the plans.

One possible solution could have been if they do the same statistical estimation game and then offer a choice to each student: you can take your estimated grades or go for an exam. If you take the exam you get what you het there. It might go lower or higher than the estimate. This would have incentivised that only those who are certain of themselves and their estimated grades are insuficient towards their goals will go for the tests. And since these are outliers already there will be a lot less of them than the general population, thus it is easier to organise socially distanced exams for them.

Of course nothing is easy at scale and many details need to be ironed out but thats what we keep the Miniszry of Education for.

Re: Why did the A-level algorithm say no?

#52

Post by mjburgess » Sun, Aug 16, 2020, 9:04 PM UTC

kitd wrote: post #47

mjburgess wrote: post #28

Earlier quoted context omitted.

> it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented . And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teach…

Teachers predict grades all the time. It's part of how student progress is tracked. We are constantly being told what level our kids are at and predicted level at exam time. Part of the problem is that teachers often push the "prediction" slightly higher as a means of motivating the student. Now those predictions are being used as the real deal. I am pretty convinced that most teachers with more than superficial expe…

Well as a point of fact, they didnt. Their requested, "accurate" grades represent a 20% increase in general attainment. That has never happened in the history of modern education, and occurs only when, eg., civil wars end.

As a comparative reference, we'd expect an at-most, 3% increase in IQ in the general population.

Re: Why did the A-level algorithm say no?

#53

Post by stoneman24 » Sun, Aug 16, 2020, 9:05 PM UTC

mjburgess wrote: post #28

corin_ wrote: post #23

Two things worth noting that the article doesn't go into... 1) "The year-on-year rise in proportion of students achieving A or A+ grades was much higher at independent schools than state comprehensives" https://www.theguardian.com/education/2020/aug/13/england-a-... Essentially, students lucky enough to be in private education were more likely to see their grades go up / stay the same, compared to students receiving…

> it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented . And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teach…

Given the extent of some candidates differences and the impact to their life (denied entrance to offered Uni courses for that year) there’s some very serious outrage. If you were part of good class in a school with a poor record, then you’re stuffed. It’s a unprecedented situation and the only guide that you have are teacher grades.

Re: Why did the A-level algorithm say no?

#54

Post by mjburgess » Sun, Aug 16, 2020, 9:07 PM UTC

corin_ wrote: post #49

mjburgess wrote: post #39

Earlier quoted context omitted.

Right, and what was the previous yoy disparity? If the previous YoY was also 5%/2%, then it is just repeating the trend. It isnt the job of statisticians to "fix" social inequality by BSing some grades. Its their job to estimate likely exam results based on previous performance. There's absolutely no grounds on which to assume that growth in A*/A across these demographics would be the same; and of course, very many r…

I can't find that data easily, if I'm bored tomorrow I might try to do the research myself - all the data is available but raw not top level overview at https://www.compare-school-performance.service.gov.uk/downlo... It's only been a couple of days, we'll get more data journalists breaking it down better soon. You know what would be really helpful? If the algorithm was released to be publicly reviewed. Oh, but the go…

I agree. It should be published.

However more important than the algorithm is just the media reporting on what ofqual actually say. I've barely read any of their actual analysis in news on the issue.

Key facts: students are always ranked and graded on a curve. Student populations have a fixed number of grades awarded every year, etc.

The system is specifically designed in recognition that grades are positional goods, ie., they are valuable only to the degree that other people do not have them.

That's the view of the education system every year. And its the basic assumption of this process. They will never over-award A grades specifically because they want universities/business/etc. to index them to a percentile performance in the population.

Re: Why did the A-level algorithm say no?

#55

Post by dynamite-ready » Sun, Aug 16, 2020, 9:09 PM UTC

mjburgess wrote: post #28

corin_ wrote: post #23

Two things worth noting that the article doesn't go into... 1) "The year-on-year rise in proportion of students achieving A or A+ grades was much higher at independent schools than state comprehensives" https://www.theguardian.com/education/2020/aug/13/england-a-... Essentially, students lucky enough to be in private education were more likely to see their grades go up / stay the same, compared to students receiving…

> it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented . And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teach…

What doesn't quite make sense, is the fact that these students sat 'mock' examinations earlier in the year. Presumably, these 'mock' exams are based on an empirical standard, similar to the one used for the 'final' exam. So the first question is, why weren't these 'mock' results the main anchor for these derived results?

Re: Why did the A-level algorithm say no?

#56

Post by lhnz » Sun, Aug 16, 2020, 9:09 PM UTC

mjburgess wrote: post #48

lhnz wrote: post #44

Earlier quoted context omitted.

> The issue is that all teachers significantly > over-predicted their students grades, > but less so at high-performing schools. "All teachers" is a sweeping statement. There are teachers in comprehensive schools that are tough-markers, too. My wife, who is a teacher in a London school, was annoyed in both directions - there were teachers that she felt had given away high grades, but there was also a teacher that had…

I meant all teachers across school types, not all teachers individually. In both state and private schools, teachers over-predicted. But higher-performing schools have higher and more uniform results, so there's less room to actually over predict -- so there's comparatively less reduction. That's just an in-built concequence of what it means to be a high performing school

Sure. It makes sense.

I don't believe in the media conspiracy that the Tories created an algorithm that systematically awards higher grades to rich kids because of inherent classism or even by accident.

But I do really disagree with the insistence that all teachers in comprehensive schools were conspiring to give their kids higher grades than they deserved -- this view was frequently repeated on social media. Even if there were some that were doing this and it was apparent from the statistics, teachers are not a homogenous group and there are many teachers that take fair grading very seriously and even take enjoyment in being tough markers.

I don't know the data but my own worry is that this will have hurt the high variance or lazy kids that won't normally get high grades but would normally settle down and revise hard for such an important exam. I hope that these kids will be given the opportunity to resit the exam for free, since I doubt an algorithm could capture the variance that exists every year.

Re: Why did the A-level algorithm say no?

#57

Post by corin_ » Sun, Aug 16, 2020, 9:11 PM UTC

dynamite-ready wrote: post #55

mjburgess wrote: post #28

Earlier quoted context omitted.

> it's hard not to be sceptical that this wasn't intentional and class-motivated. It's incredibly easy "not to be sceptical". The teacher's predicted grades are, on average, unprecedented . And represent many times greater than any previous yoy increase. This is far less true at top-performing schools where teachers are predicting far more consistent grades (by the very nature of being a top performing school). Teach…

What doesn't quite make sense, is the fact that these students sat 'mock' examinations earlier in the year. Presumably, these 'mock' exams are based on an empirical standard, similar to the one used for the 'final' exam. So the first question is, why weren't these 'mock' results the main anchor for these derived results?

The problem with mock exams is that they don't follow any national rules on how they're used. Some schools do them in November, some in March. Some teachers mark them having not been trained in A Level marking. Some schools may do mocks in exact exam conditions, others might not bother with one part of the test, or have them done without supervision. Some teachers grade them accurately, some intentionally grade them harshly to motivate their students to study. Etc.

Re: Why did the A-level algorithm say no?

#58

Post by corin_ » Sun, Aug 16, 2020, 9:18 PM UTC

mjburgess wrote: post #54

corin_ wrote: post #49

Earlier quoted context omitted.

I can't find that data easily, if I'm bored tomorrow I might try to do the research myself - all the data is available but raw not top level overview at https://www.compare-school-performance.service.gov.uk/downlo... It's only been a couple of days, we'll get more data journalists breaking it down better soon. You know what would be really helpful? If the algorithm was released to be publicly reviewed. Oh, but the go…

I agree. It should be published. However more important than the algorithm is just the media reporting on what ofqual actually say. I've barely read any of their actual analysis in news on the issue. Key facts: students are always ranked and graded on a curve. Student populations have a fixed number of grades awarded every year, etc. The system is specifically designed in recognition that grades are positional goods,…

> Key facts: students are always ranked and graded on a curve. Student populations have a fixed number of grades awarded every year, etc.

I'm pretty sure that's not correct, please find a source suggesting that. There have been major reforms (primarily the changing A Levels back from the modular version that Labour had previously introduced) and these were specifically aimed at reducing grade inflation. If there were a fixed number of grades each year there wouldn't be any inflation to counter.

Re: Why did the A-level algorithm say no?

#59

Post by stoneman24 » Sun, Aug 16, 2020, 9:20 PM UTC

Twirrim wrote: post #46

Jabbles wrote: post #3

Do teachers recommend a grade, or a score? If they only recommend grades, how did the algorithm pick which students to change the grades for?

Here's a write-up of the situation, shared on Facebook publicly, from a friend involved in careers at a UK Sixth Form college (where students take their A-Levels): ################### I've spent the last two days dealing with very upset young people who have had their plans ripped up. Some will still be able to go to university, but not the one they planned on and have been forced into clearing, some won't be able to…

Thank you for information. As a Scot, with a relative involved (she got what she needed), I am grateful, that the Scot gov changed their minds and used teachers grades without moderation. Even though our mocks (“prelims”) are produced by the SQA who also do the main exams. It’s a unprecedented situation, let’s cut some slack and base it on the best info that we have, the teachers view. I hope English gov see sense but so far they don’t have demonstrated much.

Re: Why did the A-level algorithm say no?

#60

Post by jarvist » Sun, Aug 16, 2020, 9:23 PM UTC

marksomnian wrote: post #30

Jabbles wrote: post #3

Do teachers recommend a grade, or a score? If they only recommend grades, how did the algorithm pick which students to change the grades for?

As I understood it: 1. Teachers submit both an estimated grade and a relative rank ordering for each student 2. Ofqual calculate a distribution based on the predicted grades 3. They compare it to grade distributions at that school from previous years 4. If need be, they shift the distribution up or down to be roughly in line with that school's prior results 5. They then use the rank ordering to assign students grades…

It's more complicated than this. They haven't just taken the rank ordering by the teachers and projected onto a curve.

Here is the 319 page document published on results day. This is the totality of the current description of the 'algorithm'. https://assets.publishing.service.gov.uk/government/uploads/...

There's some kind of predictive linear model (page 40) which is based on a student's prior attainment (i.e. GCSE and SATs 'data'), but then modelled at the level of the exam centre (i.e. school). These models are separately fitted for different bins (I think ten are shown at one point) of achievement.

The whole document is pretty incoherent. I suspect the 'algorithm' is a set of excel spreadsheets.