Building the most inaccessible site with a perfect Lighthouse score (2019) • Hacker News Board

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#21

Post by dsr_ » Sat, Jun 06, 2020, 9:16 PM UTC

ChrisMarshallNY wrote: post #15

One of the "philosophers' stone" goals of the software industry is to completely replace human testing with automated testing. Basically, I think automated testing is a very good thing, and we should definitely try to do as much of it as possible. So we can clear the way for more useful and meaningful human testing. I've always thought that the engineers in QC should be just as skilled and qualified as the ones build…

And for specialized products, you want a range of testers from "this is my first day working with this" up through "I send you bug reports so often your helpdesk knows my name."

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#22

Post by anoncareer0212 » Sat, Jun 06, 2020, 9:17 PM UTC

ljm wrote: post #9

luckylion wrote: post #6

Imho, the problem with Lighthouse (and Pagespeed before it) isn't that they're not perfect, it's that they assign scores/grades. When Google assigns a score to something, people understand it to mean highest score = best and start optimizing for the grade Google gives them, not for performance and user experience, which the grade is supposed to represent. It would be more fruitful to list the issues and their severit…

The problem, as I see it, is it more or less makes Google the arbiter of the internet, and when their purpose is to provide internet advertising then I see a conflict of interest. Especially when things like "supporting AMP" will likely impact your score. While unfortunately many businesses have to care about Google SEO due to their complete monopoly status in the search field (one that is of course growing increasin…

I'm not sure the argument in your post or the parents hold water - lighthouse scores aren't a "judge, jury, and executioner"

sometimes dev tools are just dev tools, and nothing is perfect.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#23

Post by dccoolgai » Sat, Jun 06, 2020, 9:17 PM UTC

I just read an article a couple days ago about how even YouTube widgets and stuff have huge a11y problems. I think it's time to admit Google is terrible at accessibility. All their devrels talk like it's important and have all these beautiful demos but whenever you look behind the curtain at their products it's terrible.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#24

Post by runawaybottle » Sat, Jun 06, 2020, 9:20 PM UTC

naringas wrote: post #17

ChrisMarshallNY wrote: post #15

One of the "philosophers' stone" goals of the software industry is to completely replace human testing with automated testing. Basically, I think automated testing is a very good thing, and we should definitely try to do as much of it as possible. So we can clear the way for more useful and meaningful human testing. I've always thought that the engineers in QC should be just as skilled and qualified as the ones build…

>I've always thought that the engineers in QC should be just as skilled and qualified as the ones building the product. I agree, but then they need to be equally well payed... and I don't think it is possible to find a manager willing to pay them as much

It’s probably a matter of how Important your application actually is. If it’s dealing with critical numbers or features that have a real cost when it breaks, then the investment into good testing pays for itself.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#25

Post by gav » Sat, Jun 06, 2020, 9:24 PM UTC

naringas wrote: post #17

ChrisMarshallNY wrote: post #15

One of the "philosophers' stone" goals of the software industry is to completely replace human testing with automated testing. Basically, I think automated testing is a very good thing, and we should definitely try to do as much of it as possible. So we can clear the way for more useful and meaningful human testing. I've always thought that the engineers in QC should be just as skilled and qualified as the ones build…

>I've always thought that the engineers in QC should be just as skilled and qualified as the ones building the product. I agree, but then they need to be equally well payed... and I don't think it is possible to find a manager willing to pay them as much

I think this is important and requires a big shift in how most organizations think of QA.

A lot of QA being done is very mechanical and and done by junior staff offshore to keep the cost as low as possible. This causes the value to be low too, for example, the QA team for a F100 company I worked with would meticulously test against specs and file bugs such as "the error message is misaligned on page XYZ". Which was true, but they missed that the error message didn't make any sense at all.

Improved automated testing has the opportunity to free up people to move from Quality Assurance (preventing defects) to being the voice of the user and ensuring quality products. This shift needs a completely different skill set and mindset, but equally it needs organizations to rethink both the cost of software engineering and the value it creates.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#26

Post by mattlondon » Sat, Jun 06, 2020, 9:35 PM UTC

kopite wrote: post #13

First, let me start by saying that this is a good article and sheds light on one of the challenges of accessibility adoption. All of tools like lighthouse, axe-core etc. run a subset of tests that gives a false sense of security about accessibility. Similarly, tools like Accessibility Insights for Web has a fast-pass option, which does the same thing where it runs a subset of tests to catch the most common issues on…

Where I am, we use axe-core as part of our automated integration testing. Any change that regresses on accessibility (i.e. new severe a11y issue etc) is blocked from submission in the same way that it would be blocked from submission by causing tests to fail.

I find this useful as it prevents "accessibility rot" where people think "Oh we'll sort out the aria stuff later".

But yeah I agree that a human test is best. I think of axe et al as an equivalent of a "static analysis" that can pick out the obvious mistakes, but wont understand the dynamic nature of the application - it is after all great for flagging the "easy" problems, allowing the human tester to focus on the major a11y issues without ending up raising hundreds of bugs for every button and link and colour combination that should have been sorted out way before any human testing is involved.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#27

Post by userbinator » Sat, Jun 06, 2020, 9:40 PM UTC

Having seen many other instances where chasing numbers has lead to a worse outcome, I think "metrics driven development" is an abomination that must be abolished. Unfortunately, management seems to really like the idea of turning everything into a number and increasing it at all costs --- I have fought against such things, and when I pointed out all the negatives associated with it, they would often agree; but then dismiss the thought completely with a response that essentially means "but it makes the numbers look better."

As the saying goes: "Not everything that counts can be counted, and not everything that can be counted, counts."

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#28

Post by theomh » Sat, Jun 06, 2020, 9:48 PM UTC

userbinator wrote: post #27

Having seen many other instances where chasing numbers has lead to a worse outcome, I think "metrics driven development" is an abomination that must be abolished. Unfortunately, management seems to really like the idea of turning everything into a number and increasing it at all costs --- I have fought against such things, and when I pointed out all the negatives associated with it, they would often agree; but then d…

I think what you are describing is a case of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

https://en.wikipedia.org/wiki/Goodhart%27s_law

On the other hand, a true accessibility score (for example, one awarded by a human or a robust grading process) could reliably quantify accessiblity. Goodhart wouldn't apply in that case.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#29

Post by the_pwner224 » Sat, Jun 06, 2020, 10:13 PM UTC

anoncareer0212 wrote: post #22

ljm wrote: post #9

Earlier quoted context omitted.

The problem, as I see it, is it more or less makes Google the arbiter of the internet, and when their purpose is to provide internet advertising then I see a conflict of interest. Especially when things like "supporting AMP" will likely impact your score. While unfortunately many businesses have to care about Google SEO due to their complete monopoly status in the search field (one that is of course growing increasin…

I'm not sure the argument in your post or the parents hold water - lighthouse scores aren't a "judge, jury, and executioner" sometimes dev tools are just dev tools, and nothing is perfect.

Wouldn't be surprised if they use it in search rankings, chances are they already do. And not being on Google effectively == dead. They already take into account page load speed (which discriminates against complex but useful & good web apps) and iirc also whether you have AMP.

Re: Building the most inaccessible site with a perfect Lighthouse score (2019)

#30

Post by _puk » Sat, Jun 06, 2020, 10:15 PM UTC

jameslk wrote: post #16

luckylion wrote: post #6

Imho, the problem with Lighthouse (and Pagespeed before it) isn't that they're not perfect, it's that they assign scores/grades. When Google assigns a score to something, people understand it to mean highest score = best and start optimizing for the grade Google gives them, not for performance and user experience, which the grade is supposed to represent. It would be more fruitful to list the issues and their severit…

My guess is if there wasn't a Lighthouse score, the focus would be on another singular metric instead, like First Contentful Paint for web performance. The score, while imperfect, helps at least to have a focus on something that weighs multiple metrics. As an industry, I'm wary of having a single company define these scores. It seems like a conflict of interest. I'd rather have this in control by a standards group.

My issue with the score is that it is numerical. If I get 91, I want 100.

Having a traffic light style system whereby green is good, red is bad etc. would give a decent target without the costly Uber optimisation for perfection.

To your point of a single company defining these scores - a standards body would be great, I agree, but there's a danger it'll be like all the other web standards bodies, whereby Google is a member, albeit one with pretty much all the power.