EY has been quietly laying people off for the last year solid.
It's unsurprising that trying to do more with less results in lower quality.
EY Canada published a cybersecurity report and most citations were hallucinated
11–20 of 156 posts
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#12The problem we're seeing across many professions is AI output is not getting vetted by knowledgeable people, whether it's an experienced analyst, senior engineer, expert attorney, or the resident physician. At best they skim, at worst they don't even see it at all before it's published, pushed to production, distributed to clients, or submitted to the court. In many cases the skills are available in house to do the n…
Part of the problem: you get given a complete document to review after it's been fully baked.
I'm pushing the need for basic engineering principles across whole organisations.
You wouldn't give an engineer 1000 lines of code to review without the original spec of what you're trying to achieve for context (at a minimum, ideally the reviewer was in the room when the work was introduced, and has full context).
So, these docs, they're given as an all or nothing.
Do you push back on the 39th metric that is defined to the utmost detail? Or just resign yourself to the fact that it is what it is?
A one (6 is the goto if we're talking Amazon?!) pager.. "this is what I am proposing" at least gives the skeleton of the idea to push back at the general shape of the idea, refine it, before all the emotional investment of your precious report being complete.
Y'know.. the traditional product running through the spec in a SCRUM* environment.. the engineers doing proper code reviews..
* Yes SCRUM is dead, but that's another thing.
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#13Holy horrible UI
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#14This sort of thing is a complete embarrassment to a firm like EY, where people are paying them a lot of money for advice. They’ve basically demonstrated that their market leading research is just someone asking questions to ChatGPT.
If you ever needed evidence to not buy “advice” from such outfits, this is exhibit one.
Hopefully they at least fired the partner that published this steaming pile of AI slop.
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#15The problem we're seeing across many professions is AI output is not getting vetted by knowledgeable people, whether it's an experienced analyst, senior engineer, expert attorney, or the resident physician. At best they skim, at worst they don't even see it at all before it's published, pushed to production, distributed to clients, or submitted to the court. In many cases the skills are available in house to do the n…
> the idea that Amazon would allow human bottlenecks to appear across projects and underlying infrastructure is ridiculous. Why?
Amazon is fairly well known to ruthlessly optimize every process.
So if they're having humans proofread what the AI produces, they must have found that to be necessary.
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#16The problem we're seeing across many professions is AI output is not getting vetted by knowledgeable people, whether it's an experienced analyst, senior engineer, expert attorney, or the resident physician. At best they skim, at worst they don't even see it at all before it's published, pushed to production, distributed to clients, or submitted to the court. In many cases the skills are available in house to do the n…
> In many cases the skills are available in house to do the necessary vetting, but these people are already overwhelmed with their existing day to day.
I think a lot of the time it's just pure laziness. AI gives people a magical "do all the work for me" button and it can bring out the worst in them.
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#17I don't quite get it why they can't take another LLM and vet the output of the first with the second one. Surely they would not have the same hallucinations and would be able to detect hallucinations of the earlier LLM. Maybe it would cost too much in terms of tokens?
I don't know but I would expect it to be realtively easy for an LLM to detect "hallucinations".
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#18What a horrible page to navigate
Very difficult to use on mobile.
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#19I don't quite get it why they can't take another LLM and vet the output of the first with the second one. Surely they would not have the same hallucinations and would be able to detect hallucinations of the earlier LLM. Maybe it would cost too much in terms of tokens? I don't know but I would expect it to be realtively easy for an LLM to detect "hallucinations".
>I don't quite get it why they can't take another LLM and vet the output of the first with the seond one.
I think this may be part of the problem. The actual humans creating the report don't have the expertise to know which one to trust. At least that was what consulting was like in my experience at a similar firm.
Re: EY Canada published a cybersecurity report and most citations were hallucinated
#20I don't quite get it why they can't take another LLM and vet the output of the first with the second one. Surely they would not have the same hallucinations and would be able to detect hallucinations of the earlier LLM. Maybe it would cost too much in terms of tokens?
I don't know but I would expect it to be relatively easy for an LLM to detect "hallucinations".