Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

India's top court angry after junior judge cites fake AI-generated orders

bbc.com

Post reply on HN
Print view View original

21–30 of 212 posts

Re: India's top court angry after junior judge cites fake AI-generated orders

#21

Post by ben_w »

duskdozer wrote: post #8
voidUpdate wrote: post #2

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

I'm continually amazed at how much faith people have in them. I guess since they can sound like people and output really authoritative and confident text it just overrides any skepticism subconsciously?

Much as I like them, I do frequently remind myself of two things:

1) https://en.wikipedia.org/wiki/Clever_Hans

2) https://archive.org/details/nextgen-issue-26 as an example of how in the 90s we has rapid cycles of a new tech (3d graphics) astounding us with how realistic each new generation was compared to the previous one, and forgetting with each new (game engine) how we'd said the same and felt the same about (graphics) we now regarded as pathetic.

So yes, they do sound "authoritative and confident text it just overrides any skepticism subconsciously", but you shouldn't be amazed, we've always been like this.

Re: India's top court angry after junior judge cites fake AI-generated orders

#22

Post by fidotron »

codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

Using an LLM to automate is simply the newer cheaper outsourcing with much of the same entertainment, but less food poisoning and air travel.

Over the last 20 years a lot of engineering (proper eng, not software) work in the west has been outsourced to cheaper places, with the certified engineers simply signing off on the work done elsewhere. This results in a cycle of doing things ever faster/more cheaply and safeguards disappearing under the pressure to go ever cheaper and faster.

As someone else pointed out, LLMs have just really exposed what a degraded state we have headed into rather than being a cause of it themselves. It's going to be very tough for people with no standards - they'll enjoy cheap stuff for a while and then it will all go away. Surprised Pikachu faces all round.

(I'm pro AI btw, just be responsible.)

Re: India's top court angry after junior judge cites fake AI-generated orders

#23

Post by lukan »

the_af wrote: post #16
codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

They cannot even claim they weren't aware of the danger. LLM hallucinations have been a discussed topic, not some obscure failure mode. Almost every article on problems with AI mentions this. So the judge was lazy, incompetent, or both.

Not just discussed, but under every chat interface explicitely mentioned "This tool can make misstakes"

(Sure, more honest would be "this tool makes stuff up in a convincing way")

Re: India's top court angry after junior judge cites fake AI-generated orders

#24

Post by moron4hire »

duskdozer wrote: post #8
voidUpdate wrote: post #2

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

I'm continually amazed at how much faith people have in them. I guess since they can sound like people and output really authoritative and confident text it just overrides any skepticism subconsciously?

It's mind boggling how much people claim to like LLMs when you would never design any other piece of software to operate like LLMs do. Designing a system that interact with the user through natural text creates an awful experience. It slows down every interaction as you dig through all the prose to get to the key information. It turns every computer interaction into a school math word problem.

Re: India's top court angry after junior judge cites fake AI-generated orders

#25

Post by kingstnap »

codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

> excuse to blame the tool

The issue is ultimately blaming people doesn't really solve things. Unless its genuinely a one-of-a-kind case. But if this happened once its probably going to happen again, and this isn't the first such case of LLM hallucinations in law.

It's weird to think this way, because its easy to just point at a person for a specific instance, but when you see something repeat over and over again you need to consider that if your ultimate goal is to stop something from happening you have to adjust the tools even if the people using them were at fault in every case.

Re: India's top court angry after junior judge cites fake AI-generated orders

#27

Post by fidotron »

cmiles8 wrote: post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

> Some human still has to be accountable. Someone has to get fired / go to jail when something screws up.

The turning point will be when threatening an AI with being unplugged for screwing up works in motivating it to stop making things up.

Some people will rightly point out that is kind of what the training process is already. If we go around this loop enough times it will get there.

Re: India's top court angry after junior judge cites fake AI-generated orders

#28

Post by hek2sch »

cmiles8 wrote: post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

Isn't just the issue stemming simply from not using the right tool? When the stakes are high and you should be checking details, the right tools are grounded Ai solutions like nouswise and notebooklm and not the general purpose chatbots that almost everyone knows they might hallucinate. I also do believe that this use case is definitely a low hanging fruit to automat a lot of manual work but it comes with new requirements like transparency to help with verifying the responses.

Re: India's top court angry after junior judge cites fake AI-generated orders

#30

Post by n1b0m »

chris_wot wrote: post #18

In Australia, our universities are finding that a large proportion of Indian students have been using GenAI for cheating. Often they get away with it. I'm not saying that people other than Indian overseas students cheat, but it does seem more entrenched. I'd love to know why. It doesn't actually help in the long term!

Indian students have embraced GenAI at a rate significantly higher than the global average, with nearly 90% of students in some surveys actively using these tools.

Government Policy and National Initiatives: The National Education Policy (NEP 2020) has shifted the focus toward digital literacy. The government has introduced AI as a skill subject for younger grades and launched programs like AI for All to promote nationwide awareness.

Post reply on HN