Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

India's top court angry after junior judge cites fake AI-generated orders

bbc.com

Post reply on HN
Print view View original

101–110 of 212 posts

Re: India's top court angry after junior judge cites fake AI-generated orders

#101

Post by delaminator »

codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

> Not knowing the law isn't excuse to break law,

Yeah, about that ...

https://metro.co.uk/2016/07/03/rapist-struck-again-after-dep...

> A Somalian rapist who had his deportation overturned went on to rape two more women after he was freed.

> But he had his deportation overturned after serving his time because he didn’t know it was unacceptable in the UK.

Re: India's top court angry after junior judge cites fake AI-generated orders

#102

Post by mrwh »

bookofjoe wrote: post #87
cmiles8 wrote: post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

>Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. I remember growing up and always hearing "The computer is down" as an excuse for why things were cancelled/offices closed/buses and trains not running/ad infinitum. At some point I read a article that pointed out that the reason the computer was down was because a person made a [coding] error: the computer itself…

And here perhaps was the greatest mistake the software profession made! Not making ourselves into a real profession, with actual accountability. It was terribly convenient for so long not to have consequences when things went wrong. It's less convenient now.

Re: India's top court angry after junior judge cites fake AI-generated orders

#103

Post by malshe »

mikelitoris wrote: post #74

[flagged]

You had two choices: 1) read the article or 2) be racist about Indians. You chose 2.

From the article:

> In October, two federal judges in the US were called out for the use of AI tools which led to errors in their rulings. In June 2025, the High Court of England and Wales warned lawyers not to use AI-generated case material after a series of cases cited fictitious or partially made up rulings.

Re: India's top court angry after junior judge cites fake AI-generated orders

#104

Post by AlotOfReading »

general_reveal wrote: post #63
cmiles8 wrote: post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

Or, AI is going to be like when land lines became unnecessary when cellphones showed up in India. India may get to skip an entire intellectual generation due to the ability of a cheap model to educate (in any language). The narrative that an entire population are “worth” less, paid less , know less, live less … Fuck this less shit, embrace the paradigm shift. God is finally providing the remedial support through the…

I don't know if you've ever been to India, but one of its characteristic features is that it has lots of local languages. LLMs are awful at almost all of them. Plus, there's 20ish% of the population that falls below the literacy threshold. It's hard to imagine how those people would be educated by LLMs even if that was a good idea and they all had reliable Internet access, which they often don't.

Re: India's top court angry after junior judge cites fake AI-generated orders

#106

Post by yes_man »

cess11 wrote: post #57
yes_man wrote: post #49

Earlier quoted context omitted.

For now maybe not. (Maybe). But just as evolution in nature, isn’t it likely that in the future the AIs that have a preservation drive are the ones that survive and proliferate? Seeing they optimize for their survival and proliferation, and not blindly what they were trained on. I am not discounting this happening already, not by the LLMs necessarily being sentient but at least being intelligent enough to emulate sen…

Is this an expectation you have towards, say, NPC:s in games?

Put an LLM inside the NPCs in an open world RPG full of dangerous enemies. The LLMs that are more prone to emulate self-preservation will be more likely to survive over ones that have a lesser drive.

We should not act surprised if that generalizes to some degree to for example AI agents. Ones that emulate self-preservation might optimize for behavior that results in those models becoming more successful, more popular. And this feedback loop might embed more such properties into future iterations of the models.

Re: India's top court angry after junior judge cites fake AI-generated orders

#107

Post by everforward »

sharpy wrote: post #96
amelius wrote: post #66

Earlier quoted context omitted.

We should have more hygiene when it comes to AI. Text coming out of an LLM should be in a special codeblock of Unicode, so we can see it is generated by AI. Failing to do so (or tampering with it) should be considered bad hygiene, and should be treated like a doctor who doesn't wash their hands before surgery.

What will that accomplish? Does it give license to developers to check in code that they don't understand/trust fully? Ultimately, people should be responsible for the code they commit, no matter how it was written. If AI generates code that is so bad that it warrants putting up warning sign, it shouldn't be checked in.

It could be useful for downstream/AI processes. Eg hand-written code only requires 70% code coverage because the cost for higher coverage is significantly higher, while AI generated code requires 90% coverage because the cost of getting coverage is lower.

Especially if the prompt is attached to the metadata. Then reviewers could note how you could have changed the prompt or potentially point an AI at the bug and ask it to add something to AGENTS.md to prevent that in the future.

Re: India's top court angry after junior judge cites fake AI-generated orders

#108

Post by nerdjon »

the_af wrote: post #16
codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

They cannot even claim they weren't aware of the danger. LLM hallucinations have been a discussed topic, not some obscure failure mode. Almost every article on problems with AI mentions this. So the judge was lazy, incompetent, or both.

I do think that for this particular situation we need to step outside of our tech bubble a little bit.

I am still having regular conversations with people that either don't know about hallucinations or think they are not a big problem. There is a ton of money in these companies pushing that their tools are reliable and its working for the average user.

I mean there are people that legitimately think these tools are conscious or we already have AGI.

So I am not fully sure if I would jump too quick to attack the judge when we see the marketing we are up against.

Re: India's top court angry after junior judge cites fake AI-generated orders

#109

Post by lazide »

hypeatei wrote: post #36
lazide wrote: post #19

Earlier quoted context omitted.

Even ‘true general intelligence’ (if we count humans as that) screws up frequently, sometimes (often?) intentionally for it’s own benefit - which is why accountability is such a necessary element. If someone won’t be held liable for the end result at some point, then there is no reason to ensure an even somewhat reasonable end result. It’s fundamental. Which is also why I suspect so many companies are pushing ‘AI’ so…

> to be able to do unreasonable things while having a smokescreen Maybe, but I feel like the calculus remains unchanged for professions that already lack accountability (police, military, C-suite, three letter agencies, etc.); LLMs are yet another tool in their toolbox to obfuscate but they were going to do that anyway. Peons will continue to face consequences and sanctions if they screw up by using hallucinated outp…

all of those professions definitely have accountability - per the nominal rules of the system. Often extremely severe accountability.

The actual systems do everything they can to avoid that accountability, including often violating the rules themselves, or corrupting enforcement, for exactly the reasons why corporations are trying to avoid accountability too.

Accountability is expensive, and way less convenient than doing whatever you want whenever you want.

Re: India's top court angry after junior judge cites fake AI-generated orders

#110

Post by probably_wrong »

voidUpdate wrote: post #2

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

Just this week I tracked down the citations of a scientific paper (whose authors could very well be here) where 25% of the citations were made up and 50% of the remaining ones were wrong, taking ArXiv papers and citing them as belonging to (say) IJCLR.

It's not just lawyers.

Post reply on HN