Live data from Hacker News

India's top court angry after junior judge cites fake AI-generated orders

bbc.com

141–150 of 212 posts

Re: India's top court angry after junior judge cites fake AI-generated orders

#141

Earlier quoted context omitted.

>why so many companies are now saying they see zero ROI from AI efforts. I strongly suspect this is because workers are pocketing the gains for themselves. Report XYZ usually takes a week to write. It now takes a day. The other 4 days are spent looking busy. The MIT report that found all these companies were getting nowhere with AI, also found that almost every worker was using AI almost daily. But using their person…

If that were the case, this site and certain subreddits would have a lot of posts and comments with people crowing about how much time they are getting back. I haven’t seen that, but I haven’t gone looking for it either.

> people crowing about how much time they are getting back

Trust me, if it wasn’t for RTO, I probably would be lol.

Re: India's top court angry after junior judge cites fake AI-generated orders

#142
post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

> You can make humans more productive

If productivity is 10x unless work increases 10x jobs will be gone.

Re: India's top court angry after junior judge cites fake AI-generated orders

#143
post #123
post #83

Earlier quoted context omitted.

> won't solve the core problem. what's the core problem tho? Because if the core problem is "using ai", then it's an inevitable outcome - ai will be used, and there are always incentive to cut costs maximally. So realistically, the solution is to punish mistakes. We do this for bridges that collapse, for driver mistakes on roads, etc. The "easy" fix is to make punishment harsher for mistakes - whether it's LLM or not…

The core problem is that the tool provides output that looks right and is right a lot of the time, but also slips in incorrect stuff in a hard to notice way. Punishment isn't a problem because it doesn't work. If you create a system that lulls people into a sense of security, no punishment will stop them because they aren't doing it thinking "it's worth the risk", it's that they don't see the risk. There are so many…

> The correct solution was automated tooling that validates the URL

that's because this particular problem has a solution.

The issue here is that there's no such a tool to automatically validate the output of the LLM - at least, not yet, and i don't see the theoretical way to do it either.

And you're making the punishment as being getting fired from the job - which is true, but the company making the mistake also gets punished (or should be, if regulatory capture hasn't happened...). This results in direct losses for the company and shareholders (in the form of a fine, recalls and/or replacements etc).

Re: India's top court angry after junior judge cites fake AI-generated orders

#144
This is going to be a huge problem in every sector. I have been exploring solution in this space for fintech and so far what Resemble AI is doing [1] is probably the best way to defend.

The surface level for us is not just LLM generated text, it is also the combination of AI augmented audio (for incoming calls) and then for our own voice agents being able to protect and identify services cloning our own agent voices with watermarking.

It's not fun, as we are constantly catching up.

[1] https://www.resemble.ai/detect/

Re: India's top court angry after junior judge cites fake AI-generated orders

#145
post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

>why so many companies are now saying they see zero ROI from AI efforts. I strongly suspect this is because workers are pocketing the gains for themselves. Report XYZ usually takes a week to write. It now takes a day. The other 4 days are spent looking busy. The MIT report that found all these companies were getting nowhere with AI, also found that almost every worker was using AI almost daily. But using their person…

That’s certainly a … convenient … explanation.

Re: India's top court angry after junior judge cites fake AI-generated orders

#146
post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

>why so many companies are now saying they see zero ROI from AI efforts. I strongly suspect this is because workers are pocketing the gains for themselves. Report XYZ usually takes a week to write. It now takes a day. The other 4 days are spent looking busy. The MIT report that found all these companies were getting nowhere with AI, also found that almost every worker was using AI almost daily. But using their person…

While not dispositive of your idea, I think some portion of people using their personal accounts is because we collectively lack good feedback loops on the effectiveness of “AI addons” like RAG. The corporate accounts can be legitimately less useful than a “stock” account because the AI team integrates everything under the sun to show value, but the integrations become a net negative.

Ie ones that index entire company wikis. It ends up regurgitating rejected or not implemented RFCs, or docs from someone’s personal workflow that requires setting up a bunch of stuff locally to work, or etc.

A lot of tasks are not dependent on internal documentation, and it just ends up polluting the context with irrelevant, outdated or just wrong information.

Re: India's top court angry after junior judge cites fake AI-generated orders

#147

Earlier quoted context omitted.

I don't know if you've ever been to India, but one of its characteristic features is that it has lots of local languages. LLMs are awful at almost all of them. Plus, there's 20ish% of the population that falls below the literacy threshold. It's hard to imagine how those people would be educated by LLMs even if that was a good idea and they all had reliable Internet access, which they often don't.

Why’s it hard to imagine? More training data will solve whatever language lapses it has. The next miracle is that TTS is perfect now, so they don’t need to be able to read. You can convey abstract concepts as alternate abstractions, explain like I’m five but on turbosteroids. It’s the ultimate teaching tool and it’s about to be ubiquitous.

What training data? Many of these languages have very little digitized literature. Even if we assume they have sizeable extant corpuses (e.g. Tibetic/Bhoti), that's not enough. LLMs are still pretty garbage at English prose, for example.

Re: India's top court angry after junior judge cites fake AI-generated orders

#148
post #93

> Senior judges at the Supreme Court in Delhi have threatened consequences over the use of AI Setting AI aside for a moment, this reflects a broader issue in India and elsewhere. When institutions respond to new technologies with anger or threats rather than systemic thinking, it signals a deeper problem. The real challenge is not AI itself, but how complex systems adapt to change. Instead of reacting defensively, in…

No, you are reading it wrong; The Indian Supreme Court is doing the right thing.

The lower courts in India are all overloaded with pending cases (i.e. not enough judges) and so the incentives for both judges and lawyers to "outsource" to AI is very high. This needs to be done with caution and that is what the supreme court said, viz;

The Supreme Court called the case a matter of "institutional concern" and said fake AI-generated judgements had "a direct bearing on integrity of adjudicatory process".

...

The defendants challenged the order in the state's high court, pointing out that the cited orders were fake. The high court acknowledged this, but accepted that the junior civil judge had made the error in "good faith" and went on to agree with the trial court's decision anyway.

In its order, the high court said that "the citations may be non-existent, but if the learned trial court has considered the correct principles of law and its application to the facts of the case is also correct, mere mentioning of incorrect or non-existent rulings/citations in the order cannot be a ground to set aside the order".

The high court had also sought a report from the junior judge who had used the AI-generated rulings. She told the court that this was her first time using an AI tool and she had believed the citations to be "genuine". She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote.

The high court also advocated for the "exercise of actual intelligence over artificial intelligence".

Following this, the defendants appealed again, taking the matter to the Supreme Court, which was less forgiving about the impact of AI.

Coming down sternly against the fake judgements, the top court last Friday stayed the lower court's order on the property dispute. It said the use of AI while making judgements was not simply "an error in decision making" but an act of "misconduct".

"This case assumes considerable institutional concern, not because of the decision that was taken on the merits of the case, but about the process of adjudication and determination," the top court said.

PS: To get an idea of how overloaded the Indian Judicial System is; this happened in a recent case in Allahabad High Court - The order then took an unusual turn. “Since I am feeling hungry, tired and physically incapacitated to dictate the judgment, the judgment is reserved,” the judge recorded. - He had been hearing more than 30 cases on that day - https://www.hindustantimes.com/india-news/hungry-tired-allah...

Re: India's top court angry after junior judge cites fake AI-generated orders

#149
post #124

Earlier quoted context omitted.

I do think that for this particular situation we need to step outside of our tech bubble a little bit. I am still having regular conversations with people that either don't know about hallucinations or think they are not a big problem. There is a ton of money in these companies pushing that their tools are reliable and its working for the average user. I mean there are people that legitimately think these tools are c…

I find it hard to believe the people who use AI haven't read a single article about AI. That would also disqualify this judge, if it were true. This exceeds the tech bubble. My local newspaper, completely clueless about tech, runs an article about AI trouble, hallucinations and whatnot every other week. Completely missing most of the nuances, of course, but my point is that this has entered the public discourse.

It may have entered public discourse but it is not being talked about as much outside of tech spaces, and we are up against the companies pushing the complete opposite narrative.

All I can say is that I am having conversations with non technical people regularly that are not aware of the issue or think it is a largely solved issue.

Post reply on HN