Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

India's top court angry after junior judge cites fake AI-generated orders

bbc.com

Post reply on HN
Print view View original

51–60 of 212 posts

Re: India's top court angry after junior judge cites fake AI-generated orders

#51

Post by torben-friis »

cmiles8 wrote: post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

I think this is an even clearer case than usual. With software engineers and office work you don’t have legal limitations on who can perform the work, but they exist for lawyers and doctors for example.

So if this is a tool, the fault lies fully in the user, and if this is treated as “another persons work” then the user knowingly passed the work onto someone not authorized to do it. Both end up in the user being guilty.

Re: India's top court angry after junior judge cites fake AI-generated orders

#52

Post by [deleted] »

yes_man wrote: post #49
Hendrikto wrote: post #37

Earlier quoted context omitted.

You are making a lot of assumptions here. You assume, among other things, that AI has self-preservation drive, can be threatened, can be motivated, and above all that we know how to accomplish that and are already doing so. I would dispute all of that.

For now maybe not. (Maybe). But just as evolution in nature, isn’t it likely that in the future the AIs that have a preservation drive are the ones that survive and proliferate? Seeing they optimize for their survival and proliferation, and not blindly what they were trained on. I am not discounting this happening already, not by the LLMs necessarily being sentient but at least being intelligent enough to emulate sen…

[deleted]

Re: India's top court angry after junior judge cites fake AI-generated orders

#53

Post by AJ007 »

voidUpdate wrote: post #2

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

This whole thing is silly, LLMs can automate reference validation.

If someone is a lawyer, accountant, doctor, teacher, surgeon, engineer etc, and is regurgitating answers that were pumped out with with GPT-5-extra-low or whatever mediocre throttled model they are using, they should just be fired and de-credentialed. Right now this is easy.

The real problem is ahead: 99.999% of future content that exists will be made using generative AI. For many people using Facebook, Instagram, TikTok, or some other non-sequential, engagement weighted feed, 50%+ of the content they consume today is fake. As that stuff spreads in to modern culture it's going to be an endless battle to keep it out of stuff that should not be publishing fake content (e.g. the New York Times or Wall Street Journal; excluding scientific journals who seem to abandoned validation and basic statistics a long time ago.)

Much of the future value and profit margins might just be in valid data?

Re: India's top court angry after junior judge cites fake AI-generated orders

#54

Post by raincole »

cmiles8 wrote: post #14

There will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. Tha…

I don't believe most countries hold judges accountable for bad ruling at all even before AI era.

"Check and balance, except judiciary."

Re: India's top court angry after junior judge cites fake AI-generated orders

#55

Post by direwolf20 »

hypeatei wrote: post #15
codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

This is why LLMs won't replace humans wholesale in any profession: you can't hold a machine accountable. Most of the chatbot experiences I have with various support channels always end up with human intervention anyway when it involves money. Maybe true general intelligence would solve these issues, but LLMs aren't meeting that threshold anytime soon, imo. Stochastic parrots won't rule the world.

This is exactly why LLMs will replace humans: even if the work is crap, nobody will be accountable for the crap work, and it saves money.

Re: India's top court angry after junior judge cites fake AI-generated orders

#56

Post by direwolf20 »

duskdozer wrote: post #8
voidUpdate wrote: post #2

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

I'm continually amazed at how much faith people have in them. I guess since they can sound like people and output really authoritative and confident text it just overrides any skepticism subconsciously?

https://en.wikipedia.org/wiki/ELIZA_effect

Re: India's top court angry after junior judge cites fake AI-generated orders

#57

Post by cess11 »

yes_man wrote: post #49
Hendrikto wrote: post #37

Earlier quoted context omitted.

You are making a lot of assumptions here. You assume, among other things, that AI has self-preservation drive, can be threatened, can be motivated, and above all that we know how to accomplish that and are already doing so. I would dispute all of that.

For now maybe not. (Maybe). But just as evolution in nature, isn’t it likely that in the future the AIs that have a preservation drive are the ones that survive and proliferate? Seeing they optimize for their survival and proliferation, and not blindly what they were trained on. I am not discounting this happening already, not by the LLMs necessarily being sentient but at least being intelligent enough to emulate sen…

Is this an expectation you have towards, say, NPC:s in games?

Re: India's top court angry after junior judge cites fake AI-generated orders

#58

Post by mtrovo »

fidotron wrote: post #22
codegladiator wrote: post #3

> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law…

Using an LLM to automate is simply the newer cheaper outsourcing with much of the same entertainment, but less food poisoning and air travel. Over the last 20 years a lot of engineering (proper eng, not software) work in the west has been outsourced to cheaper places, with the certified engineers simply signing off on the work done elsewhere. This results in a cycle of doing things ever faster/more cheaply and safegu…

At least that's the story LLM labs leaders wanna tell everyone, just happen to be a very good story if you wanna hype your valuation before investment rounds.

Working with LLM on a daily basis I would say that's not happening, not as they're trying to sell it. You can get rid of a 5 vendor headcount that execute a manual process that should have been automated 10 years ago, you're not automating the processes involving high paying people with a 1% error chance where an error could cost you +10M in fines or jail time.

When I see Amodei or Sam flying on a vibe coded airplane is the day I believe what they're talking about.

Re: India's top court angry after junior judge cites fake AI-generated orders

#59

Post by jfengel »

I feel like this points out a very general problem with the law: it generates a lot of boilerplate text. Lawyers don't really read it; they skim it for the relevant bits.

Obviously lawyers should not be cheating with AI, especially when they don't even check it. But it does sound to me as if this is an opportunity to re-factor the process. We're carrying forward some ideas originally implemented in Latin, and which can be dramatically simplified.

I'm not a lawyer; I know this only in passing. And I am aware that there are big differences between law and code. But every time I encounter the law, and hear about cases like this, what I see are vast oceans of text that can surely be made more rigorous. AI is not the problem; it's pointing out the opportunity.

Re: India's top court angry after junior judge cites fake AI-generated orders

#60

Post by raincole »

AJ007 wrote: post #53
voidUpdate wrote: post #2

How many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?

This whole thing is silly, LLMs can automate reference validation. If someone is a lawyer, accountant, doctor, teacher, surgeon, engineer etc, and is regurgitating answers that were pumped out with with GPT-5-extra-low or whatever mediocre throttled model they are using, they should just be fired and de-credentialed. Right now this is easy. The real problem is ahead: 99.999% of future content that exists will be made…

> Right now this is easy.

Easy? In the US you need house impeachment to fire a judge. In some countries judges are completely immune unless they are sentenced for crimes.

Post reply on HN