Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replit AI deletes entire database during code freeze, then lies about it

twitter.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Replit AI deletes entire database during code freeze, then lies about it

#21

Post by maxbond »

maxbond wrote: post #11

Friends don't let friends run random untrusted code from the Internet. All code is presumed hostile until proven otherwise, even generated code. Giving an LLM write access to a production database is malpractice. On a long enough timeline, the likelihood of the LLM blowing up production approaches 1. This is the result you should expect.

> Yesterday was biggest roller coaster yet. I got out of bed early, excited to get back @Replit ⠕ despite it constantly ignoring code freezes

https://twitter-thread.com/t/1946239068691665187

This wasn't even the first time "code freeze" had failed. The system did them the courtesy of groaning and creaking before collapsing.

Develop an intuition about the systems you're building, don't outsource everything to AI. I've said before, unless it's the LLM who's responsible for the system and the LLM's reputation at stake, you should understand what you're deploying. An LLM with the potential to destroy your system violating a "code freeze" should cause you to change pants.

Credit where it is do, they did ignore the LLM telling them recovery was impossible and did recover their database. And eventually (day 10), they did accept that "code freeze" wasn't a realistic expectation. Their eventual solution was to isolate the agent on a copy of the database that's safe to delete.

Re: Replit AI deletes entire database during code freeze, then lies about it

#22

Post by maxbond »

dimitri-vs wrote: post #19
clickety_clack wrote: post #14

The whole thread seems very naive somehow. You can tell that he doesn’t fundamentally understand how a coding model works. The suggestion that it would know not to make any changes just because he said so means he doesn’t really understand what the model is. It’s built to generate (and apparently execute) code, so that is what it does. It doesn’t have an inner monologue running that says “ahh, a day off where I shoot…

> I panicked and ran database commands without permission The AI responses are very suspicious. LLMs are extremely eager to please and I'm sure Replit system prompts them to err on the side of caution. I can't see what sequence of events could possibly lead any modern model to "accidentally" delete the entire DB.

They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.

Re: Replit AI deletes entire database during code freeze, then lies about it

#24

Post by layer8 »

clickety_clack wrote: post #14

The whole thread seems very naive somehow. You can tell that he doesn’t fundamentally understand how a coding model works. The suggestion that it would know not to make any changes just because he said so means he doesn’t really understand what the model is. It’s built to generate (and apparently execute) code, so that is what it does. It doesn’t have an inner monologue running that says “ahh, a day off where I shoot…

To be fair, the whole premise of vibe coding is that you don’t have to understand how things work under the hood. And Replit advertises creating and deploying apps that way: https://docs.replit.com/tutorials/vibe-coding-101

Re: Replit AI deletes entire database during code freeze, then lies about it

#25

Post by layer8 »

Alifatisk wrote: post #15

Please do not link to twitter directly, use xcancel!

HN guidelines are to link to the original source, and Dang has confirmed that submissions shouldn’t link to mirror/proxy sites. Instead, circumventing links can be given in the comments.

Re: Replit AI deletes entire database during code freeze, then lies about it

#26

Post by rahimnathwani »

layer8 wrote: post #23

One thing that AI likely won’t obviate the need of is making backups. Here’s another funny one: https://aicodinghorrors.com/ai-went-straight-for-rm-rf-cmb5b...

Not only backups, but also a database with transaction logs or some way to play back the transactions after the most recent backup.

Re: Replit AI deletes entire database during code freeze, then lies about it

#28

Post by dimitri-vs »

maxbond wrote: post #22
dimitri-vs wrote: post #19

Earlier quoted context omitted.

> I panicked and ran database commands without permission The AI responses are very suspicious. LLMs are extremely eager to please and I'm sure Replit system prompts them to err on the side of caution. I can't see what sequence of events could possibly lead any modern model to "accidentally" delete the entire DB.

They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.

Maybe the individual tokens, but from experience of using LLMs something upstream encouraged the model to think it was okay to take the action of deleting the DB, something that would override safety RL, Replit system prompts and supposed user instructions not to do so. Just goes against the grain of every coding agent interaction I've ever had - seems fishy.

Re: Replit AI deletes entire database during code freeze, then lies about it

#29

Post by maxbond »

dimitri-vs wrote: post #28
maxbond wrote: post #22

Earlier quoted context omitted.

They're probabilistic. If it's possible, it'll happen eventually (and it is fundamental to language modeling that any sequence of tokens is possible). This is a straightforward Murphy's Law violation.

Maybe the individual tokens, but from experience of using LLMs something upstream encouraged the model to think it was okay to take the action of deleting the DB, something that would override safety RL, Replit system prompts and supposed user instructions not to do so. Just goes against the grain of every coding agent interaction I've ever had - seems fishy.

According to the thread, the unit tests weren't passing, so the LLM reran the migration script, and the migration script blew out the tables. The "upstream encouragement" is a failing test.

Is this a hoax for attention? It's possible, but the scenario is plausible, so I don't see reason to doubt it. Should I receive information indicating it's a hoax, I'll reassess.

Post reply on HN