imho, these tools would be access read-only data in datalakes, not like production dbs with write access.
Generating SQL with LLMs for fun and profit
11–20 of 29 posts
Re: Generating SQL with LLMs for fun and profit
#12superuser attack is hard to defend, no matter if that user is meat or metal.
Re: Generating SQL with LLMs for fun and profit
#13There is nothing malicious about Alter Table or Drop Table commands. These all have valid use-cases and is not something an LLM needs to guard against. If a bad-actor can issue these commands against your DB, you are already toast!
> If a bad-actor can issue these commands against your DB, you are already toast! Don't overlook the damage potential of a fresh-faced college-hire on-call at 2am with dba access to prod
Again, you are already toast. No one should have access to prod except the promote automation.
Re: Generating SQL with LLMs for fun and profit
#14There is nothing malicious about Alter Table or Drop Table commands. These all have valid use-cases and is not something an LLM needs to guard against. If a bad-actor can issue these commands against your DB, you are already toast!
> If a bad-actor can issue these commands against your DB, you are already toast! Don't overlook the damage potential of a fresh-faced college-hire on-call at 2am with dba access to prod
The college-hire is not the problem; it is the person who gave the kid prod access, or perhaps at a higher level, the person who architected the DB permissions structure. If one person alone can cause a production SEV, many things had to have gone wrong by many other people beforehand.
Re: Generating SQL with LLMs for fun and profit
#15OpenAI themselves ha(d/s) an over-the-top filter to "prevent copyright issues" which prevents it from reciting the litany or "it was the best of times.."
Why not have at a minimum a strict blacklist of which words you do not permit in the output - Kill the model immediately if it has it and flag user for review (After some smoke testing you can have a non-connected GPT instance evaluate it before it wastes a persons time, but if there's one thing I've learned from these early days of LLMs, it's that you do NOT want the general denizens of the internet to have access to it through you. OpenAI had to update their terms of service when they saw what they were getting requests for.)
A better solution solution might be more along the lines of a restricted whitelist of words that either the model itself, or the model + NLP, or model + NLP + another model etc cajoles into being both not useless and guaranteed to include not a single word you didn't intend. I guess you could call it CorpusCoercion
I would consider this mandatory for e.g. generating any content for children. The equivalent for lawyers is to whitelist in the actual correct legal precedents and their names so it can't make them up :)
LLM induced Laziness and greed are already here and will only get worse, build your kill switches and interlocks while you can on what you can.
Also GPT will often happily generate python code that will run for hours, and then suddenly you realize that the kernel is about to invoke oomkiller in a minute. Even without malicious intent you can get some interesting garbage out of webchat gpt3 models - though "build me an analysis tree of this drive" is probably a mild risk without some containerization.
I would also bet decent money the privilege escalation prompt was in part (maybe a large one) the result of openai making gpt3 cheaper and worse, they probably saw the ability to save compute by using what you provided (this is the only way to get half decent code out of it..). I would be very surprised if gpt4 (the unmodified one via API) falls for it.
Re: Generating SQL with LLMs for fun and profit
#16There is nothing malicious about Alter Table or Drop Table commands. These all have valid use-cases and is not something an LLM needs to guard against. If a bad-actor can issue these commands against your DB, you are already toast!
I've only ever queried (very large) databases but my eyes always go a bit wide when i see statements that touch tables. They scare me. They scare me when i run them on an sqlite table i made 5 minutes ago for an experiment.
I see the problem as much, much more insidious and not the expected threat vector. The past few months many of us have seen these models become increasingly worse at keeping track of details and hallucinating.
They mix in information within their context window, and the cope that OpenAI has given us for their worse ability to generate good quality output is .... more context! Great.
So what happens when that context window (which you have no real idea how they're actually implementing it) has the concept of "DROP" in it? Or what happens when It's a long day, you looked over it and it's all correct, but in some buried inner query something changed? Probably it just costs some time to debug, bu..
Obviously there should be a few safeguards before that query gets executed but i never want to see an increasingly cheapening and more wide-spread black box like GPT be able to "speak" a word which in principle can cost 6-7 figure damages or worse.
We don't let actively hallucinating people brandish firearms for a reason
Re: Generating SQL with LLMs for fun and profit
#17There is nothing malicious about Alter Table or Drop Table commands. These all have valid use-cases and is not something an LLM needs to guard against. If a bad-actor can issue these commands against your DB, you are already toast!
> If a bad-actor can issue these commands against your DB, you are already toast! Don't overlook the damage potential of a fresh-faced college-hire on-call at 2am with dba access to prod
I remember having a test suite that would connect to a local db running in a docker container and would nuke the tables and then set up the records in a known state before running through. Worked great until someone changed the connection string to point at an actual database.
Re: Generating SQL with LLMs for fun and profit
#18I think rather than just throwing this type of amazing ability out entirely due to potential malicious users, you can consider things like: - using for internal tools with relatively small userbases such as employees in your department. - using it with GPT-4 instead of 3.5 which can do a much better job of detecting malicious use. - make a read-only copy of just the data that you want to expose. - use a similar strat…
I like the part where the solution to LLMs being fundamentally the wrong interface is more LLMs.
Re: Generating SQL with LLMs for fun and profit
#19There is nothing malicious about Alter Table or Drop Table commands. These all have valid use-cases and is not something an LLM needs to guard against. If a bad-actor can issue these commands against your DB, you are already toast!
> If a bad-actor can issue these commands against your DB, you are already toast! Don't overlook the damage potential of a fresh-faced college-hire on-call at 2am with dba access to prod
When I was 18, at my first dev job, I was put in charge of trying to migrate and modernize an old PHP app for a client.
I had been there maybe 6 months. For some reason I can't recall I was meant to delete the staging RDS database.
Well, the databases weren't given human names, they were both long autogenerated strings.
I deleted the staging database and then prod stopped working
Whoops
Re: Generating SQL with LLMs for fun and profit
#20I think rather than just throwing this type of amazing ability out entirely due to potential malicious users, you can consider things like: - using for internal tools with relatively small userbases such as employees in your department. - using it with GPT-4 instead of 3.5 which can do a much better job of detecting malicious use. - make a read-only copy of just the data that you want to expose. - use a similar strat…
I like the part where the solution to LLMs being fundamentally the wrong interface is more LLMs.
LLMs are fundamentally the right interface. The dream of SQL was always to enable more people to query their data with something closer to natural language. Having an LLM in front finally makes this use case feasible. Current LLM capabilities are actually perfect for translating natural language to SQL.