Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing a SQL analyst with 26 recursive GPT prompts

patterns.app

Post reply on HN
Print view View original

51–60 of 311 posts

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#51

Post by ExxKA »

mjburgess wrote: post #37
jawns wrote: post #20

The problem is that you never really know whether the chat bot gets it right or terrifically wrong unless you already know how to do the task yourself. And in some cases, paying an analyst $50/hr. for a higher degree of confidence than you can get from a $1 chat bot is still very much worth it. The stakes are higher, too. If the chat bot gets it wrong, what are you going to do, fire it? There goes a small trickle of…

In my experience, in many cases, the so-called "first draft" (of code, content, etc.) is more just helpful for a mental warm-up. It isn't actually suitable. The time it takes to write/code/etc. isnt in the typing, that's trivial. It's in modelling the problem (, audience) and solving it. ChatGPT isnt in the game of modelling problems and solving them, it possesses only superficial semantic characteristics of the ques…

This

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#52

Post by g051051 »

> this is shockingly close to replacing an entire role at companies with only a couple hours of effort.

and

> It seems like there’s almost no limit to how good GPT could get at this.

I don't see how that's a valid conclusion given the results. 2 simple things right, moderate to difficult things wrong? Hardly a ringing endorsement.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#53

Post by ExxKA »

knodi123 wrote: post #46
LeanderK wrote: post #26

I love those ChatGPT projects! Of course it's silly and nobody is really replacing somebody with a program that confidently get half it's answers wrong. But it's fun to just naively let ChatGPT solve the problem. But I wonder what it's going to look like in a few years. Currently, it's really just a demo that got surprisingly huge traction. I think the most pressing problem is not to get ChatGPT smarter but to get it…

> nobody is really replacing somebody with a program that confidently get half it's answers wrong I gotta tell you, I use copilot to help with my coding, and it still sends a shiver down my spine when it writes the entire database migration based off of the filename, or writes 4 pages of automated tests that work right on the first try.

Which programming languages do you use?

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#54

Post by typon »

LastTrain wrote: post #14
jeltz wrote: post #6

Earlier quoted context omitted.

Yeah, it doesn't. A friend of mine played around with this and on trickier questions you got confidently incorrect answers.

That is almost worse than it completely not working.

Exactly my experience trying to get GPT-3 to produce SQLITE queries. Anything with a CTE, it gets hopelessly wrong. I was super excited to show off this to my team and after one too many false results I just quietly swept this idea under the rug.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#55

Post by layer8 »

knodi123 wrote: post #46
LeanderK wrote: post #26

I love those ChatGPT projects! Of course it's silly and nobody is really replacing somebody with a program that confidently get half it's answers wrong. But it's fun to just naively let ChatGPT solve the problem. But I wonder what it's going to look like in a few years. Currently, it's really just a demo that got surprisingly huge traction. I think the most pressing problem is not to get ChatGPT smarter but to get it…

> nobody is really replacing somebody with a program that confidently get half it's answers wrong I gotta tell you, I use copilot to help with my coding, and it still sends a shiver down my spine when it writes the entire database migration based off of the filename, or writes 4 pages of automated tests that work right on the first try.

> writes 4 pages of automated tests that work right on the first try

Not sure that’s the right criterion to assess the quality and adequacy of tests.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#56

Post by rowls66 »

Based on the natural language query provided,

"Who were the largest biotech investors in 2022?"

I can think of at least six possible answers based on these questions: 1. Does largest mean dollar amount, or number of investments? 2. Would number of investments count companies invested in or funding rounds invested in? 3. Does largest mean the largest total dollar amount invested in 2022, or does it mean larges dollar amount of new investment in 2022?

It looks like ChatGPT chose the query to mean the investors with the largest dollar amount of new investment in 2022.

When you expand your natural language query to clarify all of these ambiguities, how far away are you from a SQL query? I am not sure, but I think that you are getting pretty close.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#57

Post by migf »

"This looks like results I would expect and seems correct" is the exact same level of quality I've encountered when using these systems. It takes someone who already knows what they're doing to parse and QA the results. I feel like this is going to potentially speed up things that an expert could eventually figure out themself, but going past the expert's own knowledge is going to be disappointing and painful.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#58

Post by typpo »

I've been building something similar that handles the dirty business of formatting a large database into a prompt. Additional work that I've found helpful includes:

1. Using embeddings to filter context into the prompt

2. Identifying common syntax errors or hallucinations of non-existent columns

3. Flagging queries that write instead of read

Plus lots of prompt finessing to get it to avoid mistakes.

It doesn't execute the queries, yet. For an arbitrary db, it's still helpful to have a human in the loop to sanity check the SQL (for now at least).

Demo at https://www.querymuse.com/query if anyone's interested

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#59

Post by whoomp12342 »

This just in, ChatGPT has hosed up the read operation due to inefficent querys and not being a human being. 26 GPT promts have been replaced with a DBA, Analyst, Project Manager, Cross functional manager, regulatory specialist, junior programmer, and QA analyst.
Post reply on HN