Live data from Hacker News

Replacing a SQL analyst with 26 recursive GPT prompts

patterns.app

81–90 of 311 posts

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#81

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

> doesn't seem that GPT is significantly better or worse than a human in this regard

Probably, but I think a human is much more likely to realize that they made a mistake and good-luck convincing GPT that it's wrong about something non trivial.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#82
We looked at using all sorts of "AI" to write SQL based upon natural language prompts. As far as I am aware, the state of the art is still nowhere close enough in accuracy for us to lean into as a business.

This is the leaderboard I keep an eye on: https://yale-lily.github.io/spider

Ultimately, I don't think we will get there with semantic analysis or GPT-style techniques. There is always some human factor involved with whatever schema is developed, so you would probably need an AGI trained in the same business as whoever is being replaced by this thing.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#83
post #20

The problem is that you never really know whether the chat bot gets it right or terrifically wrong unless you already know how to do the task yourself. And in some cases, paying an analyst $50/hr. for a higher degree of confidence than you can get from a $1 chat bot is still very much worth it. The stakes are higher, too. If the chat bot gets it wrong, what are you going to do, fire it? There goes a small trickle of…

In my experience, in many cases, the so-called "first draft" (of code, content, etc.) is more just helpful for a mental warm-up. It isn't actually suitable. The time it takes to write/code/etc. isnt in the typing, that's trivial. It's in modelling the problem (, audience) and solving it. ChatGPT isnt in the game of modelling problems and solving them, it possesses only superficial semantic characteristics of the ques…

> ChatGPT isnt in the game of modelling problems and solving them, it possesses only superficial semantic characteristics of the questions posed

You can give it the datamodel and a semantic description and it will do a pretty good job of reprojecting your data.

People talk about chatgpt as being a fixed thing, it isn't. ChatGPT is the Apple II of LLMs. 3-4 moore units from now every issue that people think is a "gotcha" for chatgpt will have evaporated.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#84
post #20

The problem is that you never really know whether the chat bot gets it right or terrifically wrong unless you already know how to do the task yourself. And in some cases, paying an analyst $50/hr. for a higher degree of confidence than you can get from a $1 chat bot is still very much worth it. The stakes are higher, too. If the chat bot gets it wrong, what are you going to do, fire it? There goes a small trickle of…

> $50/hr

Is this somewhere's going rate for a good data analyst?

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#86

You didn't replace a SQL Analyst, you just gave them a query generator. End data consumers don't understand the data model, assumptions, quirks, etc. If they fire the analyst, they are going to wind up drawing a lot of bad conclusions on anything more complicated than simple aggregations.

Maybe 5% of the business/operations people I've worked with would even want to do this. The rest prefer a smart human to walk them through building a report - most of the time this is because they don't actually know what they need, and they actually need an expert to sit down and figure it out with them.

The longer I'm in the business, the more I feel like my value as a software developer is identifying and asking clarifying questions.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#87

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

I've done too little SQL to be traumatized by it, but I experienced it plenty of times doing more complex data processing on the backend.

I would e.g. spend a week on first discussing a solution, then researching both theoretical background and relevant libraries, and finally writing code to do the math and render the answers (probability distributions). I eyeball them, they seem to all match expectations, so I declare victory, post some nice screenshots to team Slack, and close the ticket.

Feeling high on success, I expect the weekend to start nicely - except, winding down on that Friday evening, I follow that little thought at the back of my head, that I realize was there since the start, quietly poking for my attention. I run one more sanity check, different from the ones I did before. The results come out completely wrong. Only then it dawns on me that I made a terrible mistake early on - I misunderstood how to implement certain step and forgot about a theoretical assumption that must hold. And so, instead of a relaxed weekend, I now dread having to explain to everyone on Monday that a) I actually need another week for this, and b) the nicely looking graphs they saw are all pure bullshit, and the things that looked correct only seemed so, and only by accident.

After a few such cases - including one where the higher-ups were nonplussed, because after I corrected the mathematical errors someone made a long time ago, the pretty graphs that looked nice on the demo stopped looking so nice - I learned to sanity-check my understanding of the problem as early as possible, and then also sanity-check the code I'm writing, and then keep testing as I go. It's all too easy to write "trivial" code that "obviously" does what you think it does, only to realize some time later that it's throwing out an entire dimension of the input data, and nobody noticed because test examples are too trivial, results look roughly correct, and no one is paying attention.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#88

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

>> The best fix imo is to slather a battery of views over your data to minimise the risk of getting the joins wrong, and it'd be interesting to see how that approach could improve the bot's quality. Yes, the success of the bot's query (at least on GPT-3) is a combination of (a literal and good) prompt and how well did you explain each of the columns to the model.

[dead]

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#89
post #71

Earlier quoted context omitted.

Glad i'm not the only one who saw this. Feels extremely overblown. Very impressive, don't get me wrong, and is yet another bridge closer to letting business side do plain text queries. But that said, stuff like this has existed for awhile, and it's certainly not replacing analysts. For a start, this assumes a VERY clean model, which is of course what you strive for, but the moment you step out of FAANG you'll find th…

>the moment you step out of FAANG Even in FAANG, most data is going to be extremely messy. * There's usually very little incentive for good documentation, so you have to guess what the table contains based on column names and exploring patterns in the data. * Enforcing a single pattern/process for data across the entire org is a pipe dream. * Logging tables are written to by hundreds of engineers across dozens of tea…

Oh yeah im not surprised to hear that, Ive just known one or two people who've been in similar sized companies as analysts, and while the underlying table structure was a nightmare, the data model they dealt with was pretty clean.

But with that in mind, thats because theres a major pipeline of people and processes to get the data to that point, and it meant that there could sometimes be significant delay on new KPI's as they had to be cleanly worked into the model, and it of course didn't represent everything.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#90

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

>> The best fix imo is to slather a battery of views over your data to minimise the risk of getting the joins wrong, and it'd be interesting to see how that approach could improve the bot's quality. Yes, the success of the bot's query (at least on GPT-3) is a combination of (a literal and good) prompt and how well did you explain each of the columns to the model.

[dead]
Post reply on HN