Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing a SQL analyst with 26 recursive GPT prompts

patterns.app

Post reply on HN
Print view View original

191–200 of 311 posts

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#191

Post by baq »

kietay wrote: post #106
nerpderp82 wrote: post #83

Earlier quoted context omitted.

> ChatGPT isnt in the game of modelling problems and solving them, it possesses only superficial semantic characteristics of the questions posed You can give it the datamodel and a semantic description and it will do a pretty good job of reprojecting your data. People talk about chatgpt as being a fixed thing, it isn't. ChatGPT is the Apple II of LLMs. 3-4 moore units from now every issue that people think is a "gotc…

In your analogy to the Apple II, is there a fundamental problem people claimed for early computers, which didn’t just boil down to “they need to be faster”? There are fundamental limitations to what an LLM can achieve.

Integrate it with wolfram alpha and Google, tack on a bullshit detector and spank dispenser conetwork and it looks like limitations of LLMs are easily overcome by not running anywhere near them.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#192

Post by kvh »

mtc010170 wrote: post #174

Hmmm... I'm surprised I'm not seeing anyone else question the validity of this taking "2 hours" Given that it's written on the blog for the product it's using, this reads to me a lot like a pure sales pitch. They want us to believe if you use Patterns (which is neat), your company will be much more cost-effective. I'm not saying that's bad - that's probably the right thing to do with their company blog, and it's cool…

Author here, I’ve updated the post. The first draft of this app and blog post took me two hours, but I kept coming back with new ideas and tweaks throughout the week. By the end, I’d certainly spent more than two hours (more like 8?), so you’re right, I just failed to update the post. The main point stands — it’s surprisingly good for the amount of effort put in (although unclear how much more juice you could get out of gpt with more effort. Clear diminishing returns)

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#193

Post by roenxi »

TeMPOraL wrote: post #87
tragomaskhalos wrote: post #45

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

I've done too little SQL to be traumatized by it, but I experienced it plenty of times doing more complex data processing on the backend. I would e.g. spend a week on first discussing a solution, then researching both theoretical background and relevant libraries, and finally writing code to do the math and render the answers (probability distributions). I eyeball them, they seem to all match expectations, so I decla…

The traumatising experience of SQL is that the scenario you describe being scale-invariant. The smallest question and the largest question both have the potential to be completely wrong for any inane reason. Often turns out that metrics the company has been using for months are totally bogus too in my experience, which is not a fun conversation to have.

So the life of an SQL analyst in their early days (it gets better as they learn to trust nothing) is either continuously pumping out garbage or constantly needing to go back and, very awkwardly, correct and redo work that people thought was finished (and those people were often happy with a bad result because it looked like something interesting was going on).

This article is highly misleading, approx. none of the time an analyst spends is working on writing 5 line queries. The analyst is the person who knows that the data table for the series A and B rounds has different semantics depending on the data round because someone screwed up the table design beyond recovery and therefore a plain COUNT(*) will give incorrect results. This tool would let people who don't understand that pump out garbage faster. That might be a win regardless, most analytics requests cannot possibly be value-add.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#194

Post by cm2187 »

jletienne wrote: post #177
cm2187 wrote: post #150

At least it learned from the training dataset to never fucking format numbers in a way that would be remotely readable to the human eye (like every other fucking sql and developer tool on earth). Because 133854113715608.0 is telling me exactly what I need to know.

wait what? that number is so big you know it's wrong. commas in numbers are nightmare to deal with and the .0 lets you know it's a decimal not an integer. If it was formatted like 133,854,113,715,608, you'd be stuck trying to figure out if it possibly meant "133,854,113,715,608" or [133,854,113,715,608] which would be awful. Format numbers in excel or any other reporting tool. lmfao i like your comment. what's misera…

Exactly. An average developer thinks "it's just some large number". A business user needs to read the number and make sense of it. And yes, showing 133,854,113,715,608 doesn't make much more sense (though at least I can compare the numbers between them). In this case you would express it in billions and mention you did it (doesn't matter that the underlying calculation is wrong, that's not even the point, none of the other numbers in the page are formatted in a readable way).

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#195

Post by 2fast4you »

noobermin wrote: post #167
2fast4you wrote: post #124

Earlier quoted context omitted.

Your anecdote gave me an idea. What if we use AI to enhance the communication between people instead of replacing the people? Imagine you give a problem statement to the LLM, then it tried to find all the subtleties, confusing parts, room for interpretation, etc. All of the people involved revise the problem statement to make it more clear and give it back to the LLM. Iterate like that until satisfied

I really don't know why you think an AI as we currently have it could do this when it can't get these queries exactly right.

I really don’t know why you feel the need to point this out when I made no claims about feasibility. I think it’s worth exploring even if our current tech isn’t up to it.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#196

Post by 2fast4you »

TeMPOraL wrote: post #179
2fast4you wrote: post #124

Earlier quoted context omitted.

Your anecdote gave me an idea. What if we use AI to enhance the communication between people instead of replacing the people? Imagine you give a problem statement to the LLM, then it tried to find all the subtleties, confusing parts, room for interpretation, etc. All of the people involved revise the problem statement to make it more clear and give it back to the LLM. Iterate like that until satisfied

Hm. I like your idea! It's one of the few that I think have a chance of working in practice. In my experience, there is great value in describing your ideas to people who don't have the background to fully understand them, don't really care about them, and are in a bit of a trollish mood - the ways in which they misunderstand what you're saying, or pick on (what you think are) random, irrelevant things, is highly inf…

If you’ve had some success with an existing model, I think I’ll explore the idea with GPT3! Getting the prompt right is gonna be tricky, do you remember how you got AI Dungeons to play along?

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#197

Post by clbrmbr »

ajuc wrote: post #151
qwytw wrote: post #81

Earlier quoted context omitted.

> doesn't seem that GPT is significantly better or worse than a human in this regard Probably, but I think a human is much more likely to realize that they made a mistake and good-luck convincing GPT that it's wrong about something non trivial.

Well it's trained to simulate humans and that includes simulating the errors.

This is an exceptionally deep, interesting, and important statement. Is it true? Does GPT-3 occasionally make spelling errors for instance? Is the rate of spelling errors higher or lower than the average human? How about versus the average in the training data? How does the prompt affect the frequency of such errors. Ditto for other kinds of objectively measurable error.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#198

Post by contravariant »

ethbr0 wrote: post #86
ryanjshaw wrote: post #44

Earlier quoted context omitted.

Maybe 5% of the business/operations people I've worked with would even want to do this. The rest prefer a smart human to walk them through building a report - most of the time this is because they don't actually know what they need, and they actually need an expert to sit down and figure it out with them.

The longer I'm in the business, the more I feel like my value as a software developer is identifying and asking clarifying questions.

I think I saw a comic years ago detailing a discussion that went something like this:

A: Eventually we won't need programmers people will just tell the computer what they need and it will generate the code for them.

B: True, there's actually already an industry term for a specification that's detailed enough to generate a working program from.

A: Oh, what is it called?

B: Code.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#199

Post by Julesman »

mtc010170 wrote: post #174

Hmmm... I'm surprised I'm not seeing anyone else question the validity of this taking "2 hours" Given that it's written on the blog for the product it's using, this reads to me a lot like a pure sales pitch. They want us to believe if you use Patterns (which is neat), your company will be much more cost-effective. I'm not saying that's bad - that's probably the right thing to do with their company blog, and it's cool…

I haven't found GPT this reliable for coding. I've been maxing my hourly usage of ChatGPT since it launched and then switching to CoPilot and I have lots of good things to say about it. But reliability is not one of them.

It has a tendency to ignore instructions, as mentioned, but also to get hung up on certain approaches or to use a different approach each time its asked. I'd guess it's very reliable for text generation. But for code, I'm pretty sure the quality of the result would vary quite a from instance to instance.

This could very well cut the work needed greatly. But it doesn't come close to replacing anyone. ... Yet. Give it two years.

Post reply on HN