Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing a SQL analyst with 26 recursive GPT prompts

patterns.app

Post reply on HN
Print view View original

201–210 of 311 posts

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#201

Post by kieselguhr_kid »

RhodesianHunter wrote: post #140
kieselguhr_kid wrote: post #131

Earlier quoted context omitted.

When I was new, I saw one of my more experienced colleagues ask a few questions that together saved the company more than $1 mil each year. ChatGPT might be a threat to automate some low-level tasks or help eliminate bugs, but it is nowhere near ready to evaluate the context of a system, understand its history, or think* through the consequences of a major business decision. * or think at all, in any meaningful way.

I agree with you to a point, but I think the only reason that it can't understand the context of a system is because it hasn't been trained on that system's code and documentation, which is obviously a future coming soon.

I'm not sure training these models on code and documentation will make that much of a difference. These models struggle significantly with subtlety, relevance, and correctness. It also doesn't have a theory of its own knowledge or confidence, and so tends to "hallucinate" and put out confidently-worded nonsense. Especially for complex and nuanced topics.

A big part of my job in software is having a very sharpened grasp of my ignorance, the ability to weigh a variety of tradeoffs, and the ability to convey my confidence of my abilities and my team's abilities. I'm not sure this is possible for this generation of AI.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#202

Post by clbrmbr »

fifilura wrote: post #123
ellisv wrote: post #19

Earlier quoted context omitted.

I think that's because most SQL analyst who can understand the data model, assumptions, quirks, etc. usually get promoted into other positions.

Unless they actually like what they do... It can actually be pretty rewarding to be the person who knows most about the data in the company, while solving logic puzzles during the day. PS. i do hope most analysts solve more interesting problems than the ones in TFA.

Is that TFA as in RTFM?

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#203

Post by xwolfi »

kvh wrote: post #192
mtc010170 wrote: post #174

Hmmm... I'm surprised I'm not seeing anyone else question the validity of this taking "2 hours" Given that it's written on the blog for the product it's using, this reads to me a lot like a pure sales pitch. They want us to believe if you use Patterns (which is neat), your company will be much more cost-effective. I'm not saying that's bad - that's probably the right thing to do with their company blog, and it's cool…

Author here, I’ve updated the post. The first draft of this app and blog post took me two hours, but I kept coming back with new ideas and tweaks throughout the week. By the end, I’d certainly spent more than two hours (more like 8?), so you’re right, I just failed to update the post. The main point stands — it’s surprisingly good for the amount of effort put in (although unclear how much more juice you could get out…

Can we replace a webmaster with 26 chatgpt prompts?

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#206

Post by fuy »

tragomaskhalos wrote: post #45

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

> slather a battery of views over your data

One needs to be careful with this approach in terms of query performance, though. Using simple views with a couple of joins and some filtering is fine, but be very wary of stacking more than 1-2 layers of views calling each other, and especially of using things like aggregates/window functions in views, if these views then are then used as building blocks for more complex queries.

That's a recipe for breaking query optimizers and ending up with very bad query plans.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#207

Post by simonw »

rexreed wrote: post #147
simonw wrote: post #145

Earlier quoted context omitted.

It's using the GPT-3 API from OpenAI. The article describes how the prompt itself is constructed - to include details of the database schema plus some example data rows, followed by the user's question. The prompt is then sent to the GPT-3 API. The results are then sent through the API a few more times with some extra prompt engineering before deriving the final result by running the SQL against the original database…

How is it being fine-tuned to be able to generate relevant SQL queries on the crunchbase data set? How does this jibe with the article saying they imported 2.5m records from crunchbase? Where and how are you able to import or fine tune the API-accessible GPT-3 this way? Any pointers for where I can look to do the same? Edit: I noticed you edited your comment above with some details. It seems that he used GPT-3 just t…

This is genuinely the cutting edge of how you do interesting things with language models like GPT-3 at the moment.

Training these models with extra data turns out to be incredibly expensive and relatively ineffective.

Instead, the most interesting research is all around tricks like this - figuring out ways to round-trip to the language model, then query other sources of data for the information that it needs, then sending more prompts to the language model again.

I wrote a tutorial about a pattern for doing that a couple of weeks ago, but this SQL trick is a lot more sophisticated than what I've done so far: https://simonwillison.net/2023/Jan/13/semantic-search-answer...

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#208

Post by OOPMan »

xwolfi wrote: post #203
kvh wrote: post #192

Earlier quoted context omitted.

Author here, I’ve updated the post. The first draft of this app and blog post took me two hours, but I kept coming back with new ideas and tweaks throughout the week. By the end, I’d certainly spent more than two hours (more like 8?), so you’re right, I just failed to update the post. The main point stands — it’s surprisingly good for the amount of effort put in (although unclear how much more juice you could get out…

Can we replace a webmaster with 26 chatgpt prompts?

I reckon we can replace a shill with less

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#209

Post by devmor »

kvh wrote: post #192
mtc010170 wrote: post #174

Hmmm... I'm surprised I'm not seeing anyone else question the validity of this taking "2 hours" Given that it's written on the blog for the product it's using, this reads to me a lot like a pure sales pitch. They want us to believe if you use Patterns (which is neat), your company will be much more cost-effective. I'm not saying that's bad - that's probably the right thing to do with their company blog, and it's cool…

Author here, I’ve updated the post. The first draft of this app and blog post took me two hours, but I kept coming back with new ideas and tweaks throughout the week. By the end, I’d certainly spent more than two hours (more like 8?), so you’re right, I just failed to update the post. The main point stands — it’s surprisingly good for the amount of effort put in (although unclear how much more juice you could get out…

It doesn't seem like you've really replaced anyone with this. You spent 8 hours doing the work that you could have paid an SQL analyst to do in much less.

Unless you're saying that your time is worth less than you'd pay the analyst?

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#210

Post by jhugo »

devmor wrote: post #209
kvh wrote: post #192

Earlier quoted context omitted.

Author here, I’ve updated the post. The first draft of this app and blog post took me two hours, but I kept coming back with new ideas and tweaks throughout the week. By the end, I’d certainly spent more than two hours (more like 8?), so you’re right, I just failed to update the post. The main point stands — it’s surprisingly good for the amount of effort put in (although unclear how much more juice you could get out…

It doesn't seem like you've really replaced anyone with this. You spent 8 hours doing the work that you could have paid an SQL analyst to do in much less. Unless you're saying that your time is worth less than you'd pay the analyst?

They built a bot which can answer any number of questions, each of which would have needed some analyst time. Given that the analyst rotation was an entire day once every N weeks, and the bot took 1 day to make, this is going to pay for itself after 1 week.

This all assumes that the bot doesn't need tweaking for every answer — i.e. it gets at least some answers right without needing modifications to the bot — which appears to be the case based on the examples in the post.

Post reply on HN