Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replacing a SQL analyst with 26 recursive GPT prompts

patterns.app

Post reply on HN
Print view View original

151–160 of 311 posts

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#151

Post by ajuc »

qwytw wrote: post #81
tragomaskhalos wrote: post #45

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

> doesn't seem that GPT is significantly better or worse than a human in this regard Probably, but I think a human is much more likely to realize that they made a mistake and good-luck convincing GPT that it's wrong about something non trivial.

Well it's trained to simulate humans and that includes simulating the errors.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#152

Post by kulahan »

2fast4you wrote: post #124
TeMPOraL wrote: post #87

Earlier quoted context omitted.

I've done too little SQL to be traumatized by it, but I experienced it plenty of times doing more complex data processing on the backend. I would e.g. spend a week on first discussing a solution, then researching both theoretical background and relevant libraries, and finally writing code to do the math and render the answers (probability distributions). I eyeball them, they seem to all match expectations, so I decla…

Your anecdote gave me an idea. What if we use AI to enhance the communication between people instead of replacing the people? Imagine you give a problem statement to the LLM, then it tried to find all the subtleties, confusing parts, room for interpretation, etc. All of the people involved revise the problem statement to make it more clear and give it back to the LLM. Iterate like that until satisfied

Asking the right question is surprisingly hard sometimes. I'd love a tool I could converse with to come up with a question at the core of my thoughts.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#153

Post by minsc_and_boo »

ellisv wrote: post #19
berkle4455 wrote: post #15

Earlier quoted context omitted.

You just described most SQL analysts I've worked with

I think that's because most SQL analyst who can understand the data model, assumptions, quirks, etc. usually get promoted into other positions.

Yeah, IME the good business savvy SQL analysts get sucked into non-SQL assigned roles and nailing it because they write their own scripts.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#154

Post by seandoh »

arrosenberg wrote: post #9

You didn't replace a SQL Analyst, you just gave them a query generator. End data consumers don't understand the data model, assumptions, quirks, etc. If they fire the analyst, they are going to wind up drawing a lot of bad conclusions on anything more complicated than simple aggregations.

We're building https://www.olli.ai/ to help with this. User's can ask questions in natural language, but more importantly, Olli can suggest what questions to ask based on the data.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#155

Post by karmakaze »

tragomaskhalos wrote: post #45

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

What we need is to have ML inputs judged as they're being ingested. Like PageRank for data and sources.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#156

Post by pja »

Shorel wrote: post #138
TillE wrote: post #72

Earlier quoted context omitted.

The similar thought I had yesterday was that ChatGPT is pretty good at generating sample code that doesn't otherwise exist. It can be really useful if you treat it like that. Of course I've also had it call functions and use constants which don't exist, and fail to correct itself even after repeated prompting. So even that only goes so far.

Imagine if ChatGPT (or any other competitor) had access to a compiler and a set of tests for the program, a la HackerRank. You then give it the prompt and a lot of test cases to pass in order to consider the task complete. I can see this concept being implemented and working before the end of the year.

Google / Deepmind has already done this last year. They published the paper at the end of 2022 I think. (searches) yep: https://www.deepmind.com/blog/competitive-programming-with-a...

It relies on test availability of good test cases to verify the code functionality: they generate many, many possible programs, throw out all the ones that aren't syntactically correct, compile the ones that are & then test them. This works reasonably well for programming competitions & the code generated will probably improve over time.

The issue for "real-world" problems is that you can't verify the code without the test cases & writing good tests that ensure the code does what you want can be as much work as writing the code itself in the first place - the tests form a kind of mathematical co-domain to the code after all & only a small subset of problems are so simply defined that you can pin them down with just one or two tests.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#157

Post by gptgpp »

mjburgess wrote: post #120
kietay wrote: post #106

Earlier quoted context omitted.

In your analogy to the Apple II, is there a fundamental problem people claimed for early computers, which didn’t just boil down to “they need to be faster”? There are fundamental limitations to what an LLM can achieve.

You missed the ever-powerful ellipsis in what he said: Today "..." and then it will be solved. It's the most powerful operator in the whole of applied computer science.

True, I mean, just look at all of the examples of technologies that were projected to revolutionize computing and machine learning "..." eventually, here:

https://en.wikipedia.org/wiki/AI_winter

Just because they haven't happened to yet doesn't mean they never will! I've heard very good things about expert systems and IBM's Watson for Oncology.

The Harvard Business Review has a great article on all of the eventual "..." AGIs here:

https://hbr.org/2022/03/why-ai-failed-to-live-up-to-its-pote...

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#158

Post by nickpeterson »

kulahan wrote: post #152
2fast4you wrote: post #124

Earlier quoted context omitted.

Your anecdote gave me an idea. What if we use AI to enhance the communication between people instead of replacing the people? Imagine you give a problem statement to the LLM, then it tried to find all the subtleties, confusing parts, room for interpretation, etc. All of the people involved revise the problem statement to make it more clear and give it back to the LLM. Iterate like that until satisfied

Asking the right question is surprisingly hard sometimes. I'd love a tool I could converse with to come up with a question at the core of my thoughts.

Rubber Duck AI chat bot, you explain your problem and it critiques your wording leading you sharpen your argument.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#159

Post by futhey »

tragomaskhalos wrote: post #45

Anyone who's been asked more than a couple of times for data that requires a non-trivial bit of ad-hoc SQL will know the sinking "oh shit" feeling that comes when you subsequently realise you borked the query logic in some subtle way and have accordingly emailed out a completely bogus answer/report. From the article it doesn't seem that GPT is significantly better or worse than a human in this regard, although an exp…

You can ask an analyst to double check their work, and a good one will find any such mistakes.

If you ask GPT-3 to correct an error it always confidently tells you you're wrong and it made no mistake (in my experience).

It's funny but I imagine more annoying when it's eventually integrated into a paid service people rely on.

Re: Replacing a SQL analyst with 26 recursive GPT prompts

#160

Post by jeltz »

marcosdumay wrote: post #63
uoaei wrote: post #43

Earlier quoted context omitted.

Honest question: how rampant is this kind of bullshit politicking in tech companies, and how does this differ with industry, size, company age, etc.? My friends who have worked at big names like F and T have described the cultures there as following: "it's as if everyone who studied too much in high school and college missed all the parties and bullshit, so they're making up for it 'on campus' here". And the stories…

I'm pretty sure the GP was a joke. Whoever made the decision very likely wasn't intending to punish the person. It was only the consequence.

And it is not really punishing, yet at least. He likes his new job despite the mess he was put in. He loves it because he learns so many new things (analytics, how the be a better manager, etc) and how he has managed to make his team actually deliver a lot of value to the company in spite of these problems. It is possibly that the challenge will be too big for him and it will burn him out but so far he likes it.

And, no, it is very unlikely to be intended as a punishment. More likely he got it because he was too interested in getting good reports. The usual accidental volunteering where if you are too interested in something you end up doing all the hard work.

Post reply on HN