Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT is now finding bugs in databases

celerdata.com

Post reply on HN
Print view View original

171–180 of 204 posts

Re: ChatGPT is now finding bugs in databases

#171

Post by IanCal »

honeybadger1 wrote: post #157
IanCal wrote: post #146

Earlier quoted context omitted.

You mean it prompted you to go and get the contents of the page? Chatgpt does not access the internet when running. Bing chat does though.

No, I do not mean that. I provided it the URL and then asked it to use the contents of the CPU table from the invoke-request to strip away everything after the X and it not only did it, it even provided the example output of the array it modified. Here is an excerpt from my chat: " can you please modify the current array using the Microsoft listed CPUs in totality from the URL https://learn.microsoft.com/en-us/window…

It's not accessing that page live (or recently cached). That's not part of how it works. It's possible the model was updated and has learned from that, but it's also very possible it's just made up the answer. The old model cutoff was 2021 I think, I don't know if/what newer training has been done.

Re: ChatGPT is now finding bugs in databases

#172

Post by honeybadger1 »

simonw wrote: post #166
honeybadger1 wrote: post #157

Earlier quoted context omitted.

No, I do not mean that. I provided it the URL and then asked it to use the contents of the CPU table from the invoke-request to strip away everything after the X and it not only did it, it even provided the example output of the array it modified. Here is an excerpt from my chat: " can you please modify the current array using the Microsoft listed CPUs in totality from the URL https://learn.microsoft.com/en-us/window…

It hallucinated the list of processors based on the URL. Here's an experiment I just did to prove that it hallucinates content based on URLs: https://fedi.simonwillison.net/@simon/109998914078891574 Or try it yourself: Try these two prompts in separate ChatGPT sessions: > List the processors on this page: https://bit.ly/422t51T0 Then: > List the processors on this page: https://learn.microsoft.com/en-us/windows-hardw…

Interesting, and certainly convincing. I have a paid account, I wonder if this makes any difference? It would make sense to me that it is basing it's 'reality' off of the prevous 21h1 page(which would have existed) and it understands the goal I am trying to achieve.

Re: ChatGPT is now finding bugs in databases

#173

Post by simonw »

> After dozens of hours of training, ChatGPT still does not fully understand the TLP principle, and the predicates in some of its rewritten SQL are wrong. All things considered, however, sufficient training may help ChatGPT fully understand this logic at some point in the near future.

ChatGPT remembers nothing in between chat sessions - so "dozens of hours of training" here doesn't make sense.

What would make sense here instead is coming up with a prompt (within a sensible token length limit) that can be pasted into future ChatGPT sessions that gives it some extra context it needs based on prior experimentation.

Re: ChatGPT is now finding bugs in databases

#174

Post by simonw »

honeybadger1 wrote: post #172
simonw wrote: post #166

Earlier quoted context omitted.

It hallucinated the list of processors based on the URL. Here's an experiment I just did to prove that it hallucinates content based on URLs: https://fedi.simonwillison.net/@simon/109998914078891574 Or try it yourself: Try these two prompts in separate ChatGPT sessions: > List the processors on this page: https://bit.ly/422t51T0 Then: > List the processors on this page: https://learn.microsoft.com/en-us/windows-hardw…

Interesting, and certainly convincing. I have a paid account, I wonder if this makes any difference? It would make sense to me that it is basing it's 'reality' off of the prevous 21h1 page(which would have existed) and it understands the goal I am trying to achieve.

I have a paid account too. I'm very confident that it doesn't fetch URLs, despite doing an extremely convincing impression of being able to do so.

Try changing a few characters in that Microsoft URL and see if you still get the same results.

Re: ChatGPT is now finding bugs in databases

#175

Post by d4rkp4ttern »

jahsome wrote: post #25
ldjkfkdsjnv wrote: post #5

Today at work I hadnt used apache spark in a long time. I needed to do some quick and dirty data analytics on large datasets. Rather than read documentation, I just put the queries i needed in english into chatGPT, and it spit out perfectly working code. The queries werent that complex, but it saved me hours googling a bunch of random spark SQL syntax. The old school way is google sends you links, and through those c…

I've had numerous days just like this. The perfect use case for ChatGPT for my personal workflow has been to give me the 70-percent-of-the-way-there skeleton for a given query, Terraform module, or even just to prevent myself from lookup up the syntax of for loops in for the 30,000th time. I can then take that 70 percent chunk and bolt on my edge cases as I go. There hasn't really been a time where the answer didn't…

Slightly tangential. After interviewing a candidate, within 20 minutes they sent me a long thank you note that had several generic things that definitely did not apply to what we discussed. Very obviously ChatGPT-generated. Using ChatGPT in and of itself may not be all that bad, but this shows that the candidate did not even take the time to tailor it to our conversation. During the interview I was on the fence about pursuing the candidate but this justifies a clear reject.

Re: ChatGPT is now finding bugs in databases

#176

Post by didericis »

brutusborn wrote: post #100
didericis wrote: post #81

Earlier quoted context omitted.

Supervision requires knowing the results you want. Most of these ML projects are essentially just creating a best fit line/shape connecting a huge number of points in multiple dimensions and giving you output coordinates on that line/shape based on your input coordinates (as I understand it). The more supervision, the more you’re negating the value, as you’re basically telling it to make a shape more like something y…

I mixed up my terms. (I think) supervised learning refers to the use of labelled datasets, I meant fine-tuning (like in RLHF). In either case human input is necessary (and the humans definitely need to know what they want). In terms of generating novel ideas, I think chatGPT has shown this ability [0]. Human effort will be needed to sort the "good" ideas from the "bad", but I don't think this causes the value of the…

I understand the basics of gradient descent and was a math major, and while applications of that idea are very cool, I don’t think the weird shapes you can make are genuinely generative. I think that generative capability is illusory, it’s “just” smushing existing content together in a weird way. That best fit shape is inherently derivative and not how true creativity and thought which understands context works.

It’s like a weird parrot. But I think a parrot “understands” more because of the shared embedded evolutionary context it has.

That evolutionary history has the key to true intelligence somewhere, but personally I think it’s inevitably hidden/I don’t think we’ll ever understand how intuition and truly non-derivative, non propositional human thought works.

I also don’t think any of what I’m saying negates the value of these models. These models are fantastic autofill generators for a huge swath of different applications and can vastly improve productivity. I’m saying all this in a lot of threads where it comes up because it seems clear there’s going to be too much enthusiastic adoption, which is going to effectively destroy a lot of value of the internet.

The internet is the best tool for finding genuinely creative and novel ideas you were unexposed to that has ever existed. But it is increasingly dominated by derivative unoriginal content that drowns out what I would argue it was designed to help you find. I have no problem with derivative unoriginal content when it’s properly understood as such. I have a problem with how good these things seem to be at tricking people into following something derivative and blind, which seems very very dangerous.

Re: ChatGPT is now finding bugs in databases

#177

Post by lordnacho »

paxys wrote: post #19
crazysim wrote: post #3

ChatGPT can't access the external internet. The example ChatGPT instruction where they "teach it" just has the user giving it a URL to their docs and ChatGPT is like "oh ok got it" (Figure 3). I'm not sure if the blog post writers are aware it's hallucinating what the JSON implementation is and hasn't actually read the documentation at all.

There's this internet phenomenon (common on Reddit and other such forums) where you find high-quality discussions on a variety of topics and everyone seems so clever and informative, until you come across a thread on a topic that you are actually an expert on and find that every top-voted comment is objectively incorrect. ChatGPT is essentially that but on steroids. Its primary ability is being able to sound extremel…

Sounds like internet version of Gell-Mann effect

Re: ChatGPT is now finding bugs in databases

#178

Post by simonw »

simonw wrote: post #174
honeybadger1 wrote: post #172

Earlier quoted context omitted.

Interesting, and certainly convincing. I have a paid account, I wonder if this makes any difference? It would make sense to me that it is basing it's 'reality' off of the prevous 21h1 page(which would have existed) and it understands the goal I am trying to achieve.

I have a paid account too. I'm very confident that it doesn't fetch URLs, despite doing an extremely convincing impression of being able to do so. Try changing a few characters in that Microsoft URL and see if you still get the same results.

Wrote more about that here: https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces...

Re: ChatGPT is now finding bugs in databases

#180

Post by harperlee »

Felminor wrote: post #140
danielvaughn wrote: post #56

Earlier quoted context omitted.

I think a lot of overly-eager and under-informed people are jumping on the bandwagon and shouting half-baked proclamations from the rooftops to get clicks and views. But underneath all the bullshit is something truly useful, so I wouldn't necessarily say that Confidently Incorrect™ is ChatGPT's primary ability. Coincidentally, over the past few weeks of this ChatGPT craze, I've needed to create a lot of fake data to…

Just imagine how many little code snippets you used (at least I did) to make my life easier and faster A script which parses jaon for a specific key or transforming a table from y to z. Now everyone has this available to them. It's great already!

But those are the low hanging fruit that help me avoid rusting skills… and this is a common phenomenon on automation: you destroy the first rungs on the experience ladder, that no one wants to do but are essential to develop foundational skills.
Post reply on HN