Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI isn’t good enough

skventures.substack.com

Post reply on HN
Print view View original

351–360 of 374 posts

Re: AI isn’t good enough

#351

Post by pixl97 »

pdimitar wrote: post #245
pixl97 wrote: post #240

Earlier quoted context omitted.

I mean, if we're taking the random walk that nature took to get human level intelligence from the first multicellular organisms you'll have to wait 600 million years or so. AI takes a shortcut around having a timeline of a trillion different versions history optimizing the learning algorithm and putting the successful ones in a timeline, but at a cost of trillions of operations to teach it. TL;DR, the problem space i…

You are extremely generously assuming that AI evolution has a direction and a purpose/goal like the organic one has. It doesn't. We are steering it. And I doubt that we know better than the natural evolution.

Natural evolution has no direction itself, huge amounts of it are negative. It just happens those with positive traits get to hang around and breed some more.

Re: AI isn’t good enough

#352

Post by squeaky-clean »

Aperocky wrote: post #89
squeaky-clean wrote: post #87

Earlier quoted context omitted.

> LLMs won't get intelligent. I think this sentence doesn't mean much unless we have a strict definition of what intelligence means. Just today ChatGPT helped me solve a DNS issue that I would not have been able to solve on my own in one day, let alone an hour. I'd consider it already more intelligent than myself when it comes to DNS.

It's seen more DNS content than you and anybody else have seen in their entire lives, and are able to regurgitate what it read because it has far faster memory access than you did. A dictionary contain knowledge but no intelligence.

Find me a dictionary that will turn to the correct page when I say or type "what's a word that means (thing)" and yes, I'd consider it intelligent.

But still, you don't have to agree with me about what intelligence means. But it is important in these discussions to understand that not every participant shares the same definition of the term intelligence.

Re: AI isn’t good enough

#353

Post by rchaud »

csydas wrote: post #94
petermcneeley wrote: post #3

"too few people for all the jobs, for the most part" - Are wages going up? "But almost everywhere else needs people. Badly. Across retail, restaurants, manufacturing, trades, and on and on, companies are struggling to hire. " - Wow sounds like an exciting star trek future that I cant wait to be part of! "labor became more expensive than capital" - So wages must be going up right? "In essence, the authors show that fo…

I'm a bit disappointed this part of the conversation is buried under everyone arguing about whether the article is right about AI or not, since the real interesting part of the article is why the author(s) seem to think that the reason McDonalds is doing signing bonuses is because people are just just not working, and the article never seems to really understand why people aren't flocking to jobs colloquially known t…

> it really seems like AI is better suited to displace management and executives, not the people who perform specialized and physical work

Management and executives are the ones holding the purse strings for these expensive solutions. Why would they implement something that could replace them?

Re: AI isn’t good enough

#354

Post by Aperocky »

squeaky-clean wrote: post #352
Aperocky wrote: post #89

Earlier quoted context omitted.

It's seen more DNS content than you and anybody else have seen in their entire lives, and are able to regurgitate what it read because it has far faster memory access than you did. A dictionary contain knowledge but no intelligence.

Find me a dictionary that will turn to the correct page when I say or type "what's a word that means (thing)" and yes, I'd consider it intelligent. But still, you don't have to agree with me about what intelligence means. But it is important in these discussions to understand that not every participant shares the same definition of the term intelligence.

That's just more advanced indexing.

intelligence to me has to be based off initiative. In that sense, a dog or a cat have more intelligence than GPT.

I actually very much look forward silicon (or other non-biotic material) attaining intelligence, I consider that the only way that Earth civilization can colonize space. But this aint it.

Re: AI isn’t good enough

#355

Post by ambrozk »

thrwjud wrote: post #347
ambrozk wrote: post #236

Earlier quoted context omitted.

Yes, but how do they know where we are in that growth phase? If you confidently tell me that "GPT-4 is the plateau," I want to know how you know that, specifically. "Well, because all deep learning technologies eventually slow down" is not a good argument. You need to show me the diminishing returns, or give me a good theoretical argument for why we're reaching them.

The more compute you need to get state of the art performance the closer to the plateau you are. If you didn't need the compute researchers would be getting better results with smarter training. Given that the gpt family of models need more energy to train than Nevada needs to keep the lights on they are very much on the flat part of the logistic growth curve.

This isn't true. In fact, it's the reverse of true. If you think a bit more carefully about your argument, you'll realize that you've asserted that the single most revolutionary advance of modern deep networks (i.e. network architectures whose performance scales neatly with their parameter counts & training epochs) automatically portends "the plateau of forward progress."

Re: AI isn’t good enough

#356

Post by ambrozk »

coldtea wrote: post #68
dontupvoteme wrote: post #57

Earlier quoted context omitted.

It hasn't even begun to get good, we only got yesterday decent local code generation models, we haven't even begun on the fine tuning and tooling for using them.

When the "fine tuning" and focus on tooling begins, it usually means the "free" massive improvements have tapered off.

This is totally wrong. Fine-tuning techniques were a prerequisite for turning academically impressive but boring and useless models into useful, compelling user products like ChatGPT. Fine-tuning complements fundamental model improvements -- it doesn't substitute for them.

Re: AI isn’t good enough

#357

Post by jongjong »

maxlamb wrote: post #254
jongjong wrote: post #7

As impressive as AI is and what it can do. I've felt the same way in my last job. It was almost there; it was almost good enough such that we could almost ship the project but we didn't. GPT3.5 was not good enough, GPT4 seemed like it was good enough in terms of accuracy but it was too slow and we were getting throttled like crazy. I lost my job over it. When I was worried about AI taking my job, I didn't picture it…

What do you mean you lost your job over it? All because GPT4 was too slow for your project?

Yes. This is the reality in many startups. I should probably have refused to work on this project and suggested some other task; that's what my colleague did; he only worked on the front end part. I took a risk, I misjudged the tool and failed.

I wasn't even upset (I got a decent severance), I'd been let go over far more trivial things when I was a junior years ago... Like when the boss fired me the first monday after he met my girlfriend at the office christmas party because he thought she was out of my league. That's why I don't bring her (now wife) to any office events nowadays.

Re: AI isn’t good enough

#358

Post by jongjong »

tacticalDonut wrote: post #224
jongjong wrote: post #9

Earlier quoted context omitted.

Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore s…

Why didn't you go the retrieval augmented generation route with a vector store?

That was going to be the next stage, but this would have been a more complicated solution with additional challenges but definitely a lot of code could have been re-used as it would just have been a matter of substituting the database lookup portion.

There are unknowns with the vector store solution because it doesn't suffice to just fetch a few relevant sentences in arbitrary order; we had to fetch every relevant piece of information in appropriately-sized chunks (some of which had to be multiple lines, some of which required the section heading for context) to formulate a correct answer. Sometimes there was something mentioned in a different section of the text which changed the outcome. Going down the vector database route would have taken longer and involved additional learning and it's not clear that we could have reduced the input size by doing that. I still think it was a good decision to start with a regular database first given that all information mapped neatly under the headings in our table of contents and that each section was relatively short. All sections were less than 500 words but most were only about 100 to 200 words.

Re: AI isn’t good enough

#359

Post by jongjong »

BoorishBears wrote: post #24
jongjong wrote: post #9

Earlier quoted context omitted.

Our use case was that we needed to consider information from thousands of pages of documents to answer customer questions. I let GPT decide where to look based on content tables of the documents (thankfully each section was not more than ~500 words) and only fetched information from the relevant sections to let GPT answer the user's question. GPT3.5 was fast enough to be usable but often inaccurate; it would ignore s…

From what you said your solution wasn't designed correctly? You should definitely have been using BM25 and SBERT for something like this, you definitely should have been asking 3.5 for structured output and doing any math yourself. If these were answers from a fixed set of documents there's also a ton of pre-processing you should have been doing. - I recently helped a friend with a problem of identifying a certain fo…

>> you definitely should have been asking 3.5 for structured output and doing any math yourself.

I suspect that this solution would have required a lot more time given my lack of experience in these technologies and there are a lot of unknowns for that particular use case. For example, a user could ask to compare legalese from 30 different providers to find all those which met a certain conditions; even if you can narrow down the input to just a few sentences per provider, you still need to feed it 30 times that amount or else the AI has no way to compare all the providers.

With my solution, GPT only saw a tiny highly relevant fraction of all the documents. It's possible that the solutions you mentioned would have helped cut our input size further but it's not clear by how much given the complexity of the questions and the data.

Re: AI isn’t good enough

#360

Post by GolDDranks »

layer8 wrote: post #105
GolDDranks wrote: post #75

Earlier quoted context omitted.

I don't encounter a lot of "small", one-copy-paste-size problems in my daily work that I couldn't quickly solve myself, so I haven't found a lot of use for ChatGPT while coding, yet. (I reckon this is changing though.) However a few times there have been some mechanical refactoring-style grunt work I've delighted to have been able to let ChatGPT do. However, the rate ChatGPT is giving me subtly wrong results is just…

> Give it a year or two, maybe? Maybe? It’s not clear what would bring a qualitative improvement, barring massive amounts of new training data.

Based on the current front page of HackerNews, it's clear that quantitative improvement is happening all the time. I think that enough incremental progress will eventual feel like qualitative improvement. And it doesn't even need to be smarter models; improving prompts, IO systems etc. around the model might help. For example, I said I don't feel the need for help for copy-paste-sized problems, but inputing the whole repo seems to be achievable with some clever scaffolding with the IO around the model.
Post reply on HN