Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI isn’t good enough

skventures.substack.com

Post reply on HN
Print view View original

61–70 of 374 posts

Re: AI isn’t good enough

#61

Post by dontupvoteme »

cbozeman wrote: post #32
seanthemon wrote: post #28

Earlier quoted context omitted.

Curious, how does it do when you give it examples / use playground.openai? I find that no examples often leads to the same result as a befuddled junior, but with examples often it gains confidence. Also I find playground to give me much better code snippets than chat sometimes

You've identified the problem inherent in these models right now, though. There's a lot of "extraneous" information and details that appear useless and unrelated, but that long-time developers have tucked away in their brains (or really anyone who has done something at a high level for a long time), that turns out to be incredibly useful; generally these people don't require examples - they just know what the right a…

That's where good money and societal upheaval will come up, private models for corporations trained on their entire codebase and documentation.

Both young and old shall feel the pain then.

Re: AI isn’t good enough

#62

Post by dontupvoteme »

I am completely bewildered by the responses here.

GPT-3 is so good it could write code, or it could emulate an entire TV show to the point of lawsuit, or simulate an eternal debate between Herzog and Zizek. You could translate Bulgarian to Sindarin for heaven's sake. And all of this comes at a 50% tax rate or so because of the alignment tax that OAI opted for. (which doesn't even work because anyone who puts effort in can jailbreak it scarily well)

The real solution is not these monstrosities of the internet mushed together and taxed like a Belgian billionaire. It's models and frameworks specific to what you want in a given query.

It was less than 24 hours ago that we finally got a local model that can do code generation well, and we also know that Meta has a far better one that they are holding back.

There is no reason that for any query you should be restricted to a single inference run on a single fixed model, and there is no reason that we shouldn't perform a bunch of non-LLM processing on the output before the user sees it.

Switch between multiple models, fine-tune, or use LoRAs based on the query -- Python plots? Load llama-coding-python-plots. You like Plotly? Add -Plotly. Run the code in a sandbox (with hard kills on resources) and regenerate it if it does not meet standards.

You're working in linguistics? Switch to a chat model that's fine-tuned in the literature and codebase of that field.

There are a trillion ways to improve things; we're basically at the cavemen-banging-rocks-together point.

Hell, I'm a sleep deprived ESL and i just invoked the monstrosity to fix my grammar and spelling for this post. What a bloody waste of CO2.

If for nothing but to keep the Earth afloat we shouldn't be using these huge closed source SaaS things unless we need them. Repurpose them for SETI@Home or processing human genetics for healthcare, or something else that benefits mankind

Re: AI isn’t good enough

#63

Post by input_sh »

andrepd wrote: post #39
jasfi wrote: post #37

Earlier quoted context omitted.

Are you using GPT-4? If not, it's understandable. If you don't pay for ChatGPT, you get GPT-3.5. You can also get access to GPT-4 if you use the playground.

Why is it always the same reply? Yes, GPT4 is as useless as GPT3.5 on any non trivial task.

My guess is people feel the need to self-justify their $20/month subscription.

Re: AI isn’t good enough

#64

Post by dontupvoteme »

tuatoru wrote: post #15
borissk wrote: post #11

Very interesting article. IMHO a key factor in how the world changes in the next 10-20 years will be how far the current wave of AI, based on neural networks can go. If scientists and engineers manage to implement autonomous cars and robots that can do basic human tasks like clean, wait and take care of the sick and elderly we will wake up in a brand new world. It will put a lot of stress on the basics of society, su…

The real world has many trillions of parameters. Also, doing stuff in the real world is all about sensors. Progress on integrating millions of sensors into a robot has been slow.

>The real world has many trillions of parameters.

But at any given time you're probably only going to require some millions or double digit billions, unless you're a nation state or something like that.

Re: AI isn’t good enough

#65

Post by xmprt »

dontupvoteme wrote: post #49
intended wrote: post #35

Earlier quoted context omitted.

I can take a bet that it haha already failed - the hype cycle has already made a promise that LLMs can’t keep. Hallucinations to the normal person are a bug. The issue is that only humans can hallucinate. We know there is a “reality”. For an LLM, everything it does is a hallucination. That’s why you have more POCs than production goods. Your “hallucination rate” is unknown. Yesterday Ars has an article that described…

Why not "simply" multigen every (important) query and take the statistical average? Hallucinations are random, the truth isn't. This is absurdly expensive with GPT4, cheaper with 3, and dirt cheap locally with LLaMA

That only works if the generated outputs are completely independent and not correlated. I'd be interested in research that shows whether multigen actually reduces hallucination rates.

Re: AI isn’t good enough

#66

Post by coldtea »

dontupvoteme wrote: post #58
coldtea wrote: post #43

Earlier quoted context omitted.

I'm using GTP-4 and it makes trivial errors all the time. I asked it (actual names changed): "I run the Linux command line program "foo". When I use the flags -xyz, I get results, but when I use -txyz I get nothing. What could this mean?" And it told me: "The lack of results is because you didn't use the -t flag". Or I ask it some very basic music theory questions and it gets stuff wrong all the time, giving impossib…

They definitely nerfed the hell out of GPT4 via the webUI at least. Do you have API access? the old model there still gives me very good results.

I also have API access, but this was from the web

Re: AI isn’t good enough

#67

Post by usrbinbash »

ambrozk wrote: post #18

This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one stat…

> This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease.

Which is countered by...the assertion that it won't?

LLMs won't get intelligent. That's a fact based on their MO. They are sequence completion engines. They can be fine tuned to specific tasks, but at their core, they remain stochastic parrots.

> I want to know only one thing, which is what gives him the confidence necessary to say that.

I want to know only one thing, what gives the confidence to say otherwise?

Re: AI isn’t good enough

#68

Post by coldtea »

dontupvoteme wrote: post #57
coldtea wrote: post #38

Earlier quoted context omitted.

> This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Sounds like a pretty good guesstimation. Well, not cease, just fizzle out.

It hasn't even begun to get good, we only got yesterday decent local code generation models, we haven't even begun on the fine tuning and tooling for using them.

When the "fine tuning" and focus on tooling begins, it usually means the "free" massive improvements have tapered off.

Re: AI isn’t good enough

#69

Post by Kiro »

andrepd wrote: post #39
jasfi wrote: post #37

Earlier quoted context omitted.

Are you using GPT-4? If not, it's understandable. If you don't pay for ChatGPT, you get GPT-3.5. You can also get access to GPT-4 if you use the playground.

Why is it always the same reply? Yes, GPT4 is as useless as GPT3.5 on any non trivial task.

> Why is it always the same reply?

When there's one definitive answer to something that people keep repeating there's a slight chance that it's actually true. Shocking, I know.

Re: AI isn’t good enough

#70

Post by dustymcp »

coldtea wrote: post #43
jasfi wrote: post #37

Earlier quoted context omitted.

Are you using GPT-4? If not, it's understandable. If you don't pay for ChatGPT, you get GPT-3.5. You can also get access to GPT-4 if you use the playground.

I'm using GTP-4 and it makes trivial errors all the time. I asked it (actual names changed): "I run the Linux command line program "foo". When I use the flags -xyz, I get results, but when I use -txyz I get nothing. What could this mean?" And it told me: "The lack of results is because you didn't use the -t flag". Or I ask it some very basic music theory questions and it gets stuff wrong all the time, giving impossib…

it really does, it came up with multiple wrong dockerfiles for me yesterday, but it seems to correct it when you tell it
Post reply on HN