Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I trusted an LLM, now I'm on day 4 of an afternoon project

nemo.foo

Post reply on HN
Print view View original

171–180 of 215 posts

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#171

Post by maybesomaybenot »

BalinKing wrote: post #126
maybesomaybenot wrote: post #82

Earlier quoted context omitted.

Claude is like having my own college professor. I've learned more in the past month with Claude then I learned in the past year. I can ask questions repeatedly and get clarification as fine as a need it. Granted, Claude has limits, but its a game-changer. > I think the key to being successful here is to realize that you're still at the wheel as an engineer. The llm is there to rapidly synthesize the universe of infor…

> Claude is like having my own college professor. I don't use Claude, so maybe there's a huge gap in reliability between it and ChatGPT 4o. But with that disclaimer out of the way, I'm always fairly confused when people report experiences like these—IME, LLMs fall over miserably at even very simple pure math questions. Grammatical breakdowns of sentences (for a major language like Japanese) are also very hit-or-miss.…

Like I made very clear, it is great at some things and terrible at others.

YMMV. /shrugs/

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#172

Post by maybesomaybenot »

kortilla wrote: post #154
maybesomaybenot wrote: post #118

Earlier quoted context omitted.

Simple: one thing I'm learning about is RFCs for TCP/IP. I can literally go test it. It's like saying, "How do you know it is right when it says 2+2=4"? Some knowledge when taught is self-correcting. Other things I'm studying, like, say, tensor calculus, I can immediately use and know I learned it correctly.

TCP/IP is a great example though of something you can get seemingly correct and then be subject to all kinds of failure modes in edge cases you didn’t handle correctly (fragmentation, silly windows, options changing header sizes, etc).

Thanks for telling me how wrong I am! I bet you're fun at parties.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#173

Post by janderson215 »

biohcacker84 wrote: post #123
pfannkuchen wrote: post #58

Earlier quoted context omitted.

I’ve found LLMs to basically be a more fluent but also more lossy way of interfacing with stack overflow and tutorials. If a topic is well represented in those places, then you will get your answer quicker and it can be to some extent shaped to your use case. If the topic is not well represented there, then you will get circular nonsense. You can say “obviously, that’s the training data”, and that’s true, and I do fi…

That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.

Is it possible that you’re both using LLMs the same way you’d use SO and that’s the reason you see such similarities? The reason I ask is because it doesn’t not match my experience. It feels more like I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#174

Post by lostlogin »

jerf wrote: post #138
EarthMephit wrote: post #92

Earlier quoted context omitted.

I find that LLMs are almost comically bad at projects that have a hardware component like RaspberryPi or Pico, or Ardunio. I think that its because often the libraries you use are niche or have a a few similar versions, the LLM really commonly hallucinated solutions and would continually suggest that library X did have that capability. I think because often in hardware projects you often hit a point where you can't d…

Based on my own modestly successful forays into that world, I have to imagine one problem the LLMs have in that space is terrible training data. A good three quarters of any search result you search for in that space will be straight-up out-of-date and not work on your system. Then you've got all the tiny variations between dozens of chipsets, and all the confidently wrong people on the internet telling you to do non…

You’re making my eye twitch.

The number of projects I’ve done where my notes are the difference between hours of relearning The Way or instant success. Google doesn’t work as some niche issue is blocking the path.

ESP32, Arduino, Home Assistant And various media server things.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#175

Post by mercer »

KronisLV wrote: post #16

In my experience LLMs will help you with things that have been solved thousands of times before and are just a matter of finding some easily researched solution. The very moment when you try to go off the beaten path and do something unconventional or stuff that most people won't have written a lot about, it gets more tricky. Just consider how many people will know how to configure some middleware in a Node.js projec…

I've only started using LLMs for code recently, and I already tend to mentally translate what I want to something that I imagine is 'more commonly done and well represented in the training data'.

But especially the ability to just see some of the stuff it produces, and now to see its thought process, is incredibly useful to me already. I do have autism and possibly ADD though.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#176

Post by h0l0cube »

janderson215 wrote: post #173
biohcacker84 wrote: post #123

Earlier quoted context omitted.

That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.

Is it possible that you’re both using LLMs the same way you’d use SO and that’s the reason you see such similarities? The reason I ask is because it doesn’t not match my experience. It feels more like I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.

> I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.

So you're using it like Wikipedia? I find when learning something new (and non coding related) YouTube is infinitely better than an LLM. But then I prefer visual demonstration to tutorial or verbal explanation.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#177

Post by maybesomaybenot »

BalinKing wrote: post #126
maybesomaybenot wrote: post #82

Earlier quoted context omitted.

Claude is like having my own college professor. I've learned more in the past month with Claude then I learned in the past year. I can ask questions repeatedly and get clarification as fine as a need it. Granted, Claude has limits, but its a game-changer. > I think the key to being successful here is to realize that you're still at the wheel as an engineer. The llm is there to rapidly synthesize the universe of infor…

> Claude is like having my own college professor. I don't use Claude, so maybe there's a huge gap in reliability between it and ChatGPT 4o. But with that disclaimer out of the way, I'm always fairly confused when people report experiences like these—IME, LLMs fall over miserably at even very simple pure math questions. Grammatical breakdowns of sentences (for a major language like Japanese) are also very hit-or-miss.…

I think a lot of these people object to AI probably see the gross amounts of energy it is using, or the trillions of dollars going to fewer than half a dozen men (most american, mostly white).

But, once you've had AI help you solve some gnarly problems, it is hard not to be amazed.

And this is coming from a gal who thinks the idea of self-driving cars is the biggest waste of resources ever.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#178

Post by masklinn »

tashian wrote: post #114

I used Claude to help me build a side project in 4 hours that I would never have built otherwise. Essentially, it's a morphing wavetable oscillator in React ( https://waves.tashian.com ). Six months ago, I tried building this app with ChatGPT and got nowhere fast. Building it with Claude required a gluing together a few things that I didn't know much about: JavaScript audio processing, drawing on a JavaScript canvas,…

What was your workflow for doing that? Just going back and forth in a chat, or a more integrated experience in a dedicated editor?

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#179

Post by pfannkuchen »

janderson215 wrote: post #173
biohcacker84 wrote: post #123

Earlier quoted context omitted.

That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.

Is it possible that you’re both using LLMs the same way you’d use SO and that’s the reason you see such similarities? The reason I ask is because it doesn’t not match my experience. It feels more like I’m able to Matrix-upload docs into my brain like Trinity learning to fly a helicopter.

I am using it like stack overflow in the sense that I’m solving a problem and I’m using it to answer questions when I’m in an unfamiliar or non-obvious place in the problem space.

If I have a question about a first order language or framework feature or pattern, it works great. If I have a question about a second order problem, like an interaction between language or framework features, or a logical inconsistency in feature behavior, then it usually has no idea what’s going on, unless it turns out to be a really common problem such as something that would come up when working through a tutorial.

For code completion, I’ve just turned it off. It saves time on boilerplate typing for sure, but the actual content pieces are so consistently wrong that on balance I find it distracting.

Maybe I have a weird programming style that doesn’t mesh well with the broader code training corpus, not sure. Or maybe a lot of people spend more time in the part of problem-space that intersects with tutorial-space? I am not very junior these days.

That being said I definitely do use LLMs to engage with tutorial type content. For that it is useful. And outside of software it is quite a bit better for interfacing with Wikipedia type content. Except for the part where it lies to your face. But it will get better! Extrapolating never hurt anyone.

Re: I trusted an LLM, now I'm on day 4 of an afternoon project

#180

Post by danielbln »

acedTrex wrote: post #60
jdietrich wrote: post #53

Earlier quoted context omitted.

It is quite remarkable that we are already at the stage where saying "this AI is about as competent as an inexperienced college graduate" constitutes criticism . It is entirely proper for people to be engaging sceptically with LLMs at their current level of capability, but I think we should also keep in mind the astonishingly rapid growth rate in their performance. LLMs were a toy two years ago, they're now a useful…

I mean 2 years ago they were at about the same place, theres been very little practical gain from gpt4 in my opinion. No matter the model the fundamental failure cases have remained the same.

I disagree, context size alone has exploded from 8k to 200k now and that makes a huge difference. LLMs have also progressed significantly in many other metrics, code quality, understanding, etc. The recent reasoning models have upped the ante further, especially when combined with models that are good at editing code.
Post reply on HN