Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub cuts AI deals with Google, Anthropic

bloomberg.com

Post reply on HN
Print view View original

301–310 of 742 posts

Re: GitHub cuts AI deals with Google, Anthropic

#301

Post by bmitc »

thih9 wrote: post #221

I use cursor and its tab completion; while what it can do is mind blowing, in practice I’m not noticing a productivity boost. I find that ai can help significantly with doing plumbing, but it has no problems with connecting the pipes wrong. I need to double and triple check the updated code - or fix the resulting errors when I don’t do that. So: boilerplate and outer app layers, yes; architecture and core libraries,…

That's my exact experience with GitHub Copilot. Even boilerplate stuff it sucks at as well. I have no idea why its autocomplete is so bad when it has access to my code, the function signatures, types, etc. It gets stuff wrong all the time. For example, it will just flat out suggest functions that don't exist, neither in the Python core libraries or in my own modules. It doesn't make sense.

I have all but given up on using Copilot for code development. I still do use it for autocomplete and boilerplate stuff, but I still have to review that. So there's still quite a bit of overhead, as it introduces subtle errors, especially in languages like Python. Beyond that, it's failure rate at producing running, correct code is basically 100%.

Re: GitHub cuts AI deals with Google, Anthropic

#302

Post by miki123211 »

thih9 wrote: post #221

I use cursor and its tab completion; while what it can do is mind blowing, in practice I’m not noticing a productivity boost. I find that ai can help significantly with doing plumbing, but it has no problems with connecting the pipes wrong. I need to double and triple check the updated code - or fix the resulting errors when I don’t do that. So: boilerplate and outer app layers, yes; architecture and core libraries,…

For now, I mostly use AI as a "faster typist".

If it wants to complete what I wanted to type anyway, or something extremely similar, I just press tab, otherwise I type my own code.

I'd say about 70% of individual lines are obvious enough if you have the surrounding context that this works pretty well in practice. This number is somewhat lower in normal code and higher in unit tests.

Another use case is writing one-off scripts that aren't connected to any codebase in particular. If you're doing a lot of work with data, this comes in very handy.

Something like "here's the header of a CSV file", pass each row through model x, only pass these three fields, the model will give you annotations, put these back in the csv and save, show progress, save every n rows in case of crashes, when the output file exists, skip already processed rows."

I'm not (yet) convinced by AI writing entire features, I tried that a few times and it was very inconsistent with the surrounding codebase. Managing which parts of the codebase to put in its context is definitely an art though.

It's worth keeping in mind that this is the worst AI we'll ever have, so this will probably get better soon.

Re: GitHub cuts AI deals with Google, Anthropic

#303

Post by imiric »

dartos wrote: post #241
imiric wrote: post #125

Earlier quoted context omitted.

I recently tried to ask these tools for help with using a popular library, and both GPT-4o and Claude 3.5 Sonnet gave highly misleading and unusable suggestions. They consistently hallucinated APIs that didn't exist, and would repeat the same wrong answers, ignoring my previous instructions. I spent upwards of 30 minutes repeating "now I get this error" to try to coax them in the right direction, but always ending up…

FWIW I almost never ask it to write code for me. I did once to write a matplotlib script and it gave me a similar headache. I ask it questions mostly about libraries I’m using (usually that have poor documentation) and how to integrate it with other libraries. I found out about Yjs by asking about different operational transform patterns. Got some context on the prosemirror plugin by pasting the entire provider class…

To be clear, I didn't ask it to write something complex. The prompt was "how do I do X with library Y?", with a bit more detail. The library is fairly popular and in a mainstream language.

I had a suspicion that what I was trying to do was simply not possible with that library, but since LLMs are incapable of saying "that's not possible" or "I don't know", they will rephrase your prompt and hallucinate whatever might plausibly make sense. They have no way to gauge whether what they're outputting is actually correct.

So I can imagine that you sometimes might get something useful from this, but if you want a specific answer about something, you will always have to double-check their work. In the specific case of programming, this could be improved with a simple engineering task: integrate the output with a real programming environment, and evaluate the result of actually running the code. I think there are coding assistant services that do this already, but frankly, I was expecting more from simple chat services.

Re: GitHub cuts AI deals with Google, Anthropic

#304

Post by digging »

sdesol wrote: post #254
wavemode wrote: post #185

Earlier quoted context omitted.

You're conflating three different things. There's the question, "is an LLM just autocomplete"? The answer to that question is obviously no, but the question is also a strawman - people who actually use LLM's regularly do recognize that there is more to their capabilities than randomized pattern matching. Separately, there's the question of "will LLM's become AGI and/or become super intelligent." Most people recognize…

> Most people recognize that LLM's are not currently super intelligent, I think calling it intelligent is being extremely generous. Take a look at the following example which is a spelling and grammar checker that I wrote: https://app.gitsense.com/?doc=f7419bfb27c89&temperature=0.50... When the temperature is 0.5, both Claude 3.5 and GPT-4o can't properly recognize that GitHub is capitalized. You can see the response…

> I think calling it intelligent is being extremely generous ... can't properly recognize that GitHub is capitalized.

Wouldn't this make chimpanzees and ravens and dolphins unintelligent too? You're asking it to do a task that's (mostly) easy for humans. It's not a human though. It's an alien intelligence which "thinks" in our language, but not in the same way we do.

If they could, specialized AI might think we're unintelligent based on how often we fail, even with advanced tools, pattern matching tasks that are trivial for them. Would you say they're right to feel that way?

Re: GitHub cuts AI deals with Google, Anthropic

#305

Post by pnathan »

thih9 wrote: post #221

I use cursor and its tab completion; while what it can do is mind blowing, in practice I’m not noticing a productivity boost. I find that ai can help significantly with doing plumbing, but it has no problems with connecting the pipes wrong. I need to double and triple check the updated code - or fix the resulting errors when I don’t do that. So: boilerplate and outer app layers, yes; architecture and core libraries,…

In general I do not find AI a net positive. Other tools seem to do at least as well in general.

it can be used if you want the reliability of a random forum poster. which... sure. knock yourself out. sometimes there's gems in that dirt.

I'm getting _very_ bearish on using LLMs for things that aren't pattern recognition.

Re: GitHub cuts AI deals with Google, Anthropic

#306

Post by tensor »

bob1029 wrote: post #246
thih9 wrote: post #221

I use cursor and its tab completion; while what it can do is mind blowing, in practice I’m not noticing a productivity boost. I find that ai can help significantly with doing plumbing, but it has no problems with connecting the pipes wrong. I need to double and triple check the updated code - or fix the resulting errors when I don’t do that. So: boilerplate and outer app layers, yes; architecture and core libraries,…

It's the subtle errors that are really difficult to navigate. I got burned for about 40 hours on a conditional being backward in the middle of an otherwise flawless method. The apparent speed up is mostly a deception. It definitely helps with rough outlines and approaches. But, the faster you go, the less you will notice the fine details, and the more assumptions you will accumulate before realizing the fundamental e…

Why aren't you writing unit tests just because AI wrote the function? Unit tests should be written regardless of the skill of the developer. Ironically, unit tests are also one area where AI really does help move faster.

High level design, rough outlines and approaches, is the worst place to use AI. The other place AI is pretty good is surfacing api call or function calls you might not know about if you're new to the language. Basically, it can save you a lot of time by avoiding the need for tons of internet searching in some cases.

Re: GitHub cuts AI deals with Google, Anthropic

#307

Post by imiric »

geodel wrote: post #178
imiric wrote: post #125

Earlier quoted context omitted.

I recently tried to ask these tools for help with using a popular library, and both GPT-4o and Claude 3.5 Sonnet gave highly misleading and unusable suggestions. They consistently hallucinated APIs that didn't exist, and would repeat the same wrong answers, ignoring my previous instructions. I spent upwards of 30 minutes repeating "now I get this error" to try to coax them in the right direction, but always ending up…

Well it is volume business. <1% of advanced skill developers will find AI helper useless but for 99% of IT CRUD peddlers these tools are quite sufficient. All in all if employers cut down 15-20% of net development costs by reducing head counts, it will be very worthwhile for companies.

Sure, but my specific question was fairly trivial, using a mainstream language and a popular library. Most of my work qualifies as CRUD peddling. And yet these tools are still wasting my time.

Maybe I'll have better luck next time, or maybe I need to improve my prompting skills, or use a different model, etc. I was just expecting more from state of the art LLMs in 2024.

Re: GitHub cuts AI deals with Google, Anthropic

#308

Post by tensor »

oogetyboogety wrote: post #279
tanseydavid wrote: post #271

Earlier quoted context omitted.

>> The apparent speed up is mostly a deception. When I am able ask a very simple question of an LLM which then prevents me having to context-switch to answer the same simple question myself; this is a big time saver for me but hard-to-quantify. Anything that reduces my cognitive load when the pressure is on is a blessing on some level.

This might be the measurable "some" non deceptive time saving, whereas most of it is still deceptive in terms of time saved

Except actual studies objectively show efficiency gains, more with junior devs, which make sense. So no, it's not a "deception" but it is often overstated in popular media.

Re: GitHub cuts AI deals with Google, Anthropic

#310

Post by greenavocado »

greenavocado wrote: post #53

I replaced ChatGPT Plus with hosted nvidia/Llama-3.1-Nemotron-70B-Instruct for coding tasks. Nemotron produces good code. The cost different is massive. Nemotron is available for $0.35 per Mtoken in and out. ChatGPT is considerably more expensive.

Just kidding. Qwen 2.5 Instruct is superior. Nemotron is overfit to pass benchmarks.
Post reply on HN