Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub cuts AI deals with Google, Anthropic

bloomberg.com

Post reply on HN
Print view View original

411–420 of 742 posts

Re: GitHub cuts AI deals with Google, Anthropic

#411

Post by Aeolun »

0xFACEFEED wrote: post #354
bongodongobob wrote: post #339

Earlier quoted context omitted.

You. Can. Write. Tests.

How do tests account for cases where I'm looking at a 100 line function that could have easily been written in 20 lines with just as much, if not more, clarity? It reminds me of a time (long ago) when the trend/fad was building applications visually. You would drag and drop UI elements and define logic using GUIs. Behind the scenes the IDE would generate code that linked everything together. One of the selling points…

> How do tests account for cases where I'm looking at a 100 line function that could have easily been written in 20 lines with just as much, if not more, clarity?

That’s not a failure of the AI writing that 100 line monstrosity, it’s a failure of you deciding to actually use the thing.

If you know what 20 lines are necessary and the AI doesn’t output that, why would you use it?

Re: GitHub cuts AI deals with Google, Anthropic

#412

Post by girvo »

mithametacs wrote: post #379
0xFACEFEED wrote: post #329

Earlier quoted context omitted.

As a programmer of over 20 years - this is terrifying. I'm willing to accept that I just have "get off my lawn" syndrome or something. But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interactin…

You still use type systems, tests, and code review. For a lot of use cases it's powerful. If you ask it to build out a brand new system with a complex algorithm or to perform a more complex refactoring, it'll be more work correcting it than doing it yourself. But that malformed JSON document with the weird missing quotation marks (so the usual formatters break), and spaces before commas, and the indentation is wild..…

The (mostly useless boilerplate “I’m basically just testing my mocks”) tests are being written by AI too these days.

Which is mildly annoying as a lot of those tests are basically just noise rather than useful tools. Humans have the same problem, but current models are especially prone to it from what I’ve observed

And not enough devs are babysitting the AI to make sure the test cases are useful, even if they’re doing so for the original code it produced

Re: GitHub cuts AI deals with Google, Anthropic

#413

Post by chairhairair »

tensor wrote: post #382
chairhairair wrote: post #318

Earlier quoted context omitted.

I have completely the opposite perspective. Unit tests actually need to be correct, down to individual characters. Same goes with API calls. The API needs to actually exist. Contrast that with "high level design, rough outlines". Those can be quite vague and hand-wavy. That's where these fuzzy LLMs shine. That said, these LLM-based systems are great at writing "change detection" unit tests that offer ~zero value (or…

The fact that you think "change detection" tests offer zero value speaks volumes. Those may well be the most important use of unit tests. Getting the function correct in the first place isn't that hard for a senior developer, which is often why it's tempting to skip unit tests. But then you go refactor something and oops you broke it without realizing it, some boring obvious edge case, or the like. These tests are al…

https://testing.googleblog.com/2015/01/testing-on-toilet-cha...

"speaks volumes" lol

Re: GitHub cuts AI deals with Google, Anthropic

#414

Post by photonthug »

geysersam wrote: post #365
0xFACEFEED wrote: post #329

Earlier quoted context omitted.

As a programmer of over 20 years - this is terrifying. I'm willing to accept that I just have "get off my lawn" syndrome or something. But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interactin…

I'll take a stab at changing your mind. AIs are not able to write Redis. That's not their job. AIs should not write complex high performance code that millions of users rely on. If the code does something valuable for a large number of people you can afford humans to write it. AIs should write low value code that just repeats what's been done before but with some variations. Generic parts of CRUD apps, some fraction…

> I'll take a stab at changing your mind.

Not the parent but this doesn’t seem mind changing, because what you describe is the normal/boring route to slightly better productivity using new tools without the breathless hype. And the 20% increase you mention of course depends a lot on what you’re doing, so for many types of work you’d be much closer to zero.

I’m curious about the claims of “power users” that are talking very excitedly about a brave new world. Are they fooling themselves, or trying to fool others, or working at jobs where 90% of their work is boilerplate drudgery, or what exactly? Inevitably it’s all of the above.. and some small percentage of real power users that could probably teach the rest of us cool stuff about their unique workflows. Not sure how to find the signal in all the noise though.

So personally, if I were to write “change my mind”, what I’d really mean is something like “convince me there are real power users already out there in the wild, using tools that are open to the public today”.

GP mentioned machine assisted translation of a huge code base being almost completely hands-off. If that were true and as easy as advertised then one might expect, for example, that it were trivial to just rewrite media wiki or Wordpress in rails or Django with a few people in a week. This is on the easier side of what I’d confidently label as a game-changingly huge productivity boost btw, and is a soft problem chosen because of the availability of existing code examples, mere translation over original work, etc. Not sure we’re there yet.

Re: GitHub cuts AI deals with Google, Anthropic

#415

Post by sdesol »

0xFACEFEED wrote: post #329
ianbutler wrote: post #311

Earlier quoted context omitted.

I'm actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it. My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. At the end of the day it can belt out lines of code faster than I…

As a programmer of over 20 years - this is terrifying. I'm willing to accept that I just have "get off my lawn" syndrome or something. But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interactin…

> But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible.

I do think it is kind of crazy based on what I've seen. I'm convinced LLM is a game changer but I couldn't believe how stupid it can be. Take the following example, which is a spelling and grammar checker that I wrote:

https://app.gitsense.com/?doc=f7419bfb27c8968bae&samples=5

If you click on the sentence, you can see that Claude-3.5 and GPT-4o cannot tell that GitHub is spelled correctly most of the time. It was this example that made me realize how dangerous LLM can be. The sentence is short but Claude-3.5 and GPT-4o just can't process it properly.

Having a LLM rewrite large swaths of code is crazy but I believe with proper tooling to verify and challenge changes, we can mitigate the risk.

I'm just speculating, but I believe GitHub has come to the same conclusion that I have, which is, all models can be stupid, but it is unlikely that all will be stupid at the same time.

Re: GitHub cuts AI deals with Google, Anthropic

#416

Post by Nevermark »

bongodongobob wrote: post #338
tomrod wrote: post #297

Earlier quoted context omitted.

How long are you willing to iterate to get things right?

If it takes almost no cognitive energy, quite a while. Even if it's a little slower than what I can do, I don't care because I didn't have to focus deeply on it and have plenty of energy left to keep on pushing.

As my mother used to say, "I love work. I could watch it all day!"

I can see where you are coming from.

Maintaining a better creative + technical balance, instead of see-sawing. More continuous conscious planning, less drilling.

Plus the unwavering tireless help of these AI's seems psychologically conducive to maintaining one's own motivation. Even if I end up designing an elaborate garden estate or a simpler better six-axis camera stabilizer/tracker, or refactoring how I think of primes before attempting a theorem, ... when that was not my agenda for the day. Or any day.

Re: GitHub cuts AI deals with Google, Anthropic

#417

Post by mreid »

balder1991 wrote: post #400
hansonkd wrote: post #367

Earlier quoted context omitted.

> The more time I spend in a codebase the better idea I have of what the writer was trying to do. This whole thing of using LLMs to Code reminds me a bit of when Google Translate came out and became popular, right around the time I started studying Russian. Yes, copying and pasting a block of Russian text produced a block of english text that you could get a general idea of what was happening. But translating from en…

> the intention of coding should never to be to belt out as many lines as possible That’s such an underrated statement. Especially when you consider the amount of code as a liability that you’ll have to take care later.

I've heard a similar sentiment: "It's not lines of code written, it's lines of code spent."

It also reminds me of this analogy for data, especially sensitive data: "it's not oil, it's nuclear waste."

Re: GitHub cuts AI deals with Google, Anthropic

#418

Post by skywhopper »

geysersam wrote: post #365
0xFACEFEED wrote: post #329

Earlier quoted context omitted.

As a programmer of over 20 years - this is terrifying. I'm willing to accept that I just have "get off my lawn" syndrome or something. But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interactin…

I'll take a stab at changing your mind. AIs are not able to write Redis. That's not their job. AIs should not write complex high performance code that millions of users rely on. If the code does something valuable for a large number of people you can afford humans to write it. AIs should write low value code that just repeats what's been done before but with some variations. Generic parts of CRUD apps, some fraction…

I have to disagree. If there’s that much boilerplate floating around then the tooling should be improved. Pasting over inefficiency with sloppier inefficiency is just a pure waste.

Re: GitHub cuts AI deals with Google, Anthropic

#419

Post by neodymiumphish »

CephalopodMD wrote: post #404

I usually feel like i can confidently express a change I want in code faster and better than I can explain what I want an AI to do in English. Like if I have a good prompt, these tools work okay, but getting that prompt almost as hard as just writing the code itself often. Do others feel the same struggle?

I’m positive my experience pales in comparison to yours, as I don’t actually code anything beyond the occasional single use script, but YES! I hate trying to explain the exact SQL result I’m looking for or some text modification I need to be able to throw together a CTE since I have read-only access and can’t even build a temp table.

Re: GitHub cuts AI deals with Google, Anthropic

#420

Post by dartos »

imiric wrote: post #303
dartos wrote: post #241

Earlier quoted context omitted.

FWIW I almost never ask it to write code for me. I did once to write a matplotlib script and it gave me a similar headache. I ask it questions mostly about libraries I’m using (usually that have poor documentation) and how to integrate it with other libraries. I found out about Yjs by asking about different operational transform patterns. Got some context on the prosemirror plugin by pasting the entire provider class…

To be clear, I didn't ask it to write something complex. The prompt was "how do I do X with library Y?", with a bit more detail. The library is fairly popular and in a mainstream language. I had a suspicion that what I was trying to do was simply not possible with that library, but since LLMs are incapable of saying "that's not possible" or "I don't know", they will rephrase your prompt and hallucinate whatever might…

> if you want a specific answer about something

Specific is the specific thing that statistical models are not good at :(

> how do I do X with library Y?

Recent research and anecdotal experience has shown that LLMs perform quite poorly with short prompts. Attention just has more data to work with when there are more tokens. Try extending that question like “I am using this programming language and am trying to do this task with this library. How do I do this thing with this other library”

I realize prompt engineering like this is fuzzy and “magic,” but short prompts have a consistent lower performance.

> In the specific case of programming, this could be improved with a simple engineering task: integrate the output with a real programming environment, and evaluate the result of actually running the code.

Not as simple as you’d think. You’re letting something run arbitrary code.

Tho you should give aider.chat a try if you want to test out that workflow. I found it very very slow.

Post reply on HN