Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Generative AI Con

wheresyoured.at

Post reply on HN
Print view View original

461–470 of 503 posts

Re: The Generative AI Con

#461

Post by famouswaffles »

azizam wrote: post #448
famouswaffles wrote: post #445

Earlier quoted context omitted.

How much value a user gets from a tool is the users prerogative to give. The user above is talking specifically about how much value users are getting out of LLMs. The number of users who consistently return is in fact a very good argument for the plain real world value being generated by LLMs.

Incorrect, I'm afraid. The "plain real world value" does not necessarily have any correlation to the number of users, so your argument again fails to hold water. Simply consider the users that use generative AI in order to perform some unimportant work (as I hope is the case, for generative AI cannot produce anything of novelty, by design). If such work held no value to begin with, then through simple deduction you c…

>Simply consider the users that use generative AI in order to perform some unimportant work

Says who ? I certainly don't use it for unimportant work.

>(as I hope is the case, for generative AI cannot produce anything of novelty, by design)

Another Unfounded Assertion

>then through simple deduction you can conclude that the generative AI contributed nothing of value.

Nonsensical. 'Unimportant work' that people keep doing is work that needs to be done. Getting it done is providing value.

>I could say a pornography website is of extreme value to humanity.

Pornography provides a lot of value yes.

Re: The Generative AI Con

#462

Post by Ancapistani »

cmdtab wrote: post #53
Ancapistani wrote: post #41

Earlier quoted context omitted.

41 here, working in healthtech… and Devin has committed more code and closed more tickets on my behalf in the past week at my behest than I’ve done on my own in a month. It’s basically functioning as a team of entry-level junior engineers at this point. Previously I was having to spend a fair amount of time writing tickets and providing context, but lately I’ve fed all my meeting transcripts and such into an LLM and…

What kind of tickets are these? Even at non-complex tasks, I find agents struggle a lot. Can you give some examples?

Sure. One task I gave it a couple of days ago was to upgrade the version of Python used in a project. In this case, that was a task suited for a junior engineer - it was simple enough to be described fully, but complex enough to require effort.

Devin was able to recognize that the project used Poetry, was Dockerized, and that the Python version was specific in multiple places (.python-version, pyproject.toml, Dockerfile). It saw that a couple of minor dependencies didn’t support the new version of Python, so it went back and upgraded those to the most recent matching version first.

Devin had never touched the repository in question before getting this task.

I’ve given it more and less complex tasks, and yeah, it struggles with some things. I’d estimate that it consumes about 5-10% of my time but multiples my overall output by ~3x.

Re: The Generative AI Con

#463

Post by Ancapistani »

camdenreslink wrote: post #215
Ancapistani wrote: post #41

Earlier quoted context omitted.

41 here, working in healthtech… and Devin has committed more code and closed more tickets on my behalf in the past week at my behest than I’ve done on my own in a month. It’s basically functioning as a team of entry-level junior engineers at this point. Previously I was having to spend a fair amount of time writing tickets and providing context, but lately I’ve fed all my meeting transcripts and such into an LLM and…

I would be very curious about the size and complexity of this codebase. Every review of Devin I’ve seen has been very negative (burns a ton of money, gets stuck, doesn’t implement the changes you want). For large codebases (greater than 15k or 20k LOC) the context size seems like a real problem right now.

I’ve used it for everything from “change this text on a webpage” to squashing complex migrations in multiple apps in a Django monolith where migrations in one app depends on migrations in other apps.

My apologies if anyone finds this offensive, but I sorta see Devin as a fresh junior SWE hire. It doesn’t do well with tasks that require deep knowledge sometimes, but it has shallow or better knowledge of everything. I would describe it as working with a brand new SWE with an IQ of about 85 who is also on the low end of being high-functioning autistic. By that I mean that it takes most things literally and sometimes has difficulty with nuance.

> burns a ton of money, gets stuck, doesn’t implement the changes you want

The first time you use it, I think that’s pretty fair. Every time it gets stuck or does the wrong thing, when you correct it, it gives you the option to add to its “knowledge base”. That’s a bunch of additional context that it applies in only certain situations. Within a week or so of using it regularly, it’s significantly more valuable. It “learns” much faster than a human.

Example:

About a dozen of our projects all rely on a shared repository (“Enki”) that contains a Composefile, configs, and some light automation. Tests are run in Docker, and you have to navigate to the other repo’s directory to bring up the service. Some of those projects have service names in the Composefile that differ from the project name. I was able to run the steps interactively on “Devin’s machine”, tell Devin what I had done, and then tell it that this is the correct approach for any project that depends on that repository. I didn’t tell it what projects those are, or how to find out.

The next time I used Devin on a project like that, it tried to run the tests directly in a local Python environment. That didn’t work, but it tried the correct approach next. That worked, so it added a line to its knowledge base “Project uses Enki.” From that point forward it did the right thing the first time.

> For large codebases (greater than 15k or 20k LOC) the context size seems like a real problem right now.

The primary project I’m working on is a Django app. I don’t have it in front of me right now, but it’s about five years old, has been under very active development the entire time, and is comprised of about twenty apps. It’s not the largest codebase I’ve worked on, but it’s far from the smallest. I can do a line count tomorrow if you’d like.

Re: The Generative AI Con

#464

Post by Ancapistani »

krainboltgreene wrote: post #119
Ancapistani wrote: post #41

Earlier quoted context omitted.

41 here, working in healthtech… and Devin has committed more code and closed more tickets on my behalf in the past week at my behest than I’ve done on my own in a month. It’s basically functioning as a team of entry-level junior engineers at this point. Previously I was having to spend a fair amount of time writing tickets and providing context, but lately I’ve fed all my meeting transcripts and such into an LLM and…

This terrifies me.

It excites me. The only way it would really terrify me is if I were a very junior engineer right now or in college to be one.

I think we’ll see a ton of complaints about how bad the job market is in the next couple of years. That will be true, but only for juniors or for seniors who don’t embrace the tech. For seniors who do embrace it and specialize in implementing these systems, it’ll be a gold mine.

Then, over 5-10 years, our seniors will start to retire or leave the field. No one will be there to replace them. At that point we’ll see a resurgence in the job market.

Things like autocompletion and “chat with your codebase” help juniors more than seniors; agents help seniors much more than juniors. As these systems improve, their failure cases get more and more complex/nuanced - you will always need senior people with the insight necessary to figure out what’s wrong when it breaks. For a while that will help seniors and hurt juniors… right up until businesses realize that they don’t have replacements for their existing senior engineers, at which point they’ll be desperate to hire again.

Re: The Generative AI Con

#465

Post by sambapa »

yunwal wrote: post #460
sambapa wrote: post #407

Earlier quoted context omitted.

But still, the questions in that test are "solved" in the sense of "I can take a dictionary and answers these questions with full certainty". Beyond established knowledge LLMs are monkeys with typewriters, at best.

I’d like to see you ace even a middle-school level Spanish test with just a dictionary (sub Spanish with some other language if you happen to know Spanish).

It was a figure of speech. But there is nothing superintelligent about acing Spanish tests. Give me a Riemann hypothesis.

Re: The Generative AI Con

#466

Post by azizam »

famouswaffles wrote: post #461
azizam wrote: post #448

Earlier quoted context omitted.

Incorrect, I'm afraid. The "plain real world value" does not necessarily have any correlation to the number of users, so your argument again fails to hold water. Simply consider the users that use generative AI in order to perform some unimportant work (as I hope is the case, for generative AI cannot produce anything of novelty, by design). If such work held no value to begin with, then through simple deduction you c…

>Simply consider the users that use generative AI in order to perform some unimportant work Says who ? I certainly don't use it for unimportant work. >(as I hope is the case, for generative AI cannot produce anything of novelty, by design) Another Unfounded Assertion >then through simple deduction you can conclude that the generative AI contributed nothing of value. Nonsensical. 'Unimportant work' that people keep do…

You have once again conflated individualistic values with real world values, and shown bias to personal anecdotes. I am afraid you are simply incapable of understanding so there is no point to discussing this further.

Re: The Generative AI Con

#467

Post by wizzz »

vonneumannstan wrote: post #412
jcgrillo wrote: post #338

Earlier quoted context omitted.

> top tier people Who? How? This is not what I've seen where I work. There's a bunch of hubbub and generalized excitement, and lots of talk about what could be done, or what might be done, but not very much actual doing . I must just be a clueless "mid".

Yeah unironically. Guido van Rossum - "I use it every day. My biggest adjustment with using Copilot was that instead of writing code, my posture shifted to reviewing code." https://www.youtube.com/watch?v=-DVyjdw4t9I Here's Jeff Dean saying 25% of the characters in new PRs at Google are AI Generated. https://www.dwarkeshpatel.com/p/jeff-dean-and-noam-shazeer Andrej Karpathy - "I basically can't imagine going back to…

I believe your examples are - unironically - misleading.

1- he states that the generated code is most likely wrong. He is appreciative of it though because he is a very poor typer so he doesn't have to do that part as much

2- so that's not supporting your argument that the 'top' devs are using it. Besides it doesn't say how it's counted, nor how much time is spent reviewing and correcting it

3- actually okay. But is he using it for production code? Doesn't say

4- he definitely doesn't talk about coding, only brainstorming and writing text.

5- your best one. Still, the use case here is side projects not production

You might still be right, I definitely do not compare myself to these people, but trying to glue some sources together makes a poor argument.

And the subject on hand is more that just using LLMs, it's the role of LLMs in the dev work environment

Re: The Generative AI Con

#468

Post by HDThoreaun »

pnathan wrote: post #188

Directionally correct. GenAI is - imo - an assistant. Copilot does effectively templating. I can have ChatGPT read an email and check it for tone. Claude can comment on camera kit. Claude does a very nice image recognition for obscure things. What I have become persuaded of is that the /completions API is simply not much more than +10% or a low key helper. I do not need a dumber-than-intern agent going ape on my code…

> I can have ChatGPT read an email and check it for tone.

Maybe Im crazy but this alone is a trillion dollar market cap industry imo. msft is worth 3 trillion off the back of similar products. If LLMs are seen as indispensable by every office worker in the country, as I think they are, and every employee has a subscription for $20 a month we're looking at many billions in revenue.

Re: The Generative AI Con

#469

Post by fragmede »

HDThoreaun wrote: post #468
pnathan wrote: post #188

Directionally correct. GenAI is - imo - an assistant. Copilot does effectively templating. I can have ChatGPT read an email and check it for tone. Claude can comment on camera kit. Claude does a very nice image recognition for obscure things. What I have become persuaded of is that the /completions API is simply not much more than +10% or a low key helper. I do not need a dumber-than-intern agent going ape on my code…

> I can have ChatGPT read an email and check it for tone. Maybe Im crazy but this alone is a trillion dollar market cap industry imo. msft is worth 3 trillion off the back of similar products. If LLMs are seen as indispensable by every office worker in the country, as I think they are, and every employee has a subscription for $20 a month we're looking at many billions in revenue.

20-60 million office workers in the US * $20/month = $5-15 billion, but for a specialist AI, companies can charge more. $200/month * 10 million people = 24 billion.

Re: The Generative AI Con

#470

Post by famouswaffles »

azizam wrote: post #466
famouswaffles wrote: post #461

Earlier quoted context omitted.

>Simply consider the users that use generative AI in order to perform some unimportant work Says who ? I certainly don't use it for unimportant work. >(as I hope is the case, for generative AI cannot produce anything of novelty, by design) Another Unfounded Assertion >then through simple deduction you can conclude that the generative AI contributed nothing of value. Nonsensical. 'Unimportant work' that people keep do…

You have once again conflated individualistic values with real world values, and shown bias to personal anecdotes. I am afraid you are simply incapable of understanding so there is no point to discussing this further.

>You have once again conflated individualistic values with real world values

I'm not. It's just nonsensical to think there is some 'real world value' independent of the people said product is targeting in the first place. You don't get to tell people what provides them value.

And by the way, this person I replied to in the first place is specifically commenting on this 'individualistic value' so I have no idea why you thought your comment was relevant if you think such distinctions exist.

You're the one who seems bent on personal anecdotes if anything if your assumption is that LLMs are used for 'unimportant work'. I did not initially bring anecdotes or assumptions into the matter at all.

>I am afraid you are simply incapable of understanding

Whatever floats your boat I guess

>there is no point to discussing this further.

Sure

Post reply on HN