Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devin: AI Software Engineer

cognition-labs.com

Post reply on HN
Print view View original

321–330 of 604 posts

Re: Devin: AI Software Engineer

#321

Post by pstorm »

jonahx wrote: post #277
pstorm wrote: post #253

Earlier quoted context omitted.

What percent does an average junior engineer solve? If it is even close, these models can be run all day and night for cheaper than one yearly SWE salary.

The problem is that you still need a human in the loop to determine if you're in the 13% success bucket, or 87% failure bucket, and the time it takes to make that determination is still a significant fraction of just solving the problem. So the actual value here is not "13% of all issues fixed for the cost of compute," but more like "a discount on human time for 13% of the issues". But you also have to factor in the…

You raise a good point - AI + human review might end up being more time than just a human doing everything. I can see a certain subset of issues could be simple enough to done by AI and a quick review - like changing a button color or fixing clearly defined bugs. Time will tell how much work gets shifted over to AI + human review, but I'm betting on most of it.

Re: Devin: AI Software Engineer

#322

Post by notpachet »

reacharavindh wrote: post #268
CipherThrowaway wrote: post #182

Earlier quoted context omitted.

This is part of the problem with the whole discourse of comparing human writers to LLMs. Superficial things like style and tone aren't the problem, but they are overwhelmingly the focus of these discussions. It's funny to see, because developers are so sensitive about being treated like code monkeys by their non-technical colleagues. But these same devs turn around to treat other professionals as word monkeys, or pix…

I actually agree with you that professional writers _can_ write/communicate much better than LLMs. However, I’ve read way too many articles or chapters in books that are so full of needless fluff before they get to the point. It’s almost as if they wanted to show off that they can write all that and somehow connect it to the main part of the article. I’m not reading the essay to appreciate the writer’s ability to nar…

Perhaps the pointless fluff you're describing is actually chaff: countermeasures strategically deployed ahead of time by IQ 180 writers in order to preemptively water down any future LLM's trained on their work.

Then the humans can make a heroic return, write surgical prose like Hemingway to slice through the AI drivel, and keep collecting their paychecks.

Bonus points if you can translate this analogy to software development...

Re: Devin: AI Software Engineer

#323

Post by narrator »

ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

As a developer who is good at object oriented design, architecture, and sucks at leetcode stuff, I have been able to use it to make myself probably twice as productive as I otherwise would be. I just have a conversation with GPT-4 when it doesn't do what I want. "Could you make that object oriented?" "Could you do that for this API instead, here let me paste the docs in for you."

I think people want it to completely replace developers so they can treat programming as a magic box, but it will probably mostly help big picture architecture devs compete with people who are really good at Leetcode type algorithm stuff.

Re: Devin: AI Software Engineer

#324

Post by PodgieTar »

joshuahutt wrote: post #282
singularity2001 wrote: post #263

Earlier quoted context omitted.

So what should the AI build instead? Specifically with regard to UI. I don't want my banking app to run on a bunch of non-deterministic prompts.

Great point. I have no idea. What are your banking use cases? For me, it's mostly information I want. I don't really need a full app for that. I want to know: 1) How much money do I have? 2) Did a check I cashed clear yet? 3) Are there any unusual charges? How is my spending this month? 4) Anything I should look into? For actions I'd want to take: 1) Deposit check 2) Transfer money from account to account 3) Make a p…

This sounds like hell.

Why on earth would you want that? Open the app, go to the account, enter the money with specificity, select the account to transfer it to, click the button.

Sure, being able to say "Transfer 200 to Steve" is nice and all but.. I just don't... consider it much better than the process we have today?

Re: Devin: AI Software Engineer

#325

Post by ramesh31 »

PheonixPharts wrote: post #288
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

> I've personally seen teams working nights and weekends, implementing solutions to never before seen problems in a few weeks, and still getting a thumbs down when they cross the finish line.

This is an important lesson that all SWEs should take to heart. Nobody cares about your novel algorithm. Nobody cares about your high availability architecture. Nobody cares about your millisecond network latency optimizations. The only thing that anyone actually using your software cares about is "Does the screen with lights and colors make the right lights and colors that solve my problem when I click on it?". Anything short of that is yak shaving if your role is not pure academic R&D.

Re: Devin: AI Software Engineer

#326

Post by YeGoblynQueenne »

xcv123 wrote: post #286
empath-nirvana wrote: post #220

Earlier quoted context omitted.

Chat GPT4 is a technological miracle, but it can only produce trite, formulaic text and it's _relentlessly_ polly-anna-ish. Everything reads like ad copy and it's easily identifiable.

Fix your prompt. Just accepting the default style is a rookie mistake. Ask it to "rewrite that in the tone of an English professor" or "rewrite that in the style of a redneck rapper" or "make that sound less like generic ad copy". Get into an argument back and forth with the LLM and tell it the previous response is crap because of XYZ.

Or, you know, spend the half hour that would take writing your stuff yourself.

Re: Devin: AI Software Engineer

#327

Post by HarHarVeryFunny »

PheonixPharts wrote: post #288
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

Agreed, and I think that many of the problems that people think LLMs will become capable of, in fact require AGI.

It may well turn out that LLMs are NOT the path to AGI. You can make them bigger and better, and address some of their shortcomings with various tweaks, but it seems that AGI requires online/continual learning which may prove impossible to retrofit onto a pre-trained transformer. Gradient descent may be the wrong tool for incremental learning.

Re: Devin: AI Software Engineer

#328

Post by feoren »

golergka wrote: post #167
MSFT_Edging wrote: post #67

Humans seek work that provides satisfaction and meaning in their life. For every technological advancement, artisans are the first to be made obsolete. Sure we have landfills full of unworn textiles, the market says its good, but overall, we keep destroying what allows humans to seek meaning. Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. We have outsourced art…

> What will be left? Where will we be? Just stranded without dignity or purpose, left to rot when we no longer produce value. Nobody stops you from paying $1000 for a shirt made by artisans right now. Do you want to? > Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. It's not somebody else who decided that. It's you.

> > Our governments and society have made it clear, if you don't produce value, you don't deserve dignity.

> It's not somebody else who decided that. It's you.

No. The Republican National Convention cheered the idea of letting the poor and sick just die off in the streets. They cheered. Ron Paul asked "what are we supposed to do? Let our sick and poor just die cold in the streets?" and they cheered.

Jobs are sacred in the U.S. Job creators must be worshipped. Hard-working Americans are the lifeblood of yadda yadda. As soon as you don't have a job: fuck you, scum, you deserve to die in the streets. You are no longer of use to the wealthy, so you do not even have the dignity to sleep on benches or under bridges: they add spikes to any area you might find any comfort. Your children cannot receive an education. You get to disappear from view into some secluded slum until you die of the cold. It's not GP that decided that. It's tens (hundreds?) of millions of Americans who will cheer on your death if you lose your job. Most of which, of course, are a couple paychecks or a major illness away from being homeless themselves. Do not act like that's not a real thing.

Re: Devin: AI Software Engineer

#329

Post by notpachet »

kjqgqkejbfefn wrote: post #292
CipherThrowaway wrote: post #204

Earlier quoted context omitted.

Yes, I've used GPT-4. The writing sounds better, but it still sucks at writing. Most importantly, it feels like it sucks just as much as GPT-3.5 in some deeply important ways. If you use GPT-4 day-to-day, you've probably encountered this sense of a capability wall before. The point where additional prompting, tweaking, re-prompting simply doesn't seem to be yielding better results on the task, or it feels like the is…

[flagged]

Jesus Christ, that's horrible. It's something a clever fourth-grader would write.

> my first stint with a hard-on over a clever metaphor

That's all it is.

Re: Devin: AI Software Engineer

#330

Post by notpachet »

narrator wrote: post #323
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

As a developer who is good at object oriented design, architecture, and sucks at leetcode stuff, I have been able to use it to make myself probably twice as productive as I otherwise would be. I just have a conversation with GPT-4 when it doesn't do what I want. "Could you make that object oriented?" "Could you do that for this API instead, here let me paste the docs in for you." I think people want it to completely…

> it will probably mostly help big picture architecture devs compete with people who are really good at Leetcode type algorithm stuff.

The competition should be happening in the other direction.

Post reply on HN