Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devin: AI Software Engineer

cognition-labs.com

Post reply on HN
Print view View original

271–280 of 604 posts

Re: Devin: AI Software Engineer

#271

Post by ThalesX »

sesm wrote: post #207
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

I use ChatGPT every day and it’s excellent at: - replacing StackOverflow and library documentation - library search - converting between formats and languages - explaining existing code/queries - deobfuscating code - explaining concepts (kinda hit or miss) - helping you get unstuck when debugging or looking for solution (‘give me possible reasons for …’) I feel like many of this things require asking the right questi…

> replacing StackOverflow and library documentation

I find it horrible at replacing library documentation

> I feel like many of this things require asking the right questions, which assumes certain level of experience. But once you reach this level, it’s an extremely valuable assistant.

I've been using LLM products since incipience. I use them in my daily work life. It's a bit tiring hearing this 'right questions', 'level of experience' and 'reach this level'. Can you share anything concrete that you achieved with ChatGPT that would blow my mind?

I keep hearing this 'you need to ask the right kind of questions bro' from people that never build a single product in their life, and it makes me question my ability to interact with LLM but I never see anything concrete.

Re: Devin: AI Software Engineer

#272

Post by LegibleCrimson »

burningChrome wrote: post #180
MSFT_Edging wrote: post #67

Humans seek work that provides satisfaction and meaning in their life. For every technological advancement, artisans are the first to be made obsolete. Sure we have landfills full of unworn textiles, the market says its good, but overall, we keep destroying what allows humans to seek meaning. Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. We have outsourced art…

I often ask a similar question of what happens when we, as humans, have offloaded everything to technology, to AI to Robots? What kind of a society will we have then? When you no longer have to think about how to do something, or how to build or repair something, or create something original from your imagination. I shudder to think the direction this is all leading to.

I'm kind of an AI pessimist, but I think that sounds like it could be a wonderful society. When the only work you need to do is work that you actually want to do. It could free up people to actually chase their dreams and achieve what they want to, without having to constantly chase subsistence.

The reason I'm a pessimist is that I mostly see society preserving the status quo. I don't see AI democratizing things and freeing us, because we have fetishized the concepts of work and profit that we can't imagine a society that functions properly without demanding those two things be put above all else.

I don't envision a dystopia or a utopia, I envision a future where AI disenfranchises people who should be taken care of, bolsters profits of the already powerful, and replaces the most fulfilling human pursuits without actually saving people from unfulfilling toil, mostly because society will bend backwards to try to preserve the status quo.

Re: Devin: AI Software Engineer

#273

Post by machiaweliczny »

burningChrome wrote: post #180
MSFT_Edging wrote: post #67

Humans seek work that provides satisfaction and meaning in their life. For every technological advancement, artisans are the first to be made obsolete. Sure we have landfills full of unworn textiles, the market says its good, but overall, we keep destroying what allows humans to seek meaning. Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. We have outsourced art…

I often ask a similar question of what happens when we, as humans, have offloaded everything to technology, to AI to Robots? What kind of a society will we have then? When you no longer have to think about how to do something, or how to build or repair something, or create something original from your imagination. I shudder to think the direction this is all leading to.

Raising kids, social stuff, exercise, travel, any leisure stuff like racing, horse ridding, acrobatics. entertainment, cooking, art, gardening etc. Just check what rich girls do and you will see they aren’t bored though they don’t have to work.

Re: Devin: AI Software Engineer

#274

Post by ThalesX »

spaceman_2020 wrote: post #230
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

As with everything about AI, HN once again shows a remarkable inability to project into the future. This site has honestly been absolutely useless when discussing new technology now. No excitement, no curiosity. Just constantly crapping on anything new and lamenting that a brand new technology is not 100% perfect within a year of launch. Remove "Hacker" from this site's name, because I see none of that spirit here an…

This is a post about a present product launch. The future, maybe tomorrow, will be filled with wonder and amazement. Today, we need to understand reality. Not all of us appreciate empty hype. Hackers tinker with reality and build the future. Marketers deal with thin promises.

Re: Devin: AI Software Engineer

#275

Post by __loam »

CipherThrowaway wrote: post #112
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

Ditto. I started out excited about LLMs and eager to use them everywhere, but have become steadily disillusioned as I have tried to apply them to daily tasks, and seen others try and fail in the same way. Honestly, LLMs can't even get language right. They produce generic, amateurish copy that reads like it's written by committee. GPT can't perform to the level of a middle market copywriter or content marketer. I am c…

> For me the "plateau of productivity" after the disillusionment has been using LLMs a bit like search engines. Quick standalone summaries, snippets or thoughts. A nice day-to-day productivity boost, but nothing that's going to allow me to work less hard.

And it only took one of the most computationally expensive processes ever devised by man.

Re: Devin: AI Software Engineer

#276

Post by sergiotapia »

singularity2001 wrote: post #263
joshuahutt wrote: post #261

Earlier quoted context omitted.

We're solving the wrong problem. People trying to use cars to pull horse carts are doomed to fail. Trying to use AI to build the software of yesterday is a waste of time.

So what should the AI build instead? Specifically with regard to UI. I don't want my banking app to run on a bunch of non-deterministic prompts.

Akin to alchemy, spring up the UI to solve the user's problem. When timelines are shortened from weeks to hours, what can we build?

Can a user just talk to a computer and solve their problem regardless of the platform?

Re: Devin: AI Software Engineer

#277

Post by jonahx »

pstorm wrote: post #253
Oras wrote: post #238

From their twitter: > When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. While it is a progress, its far away from being useful to be a software engineer.

What percent does an average junior engineer solve? If it is even close, these models can be run all day and night for cheaper than one yearly SWE salary.

The problem is that you still need a human in the loop to determine if you're in the 13% success bucket, or 87% failure bucket, and the time it takes to make that determination is still a significant fraction of just solving the problem.

So the actual value here is not "13% of all issues fixed for the cost of compute," but more like "a discount on human time for 13% of the issues". But you also have to factor in the time taken on the 87% of issues where leading you down a wrong path can be adding time versus human only. It's not clear to me how it all shakes out, and would require large-sample experiments with humans to determine. I would bet the final margins are small though.

Re: Devin: AI Software Engineer

#278

Post by ThalesX »

rapind wrote: post #270
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

> Just give me a product that works as advertised Almost no products fit this description, and if they do then the marketing department is getting fired. Does a Mcdonalds burger look like the picture? If you go in with a healthy dose of cynicism IMO LLMs can impress. I’d call it a better google search and autocomplete on steroids.

> Does a Mcdonalds burger look like the picture?

Sometimes? But I don't go to McDonalds for the loss function between the picture and actual product. I go for the fast food and good taste (YMMV).

> If you go in with a healthy dose of cynicism IMO LLMs can impress.

I use them everyday in one way or another. But they're not replacing me coding today. Maybe tomorrow. And I go in with a healthy dose of optimism when I say this.

> I’d call it a better google search and autocomplete on steroids.

Sure, but this particular discussion is not about its ability to replace Google Search and / or Autocomplete.

Re: Devin: AI Software Engineer

#279

Post by RyEgswuCsn »

If you need AI to help you program an algorithm, then you shouldn't be using it because you can't tell if AI's solution is correct.

If you can tell if a solution is correct or not --- well, then you don't need to have AI write it for you.

I think AI programming can only work when the industry begin to treat "almost working" systems backed by human customer service as acceptable.

Re: Devin: AI Software Engineer

#280

Post by Eager »

beepbooptheory wrote: post #255
Eager wrote: post #218

Earlier quoted context omitted.

It's incredible how far behind HN of all places is w.r.t. what the current best tech is. So many people talking about GPT-4 here, or even 3.5 when the SOTA has moved way along. Gemini Advanced is also a great model, but for other reasons. That thing really knows a boat load of low level optimization tricks.

I'm sure you know what your talking about, but pushing the point that what is "best" or worth talking about is something that changes like every month does not really help defend against the case that most of this is just hype-churn or marketing.

I'm not pushing what to talk about so much as pushing the point not to talk about stuff that is obsolete and starting to smell.

It's that hype-churn marketing that is a motivating factor for the groups to innovate, much like Formula 1. It might be distasteful, but that doesn't mean it isn't working.

Post reply on HN