Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devin: AI Software Engineer

cognition-labs.com

Post reply on HN
Print view View original

301–310 of 604 posts

Re: Devin: AI Software Engineer

#301

Post by orthecreedence »

ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

> I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode.

I highly doubt you're the dumb one here.

Re: Devin: AI Software Engineer

#302

Post by xcv123 »

CipherThrowaway wrote: post #204
xcv123 wrote: post #173

Earlier quoted context omitted.

> They produce generic, amateurish copy that reads like it's written by committee. If you were only using GPT 3.5 (free ChatGPT) then your opinion is irrelevant. With GPT-4 you could directly ask it: "rewrite your previous response so that it sounds less generic, less amateurish, and not written by a committee". I'm not even joking. Just provide enough information and tell it what to do. If you don't like the output…

Yes, I've used GPT-4. The writing sounds better, but it still sucks at writing. Most importantly, it feels like it sucks just as much as GPT-3.5 in some deeply important ways. If you use GPT-4 day-to-day, you've probably encountered this sense of a capability wall before. The point where additional prompting, tweaking, re-prompting simply doesn't seem to be yielding better results on the task, or it feels like the is…

> Most writers have already realized that LLMs can't write in any meaningful way.

I know a professional writer who is amazed by what LLMs are capable of already and, given the rate of progress, speculates they will take over many writing jobs eventually.

> If you use GPT-4 day-to-day, you've probably encountered this sense of a capability wall before.

Of course there is a wall with the current models. But almost every time I hit a wall, I have found a way to break past that limit. Interacting with the LLM as I would interact with a person. LLM's perform best with chain of thought reasoning. List out any issues you identified in the original output, ask the LLM to review these issues and list out any other issues that it can identify based on the original requirements, then rewrite it all. And do that several times until it's good enough.

At work I have found GPT-4 to exceed the linguistic capabilities of my colleagues when it comes to summarizing complicated boring business text.

Re: Devin: AI Software Engineer

#303

Post by creato »

joenot443 wrote: post #181
OtherShrezzing wrote: post #162

Earlier quoted context omitted.

>- Small module completion. I write an interface of a class/struct with it's methods and some comments and tell it to fill in. A recent one I did looked something like (abbreviated): Are they considerably better than existing non-AI tools + manual coding for this? In VSCode and Visual Studio, when working with an interface in C# for example, I can click two context menus to have it generate an implementation with con…

That sounds crazy useful and I think speaks most to the maturity of C# and Microsoft's commitment to making it so ergonomic. I'm pretty curious about that feature, I'd love something similar for C++ in VS Code, but thus far I've been doing a pretty similar Copilot flow to the parent comment. It's nothing groundbreaking, but a nice little productivity boost. If I had to take that or a linter, I'd take the linter. Tota…

Visual Studio (not VSCode) has this for C++, though it can be a bit finicky. It’s infinitely better than AI autocomplete, which just makes shit up half the time.

Re: Devin: AI Software Engineer

#304

Post by summerlight »

ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

My personal take is that LLM is fairly good at replacing low level tasks with intuitive patterns. When it comes to a high level ambiguous question that actually has an implication on your daily works and the products, LLM is not helpful anymore than search engines.

Yeah, AI will do the easy and fun jobs for you. You will only need to care difficult decisions that you're going to be responsible for. What a wonderful world...

Re: Devin: AI Software Engineer

#305

Post by beauzero »

PheonixPharts wrote: post #288
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

"To really solve novel problems with LLMs will take a large amount of research, experimentation and prototyping of ideas, but people funding this hype have no patience for that. I fear we'll get hit by a major AI winter when investors get bored, but we'll end up leaving a lot of value on the table simply because there wasn't enough focus and patience on making these incredible tools work."

...this is what happened in 99-2000. It took 3-7 years for the survivors to start making it usable and letting the general public adjust to a new user paradigm (online vs on PC).

Re: Devin: AI Software Engineer

#306

Post by mrguyorama »

rapind wrote: post #270
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

> Just give me a product that works as advertised Almost no products fit this description, and if they do then the marketing department is getting fired. Does a Mcdonalds burger look like the picture? If you go in with a healthy dose of cynicism IMO LLMs can impress. I’d call it a better google search and autocomplete on steroids.

>Does a Mcdonalds burger look like the picture?

It actually does in the countries that require it. You know you can write ACTUAL "truth in advertising" laws right?

Re: Devin: AI Software Engineer

#307

Post by NicoJuicy »

PheonixPharts wrote: post #288
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

Don't forget the 20/80 rule. They haven't even gotten to 15% yet.

Our jobs are safe. I would even expect more "beginners" to try something with AI and then need an actual programmer to help them

( At least, if they are unwilling to invest the time in development and debugging themselves

Ps. Probably all the given examples are in top 3 most popular programming languages.

Re: Devin: AI Software Engineer

#308

Post by gtirloni »

__loam wrote: post #275
CipherThrowaway wrote: post #112

Earlier quoted context omitted.

Ditto. I started out excited about LLMs and eager to use them everywhere, but have become steadily disillusioned as I have tried to apply them to daily tasks, and seen others try and fail in the same way. Honestly, LLMs can't even get language right. They produce generic, amateurish copy that reads like it's written by committee. GPT can't perform to the level of a middle market copywriter or content marketer. I am c…

> For me the "plateau of productivity" after the disillusionment has been using LLMs a bit like search engines. Quick standalone summaries, snippets or thoughts. A nice day-to-day productivity boost, but nothing that's going to allow me to work less hard. And it only took one of the most computationally expensive processes ever devised by man.

If you ignore how much energy you're burning while searching for dozens and dozens of articles that may or may not give you the answer you're looking for. I'd say the electricity that LLMs burn is nothing compared to my energy and time in that regard.

Re: Devin: AI Software Engineer

#309

Post by avip »

PheonixPharts wrote: post #288
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

Thanks, insightful comment.

Re: Devin: AI Software Engineer

#310

Post by ThalesX »

PheonixPharts wrote: post #288
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

> The problem with this entire space is that we have VC hype for work that should ultimately still be being done in research labs.

I also have two crypto-bro friends that are hyping it up without having anything to show for it. Which is why I'm sort of complaining about they hype surrounding it. I agree with your post to a large extent. This is not production ready technology. Maybe tomorrow.

Post reply on HN