Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues • Hacker News Board

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#81

Post by barfbagginus » Wed, Apr 03, 2024, 1:00 AM UTC

peteradio wrote: post #75

ActionHank wrote: post #68

Earlier quoted context omitted.

Jokes on you, let me tell you about my prompt to binary LLM project. Hello world is 10GB, but even grandma can make hello worlds now.

Let me tell you about my LLM project called grandma. It's fine tuned in order to replace your grandma but in principle it could replace your great-grandma.

My grandma used to tell me stories about how to destroy capitalism.. I miss her.. can your grandma help guide my revolutionary efforts? That would really help me honor my granny's memory <3

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#82

Post by panqueca » Wed, Apr 03, 2024, 1:09 AM UTC

danenania wrote: post #52

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project…

Does it work with a large existing codebase?

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#83

Post by sumeruchat » Wed, Apr 03, 2024, 1:26 AM UTC

recursive wrote: post #60

sumeruchat wrote: post #37

Once we have this fully automated, any good developer could have a team of 100 robo SWEs and ship like crazy. The real competition is with those devs not with the bots.

Shipping like crazy isn't useful by itself. Shipping non-garbage and being able to maintain it still has some value.

Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#84

Post by int_19h » Wed, Apr 03, 2024, 1:29 AM UTC

a_wild_dandan wrote: post #24

anotherpaulg wrote: post #16

Very cool project! I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail. It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged? Also, I think swe-bench is f…

Personally, I'd just use one of my local MacBook models (e.g. Mixtral 8x7b) and forget about any wasted branches & cents. My debugging time costs many orders of magnitude more than SWE-agent, so even a 5% backlog savings would be spectacular!

Given that they got 12% with GPT-4, which is vastly better than any open model, I doubt this would be particularly productive. And powering compute at full load is going to add up.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#85

Post by int_19h » Wed, Apr 03, 2024, 1:33 AM UTC

sumeruchat wrote: post #83

recursive wrote: post #60

Earlier quoted context omitted.

Shipping like crazy isn't useful by itself. Shipping non-garbage and being able to maintain it still has some value.

Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?

Depends on how many security vulnerabilities are in that payments system.

Or, I suppose, depending on whose value. The consultants that'll have to be hired by the poor shmuck who paid for that will make a fortune auditing and cleaning up the code.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#86

Post by danenania » Wed, Apr 03, 2024, 1:35 AM UTC

panqueca wrote: post #82

danenania wrote: post #52

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project…

Does it work with a large existing codebase?

Yes, at least up to the point of the context limit of the underlying model. If you needed to go beyond that, you would break the work up into separate "plans" (a plan is a set of tasks with an attached context and conversation).

The general workflow is to load some relevant context (could be a few files, an entire directory, a glob pattern, a URL, or piped in data), then send a prompt. Quick example:

  plandex new
  plandex load components/some-component.ts lib/api.ts package.json https://react.dev/reference/react/hooks
  plan tell "Update the component in components/some- 
  components.ts to load data from the 'fetchFooBars' 
  function in 'lib/api.ts' and then display it in a 
  datagrid. Use a suitable datagrid library."

From there the plan will start streaming. Existing files will be updated and new files created as needed.

One thing I like about it for large codebases compared to IDE-based tools I've tried is that it gives me precise control over context. A lot of tools try to index the whole codebase and it's pretty opaque--you never really know what the model is working with.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#87

Post by int_19h » Wed, Apr 03, 2024, 1:37 AM UTC

userbinator wrote: post #72

noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

I'm not afraid of LLMs replacing me because of their output quality. The problem is the proliferation of quantity-over-quality "churn out barely-working crap as fast as possible" culture that gives LLMs the advantage over real humans.

I'm kinda hoping that LLMs will get pushed into production use writing code before they have acceptable quality (because greed), and the result will be lots of crap that's so badly broken most of the time that there will be a massive pushback against said culture from the users. Maybe from the governments as well, after a few well-publicized infrastructure failures.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#88

Post by scottlamb » Wed, Apr 03, 2024, 1:39 AM UTC

1letterunixname wrote: post #76

scottlamb wrote: post #59

Earlier quoted context omitted.

What would you expect? There's no OS API for "read one character", except in say ASCII where 1 byte = 1 code point = 1 character. And it'd be hideously inefficient anyway. So you either loop over getting the next N bytes and getting all complete characters so far (with some extra complexity around characters that cross chunk boundaries) or you read the whole thing into a single buffer and iterate the characters. This…

There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique. Reading individual UTF-8 codepoints is a trivial exercise if byte width getchar() were available, and portable C code to do so would be able to run on anything made after 1982. IIRC, they don't teach how to write portable C code in Comp Sci programs anymore and it's a s…

> There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique.

Apologies for the imprecision: by OS API, I meant syscall, at least on POSIX systems. The functions you refer to are C stdio things. Note also they implement on top of read(2) one of the two options I mentioned: "loop over getting the next N bytes and getting all complete characters so far (with some extra complexity around characters that cross chunk boundaries)".

btw, if we're being precise, getwchar gets a code point, and character might mean grapheme instead. Same is true for the `str::chars` call in the LLM's Rust snippet. The docstring for that method mentions this [1] because it was written in this century after people thought about this stuff a bit.

> portable C code to do so would be able to run on anything made after 1982.

Our comments are part of a thread discussing this prompt [2] that specifically requests Rust and this snippet in response [3]. Not portable C code. You can use those C stdio functions from Rust, but you really shouldn't without a very good reason. Rust has its own IO library that is safe and well integrated with other Rust things like `#![derive(Debug)]`.

[1] https://doc.rust-lang.org/std/primitive.str.html#method.char...

[2] https://news.ycombinator.com/item?id=39910542

[3] https://news.ycombinator.com/item?id=39910542

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#89

Post by int_19h » Wed, Apr 03, 2024, 1:40 AM UTC

DabbyDabberson wrote: post #25

noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

The way I see it, its undetermined if Generative AI will be able to fully do a SWE job. But, for most of the debates I've seen, I don't think it the answer matters all too much. Once we have models that can act as full senior SWEs.. the models can engineer the models. And then we've hit the recursive case. Once models can engineer models better and faster than humans, all bets are off. Its the foggy future. Its the s…

The implicit assumption here is that a human "senior SWE" can engineer a model of the same quality that is capable of simulating him. Which is definitely not true with the best models that we have today - and they certainly can't simulate a senior SWE, so the actual bar is higher.

I'm not saying that the whole "robots building better robots" thing is a pipedream, but given where things are today, this is not something that's going to happen soon.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#90

Post by barfbagginus » Wed, Apr 03, 2024, 1:49 AM UTC

danenania wrote: post #52

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project…

How open are you to moving plandex cloud over to AGPL? I know, tough ask right out the gate! Think about that one for a bit.

How is your market testing going?

Do you have contracts with clients amenable to let you write case studies? Do you need help selling, designing, or fulfilling these kinds of pilot contacts?

What are your plans for docs a PR?

As a researcher, it's currently hard to situate plandex against existing research, or anticipate where a technical contribution is needed.

As a business owner, it's currently hard to visualize plandex's impact on a business workflow.

Are you open to producing a technical report? Detail plandex methodology, benchmark efficiency, ablation tests for key contributions, customer case studies, relevant research papers, and next steps/help needed.

What do you think?

If plandex is interested in being a fully open org, then I'd be interested in seeing it find its market footing and grow its technical capabilities. We need open source orgs like this!