Viewing profile — bfogelman

we’ve been using it internally (on sculptor) and the speed ups are crazy — we can now have our agents run tests all the time and iterate quickly! Excited other people can now give …

We’re running both vet and codex on all PRs to do code review and have found they compliment each other well. Vet often catches issues that codex does not!

ooh this is a great idea -- thanks for sharing!

See this comment for some differences: https://news.ycombinator.com/item?id=45428185

nope but vibekit looks interesting -- will take a look

hmm not ideal -- will try and take a look and see whats going wrong

haha honestly a little bit ya. One key thing we've learned from working on this is that lowering the barrier to working in parallel is key. Making it easy to merge, context switchi…

in the works! we want it to be possible to always have the best models and agents available

lffgggg excited to see where you take lingo log :)

Hopefully in the next couple of days! You can join the discord and we'll post an announcement when its ready https://discord.gg/GvK8MsCVgk

right now we're using docker -- we're planning to support modal ( https://modal.com/ ) for remote sandboxes and a "local" mode that might use something like worktrees

Member of the team here, happy to answer questions. Took a lot of ups, downs and work to get here but excited to finally get this out. Even more excited to share other features we'…

One thing I’d be curious to see is how well this translates to things outside of HumanEval! How does it compare to using ChatGPT for example.

Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the prob…

If you’re interested there’s a new RL benchmark that was built using Godot (disclaimer I helped make it!) https://github.com/Avalon-Benchmark/avalon