Viewing profile — janaksunil

do you have some time to chat? janak@withspecific.com

would love to chat and learn more about your set up! here's my email - janak@withspecific.com

we manually vet all codebases and companies

yes!

that's fair - for long horizon engineering tasks would speed still matter?

this is a good question. what would make you reject an otherwise working PR on design grounds?

we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app sto…

i appreciate the feedback, the benchmark is primarily long horizon real world engineering tasks on big private codebases.

we're going to open source some of our tasks and model trajectories as well

all the codebases were written pre-2023, so pre when AI got good at coding

would love to learn why?

will do! happy to chat more on janak@withspecific.com as well

we've done our best to use the native provider's harness. all models were run on 'high' reasoning. this is still v1 and tons of room for improvement - really appreciate your feedba…

makes sense

the tasks on the benchmar are long horizon swe tasks - where GLM does surprisingly well

here's where all the models messed up! its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com we also have the setup in the blog. the …

nope, these were private codebases

hey this seems really interesting - what prompted you to test multiple agents on your codebase?

i'm janak, cofounder of Specific Labs (YC F25) and one of the authors of Real-SWE. if i can help answer any questions please feel free to email me at janak@withspecific.com, happy …

I made an MCP server that plugs into Claude Code and tells you if there are credits or discounts available for tools in your stack. First version was super spammy - it fired every …