Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight

github.com

Post reply on HN
Print view View original

1–6 of 6 posts

Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight

#1

Post by slogansand »

Last month I shared Loki Mode here. Since then, benchmarks came back.

SWE-Bench: 99.67% (299/300 problems) HumanEval: 98.78% Pass@1 (162/164)

For context, most single-agent systems hit 30-50%. Best proprietary ones hover around 70-80%.

The difference is architecture. 37 specialized agent types across 6 swarms (engineering, ops, business, data, product, growth). Parallel 3-reviewer code review. Feedback loops that actually learn.

To stress test it, I pointed it at a blank folder and said "build a ServiceNow replacement." It ran for 19 hours and built FireLater - complete ticket management, workflows, CMDB, knowledge base, self-service portal. I wrote zero lines of code.

New in this version: - Kanban board to visualize agent actions in real-time - Perpetual improvement via self-healing feedback loops - Smarter swarm coordination

Still open source. MIT license. Still not selling anything.

Loki Mode: https://github.com/asklokesh/claudeskill-loki-mode FireLater (built by Loki Mode): https://github.com/asklokesh/FireLater

Happy to answer questions about the architecture or benchmarks.

Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight
github.com

Re: Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight

#2

Post by slogansand »

Author here. Quick context on the benchmarks:

We used RARV (Retrieve, Analyze, Reason, Validate) pattern with multi-agent collaboration. Each problem gets worked by specialized agents, reviewed by 3 parallel reviewers (code, business logic, security), and only merged after consensus.

The 99.67% isn't cherry-picked. Full run against standard SWE-Bench dataset. Happy to share methodology if anyone wants to reproduce.

Re: Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight

#3

Post by slogansand »

On the swarm architecture for those curious:

Engineering (8 types): frontend, backend, database, mobile, API, QA, perf, infra Operations (8 types): devops, SRE, security, monitoring, incident, release, cost, compliance Business (8 types): marketing, sales, finance, legal, support, HR, investor, partnerships Data (3 types): ML, data eng, analytics Product (3 types): PM, design, tech writer Growth (4 types): growth hacker, community, success, lifecycle Review (3 types): code, business, security

Agents don't step on each other. Frontend agent never thinks about database schemas. QA agent never writes deployment scripts. Domain isolation is key.

Re: Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight

#4

Post by slogansand »

For the skeptics (fair): FireLater repo has full git history. You can see the commits. No human intervention in the implementation phase.

I reviewed outputs and approved deployments. But architecture decisions, code, tests, docs - all Loki Mode.

It's not perfect. Some rough edges. But it works and enterprises can self-host it today.

Re: Show HN: Loki Mode hit 99.67% SWE-Bench – MAF built a SaaS overnight

#5

Post by slogansand »

vs single-agent coding assistants: They tap out around 50% on SWE-Bench. No specialization. No parallel review. No self-healing.

vs other multi-agent frameworks: Most focus on chat or simple task delegation. Loki Mode runs full SDLC - from PRD to deployed product with monitoring and business ops.

vs hiring a team: Obviously humans are better for ambiguous problems. But for well-defined PRDs, this removes the "I'll get to it this weekend" bottleneck.

Post reply on HN