Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why eval startups fail (2025)

thomasliao.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Why eval startups fail (2025)

#41

Post by brandensilva »

noelwelsh wrote: post #21
jdw64 wrote: post #6

If you look at the history of software engineering, the ones that made the most money were usually not the companies that built the applications themselves, but the ones that built the tools to verify, deploy, and build them, such as CI/CD, static analysis tools, and testing frameworks. Personally, I agree with the Goodhart problem, but isn't the reason Eval startups fail because they try to sell an 'evaluation servi…

The "shovels for gold miners" analogy is generally a good one. It applies to Nvidia, for example. It doesn't generally apply to developers though. Developer tooling is notoriously difficult to monetize. Developers themselves are a shovel.

Devs are hard to market and sell too I've heard. It's likely because they can build a lot of the stuff out there themselves when pressed. They have the most app exposure so are opinionated. It's why most devs take the open source spoils while everyone else avoids GitHub in general. Although AI has made it easy to setup locally, many still don't see the value of controlling their software or ai agents fully like devs.

Re: Why eval startups fail (2025)

#42

Post by gavinboston »

michaelbuckbee wrote: post #26

I built a simple (free) eval tool for my own uses (Github Gists + Model Outputs) after not being able to find a suitable one in the market. The market's being split into 1. Longitudinal LLM observability tooling Most eval startups have gone down the route of something more like being an observability platform for LLM inference. They want to be in your stack and running the inference to collect data on performance of…

Cool project! I haven't seen that OpenRouter workflow yet (sign into OpenRouter and it creates an API key that your app can use), that looks like an interesting pattern to investigate.

My company recently built a tool that is closer to your first category, but it's an API so it doesn't have the security (supply chain) concern of being embedded in your application.

https://endpointevaluator.com

It's built to help people manage the risk of LLMs changing underneath them and drifting from their designed behavior. Traditional deterministic testing probably won't be sufficient for apps that provide nondeterministic output, like a chatbot backed by an LLM.

The point in the linked article about the challenge of selling developer tools to developers is a good one. I think the first reaction to coding agents is "let's build everything ourselves!" but the long tail of maintenance is still there and the pendulum will probably swing back to "let's stick to our knitting."

Re: Why eval startups fail (2025)

#43

Post by gwbas1c »

thomasliao wrote: post #14
theteapot wrote: post #2

What's an eval?

(Author) It's short for "evaluation", a test for an AI model. Specifically, an AI evaluation comprises (1) a dataset of prompts (as questions / tasks / queries), (2) some way to score model performance on each prompt, like a set of correct answers or a grading rubric that you can use with an LLM autograder, and (3) a metric, such as accuracy¹. (If you're already familiar with the term "benchmark", it's the same thing…

So basically an "eval startup" is somewhere between a middleman or a Consumer Reports?

Re: Why eval startups fail (2025)

#44

Post by 999900000999 »

I’m convinced the only way to make a startup work, with a few exceptions, is to give away your product or sell below cost.

For years upon years until you get brought out. Then it’s someone else’s problem. Or you IPO and bring in new management to figure out how to make money.

VCs don’t see 20x exits happening for Eval companies, so they have trouble with the losing money for years step

Re: Why eval startups fail (2025)

#46

Post by dbish »

"eval startups have a hard time finding customers, because clients have to be technical developers who want to build with APIs, but also not technical enough to run their own evals"

To add to this, they have to be developers who aren't already using a fullstack observability solution, since it's fairly straight forward to add the eval startup featureset to an existing observability solution, and easier (plus cost effective) to just keep it all in one place.

Re: Why eval startups fail (2025)

#47

Post by redwood »

I found this pretty hard to read as the author has a very specific understanding of what an eval startup means but it is only implied rather than explicitly described. I would have thought that they were referring to the companies that provide a technology platform to enable you to do evals in an AI application context for example companies like Comet/Opik and Braintrust.

But it sounds like the author does not mean those companies at all since those are actually important in enabling the very Venn diagram he/she describes.

Based on what I assume the author's referring to they are referring to something more like a public benchmark report provider... I would say but yes that's a relatively small total addressable Market space no matter how you look at it

Re: Why eval startups fail (2025)

#48

Post by intended »

redwood wrote: post #47

I found this pretty hard to read as the author has a very specific understanding of what an eval startup means but it is only implied rather than explicitly described. I would have thought that they were referring to the companies that provide a technology platform to enable you to do evals in an AI application context for example companies like Comet/Opik and Braintrust. But it sounds like the author does not mean t…

Funnily enough, this made immediate sense to me, and I think it derives from being a situation where you need high reliability from a process, eg: I need a bot which has a 99.99% guarantee to not go out of bounds or say something incorrect.

Re: Why eval startups fail (2025)

#49

Post by alexhans »

The way eval startup is defined here is very specific and doesn't cover successful eval farmwork/SaaS vendors like Arize, Promptfoo, deepeval, etc

The author does have a point around generic benchmarks not being super valuable for companies. But evals should be seen as verifying design/behaviour constraints and can greatly aid product building, golden dataset creations and good software practices.

It's just that the aim should be "how to generate your own good evals, even if it's hard" as not so much "here's some generic evals about models".

Re: Why eval startups fail (2025)

#50

Post by dippogriff »

The current way benchmarks are done and are accepted by the community makes for really uninspired work. Until we're willing to break out of this rigid evaluation format prone to crazy overfitting and gaming, talent will move elsewhere. It is kind of a chicken and egg problem though.
Post reply on HN