Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Openlayer (YC S21) – Testing and Evaluation for AI

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Launch HN: Openlayer (YC S21) – Testing and Evaluation for AI

#11

Post by verdverm »

rishramanathan wrote: post #10
shahargl wrote: post #4

how is it different from Traceloop and openllmetry ( https://github.com/traceloop/openllmetry )?

Broadly, on the monitoring side, we’re more focused on evaluating the quality of the model’s outputs (is it violating your rules, handling specific subpopulations / edge cases correctly etc.). OpenLLMetry is more focussed on telemetry and tracing, whereas for us ‘monitoring’ is a means to running your tests on production data. Openlayer’s also intended to be used on non-LLM use cases. Here are a few other ways we’re…

Traceloop's landing page is all about model quality, not metrics. Their open source OpenLLMetry is the metrics part and hooks into the OpenTelemetry ecosystem. There should be no issue with getting alerts via the ecosystem, it's promanent on their pages.

https://www.traceloop.com/

Re: Launch HN: Openlayer (YC S21) – Testing and Evaluation for AI

#12

Post by rishramanathan »

verdverm wrote: post #8

No github*, no pricing, both likely to be issues on HN *ok, there is a gallery project, but something like this I would expect to be the open source variety of startups. I very much expect something like this to be open core.

We realize the lack of information about pricing isn’t ideal, and that people will be turned away by this. In the meantime, we do have a free plan with generous limits that allows you to get started self-serve. This plan isn’t time bounded, so there won’t be pressure to upgrade unless you need increased data limits.

On open-core — we’ve been considering open-sourcing the engine that evaluates your models. Will have more on this soon!

We’re definitely prioritizing increasing transparency, and we appreciate your feedback about it!

Re: Launch HN: Openlayer (YC S21) – Testing and Evaluation for AI

#13

Post by rishramanathan »

nextworddev wrote: post #6

Hmm YC 21- so they pivoted into this after 2 years doing something different?

We’ve actually been building a testing and evaluation platform from the start, but started with discriminative ML tasks like classification and regression. We waited to do a Launch HN because we were mostly focused on enterprise / mid-market.

These past few months, however, we’ve prioritized building out features for testing and monitoring LLMs.

LLMs certainly have their unique challenges, but the evaluation problem in general is not new, and much of what we’ve built historically is very much applicable to this new crop of ML use cases!

Re: Launch HN: Openlayer (YC S21) – Testing and Evaluation for AI

#14

Post by yubozhao »

shahargl wrote: post #4

how is it different from Traceloop and openllmetry ( https://github.com/traceloop/openllmetry )?

I think the target personas is different. While they might have the same capabilities, but the job-to-be-done is different.

openllmetry is focus on engineers, who wants to use this as more of a piping solution and it sits on top of opentelemtry. While opentelemetry is a popular solution. It is just applying a solution to a new problem.

OpenLayers to me is thinking from the ML/AI problems from ground up and while serving the data scientists and probably prompt engineers.

Post reply on HN