Live data from Hacker News

Ask HN: How to benchmark different LLM models in parallel?

news.ycombinator.com

1–3 of 3 posts

Ask HN: How to benchmark different LLM models in parallel?

Post by dhruvagga » Fri, Jul 26, 2024, 1:51 PM UTC

I was trying Langflow recently for some experiments in our open source project - https://github.com/middlewarehq/middleware to build a RAG over DORA metrics.

In my machine, langflow literally brings makes it super slow so testing each model for output is painful. Is there a way I can try out a parallel output from different models to compare?

Re: Ask HN: How to benchmark different LLM models in parallel?

Post by verdverm » Fri, Jul 26, 2024, 2:50 PM UTC

Running models locally, on your development machine, will be slow. You need beefy GPUs to get good token/sec speeds.

Run the models in the cloud, each one on a separate machine, and then invoke them remotely. You can skip the time/cost and use various APIs from 3rd parties directly.

Re: Ask HN: How to benchmark different LLM models in parallel?

Post by dhruvagga » Mon, Jul 29, 2024, 4:55 AM UTC

verdverm wrote: post #2

Running models locally, on your development machine, will be slow. You need beefy GPUs to get good token/sec speeds. Run the models in the cloud, each one on a separate machine, and then invoke them remotely. You can skip the time/cost and use various APIs from 3rd parties directly.

Any service suggestions?

Post reply on HN