Viewing profile — DiederikVink

Super stoked to finally release this on HN. Looking forward to everyone's feedback!

You're totally right and we are doing a mixed batch. What we changed was the priority of performing prefills over decodes. When looking at a variety of workloads, we realized that …

If you deploy our solution on-prem, you would be able to handle 2x the workload on the same amount of hardware. This ensures you scale up your hardware 2x slower, giving you a ~50%…

1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up …

Thanks for the feedback and the specific model suggestion! Compared to the replicate/modal solutions our big focus is to ensure you don't experience rate limits. We want to ensure …

Thanks for the suggestions! This is definitely something we'll be looking at. We're currently working on providing a more extensive interface to show users a variety of performance…

Thanks for letting us know, we've updated it now! Although we're currently only supporting text models, we do definitely have image and video generation models in our roadmap as th…

That's a great question, but its hard to get enough insight into how Groq is serving models to properly know what's missing. If I had to hazard a guess, it would be that their syst…

Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https:…

Hey, I'm one of the co-founders, thanks for letting us know! I've just run it a few times and seems fine on my end. It does take a second to load the models, but feel free to let m…