What is Groq (rate limited) missing that you aren't?
Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
11–20 of 36 posts
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#12What is Groq (rate limited) missing that you aren't?
That's a great question, but its hard to get enough insight into how Groq is serving models to properly know what's missing.
If I had to hazard a guess, it would be that their system architecture (# of chips and chip architecture itself) might not be designed for a high concurrency situation.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#13Earlier quoted context omitted.
Same here. Waited 10 seconds. Then gave up. If the list of models takes so long to load, why should I trust you with loading the models themselves? :)
Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https://console.ncompass.tech/public-models . We've put a wrong hyperlink on the website, but we've fixed that now, thanks for letting us know. Regarding us being able to reliably host models versus setting up a website largely comes down to our…
Looks like you put the private models link in your Show HN post text as well - it's worth fixing.
Are you planning to support any image or video generation models, or focusing on text for now?
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#14Earlier quoted context omitted.
Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https://console.ncompass.tech/public-models . We've put a wrong hyperlink on the website, but we've fixed that now, thanks for letting us know. Regarding us being able to reliably host models versus setting up a website largely comes down to our…
Looks like you put the private models link in your Show HN post text as well - it's worth fixing. Are you planning to support any image or video generation models, or focusing on text for now?
I've edited the text to include both links and explain the difference between them. Thanks!
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#15Earlier quoted context omitted.
Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https://console.ncompass.tech/public-models . We've put a wrong hyperlink on the website, but we've fixed that now, thanks for letting us know. Regarding us being able to reliably host models versus setting up a website largely comes down to our…
Looks like you put the private models link in your Show HN post text as well - it's worth fixing. Are you planning to support any image or video generation models, or focusing on text for now?
Thanks for letting us know, we've updated it now!
Although we're currently only supporting text models, we do definitely have image and video generation models in our roadmap as these are very compute intensive models meaning they would benefit greatly from optimizations. We'd love to hear more about any specific models you're hoping to run! Please feel free to message us with further details (diederik.vink@ncompass.tech).
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#16Since you're calling out your support for underserved models, can I request you support some SOTA embeddings models? Support for embeddings is poor from other providers with only a handful of outdated models and poor latency.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#17Since you're calling out your support for underserved models, can I request you support some SOTA embeddings models? Support for embeddings is poor from other providers with only a handful of outdated models and poor latency.
Hey, great that you mentioned this. We actually had BAAI/bge-m3 on our list of models to put up in the near future to see if people had use for it over an API. It's great to hear that this is something you're looking for. If you could let us know if there was a specific model you wanted to run, we can look into getting that put up soon.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#18Unrelated: During the dot-com boom, there was a company called nCompass Labs that developed one of the first content management systems (https://en.wikipedia.org/wiki/NCompass_Labs_Inc). Microsoft bought them in 2001. Their product was, "a plug-in for hosting ActiveX controls in Netscape Navigator named ScriptActive." ActiveX itself was a novelty, using C++ templates to define reusable and _downloadable_ web components.
All of this crap was happily replaced with JavaScript frameworks in later years. Yes, back in the early-2000s, your browser might literally download executable code just to render a custom button.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#19Unrelated: During the dot-com boom, there was a company called nCompass Labs that developed one of the first content management systems ( https://en.wikipedia.org/wiki/NCompass_Labs_Inc ). Microsoft bought them in 2001. Their product was, "a plug-in for hosting ActiveX controls in Netscape Navigator named ScriptActive." ActiveX itself was a novelty, using C++ templates to define reusable and _downloadable_ web compon…
It now makes sense that when we tested the domain ncompass.com it took us to a Microsoft home page, which is why we're ncompass.tech :)
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#20Random idea -- I think it would be cool for hosts that advertise efficiency to have a dashboard that shows total tokens per watt-hour (or whatever usage:energy metric) graphed over time for each model they host, taking into account as much of their infra as possible.
This would:
- let you boast about your cool proprietary optimizations
- naturally get better over time just from applying public algorithmic improvements
- show up hosts that refuse to do the same
- give you a good incentive to keep on top of your own efficiency and competitiveness over time
- be a good response to users who vaguely know that AI takes "a lot" of energy -- it's actually gotten a lot better, but how much better?
Happy to chat if it would help to have a neutral academic voice involved.