One vote for image inputs here.
I would love a fine-tuned qwen-2-vl-72b on demand, but most of the solutions are "talk to us" level expensive.
I'm assuming you beat the price or convenience of a replicate / modal solution?
Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
21–30 of 36 posts
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#221. Why do you have a limited number of models publicly? Do you have to configure each one manually?
2. I don't see the 50% cheaper option. According to your pricing page, 16B+ models will cost $0.90, which is the same price for Together.ai and fireworks.ai
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#23Random idea -- I think it would be cool for hosts that advertise efficiency to have a dashboard that shows total tokens per watt-hour (or whatever usage:energy metric) graphed over time for each model they host, taking into account as much of their infra as possible. This would: - let you boast about your cool proprietary optimizations - naturally get better over time just from applying public algorithmic improvement…
Thanks for the suggestions! This is definitely something we'll be looking at.
We're currently working on providing a more extensive interface to show users a variety of performance metrics of the models they're running. Having efficiency metrics would be a great addition.
I think additionally an important facet of these tests would be providing clarity on the details of the tests to make them reproducible. I find that sometimes reported stats don't quite translate to real-world experiences. It can feel like results are presented using the workloads that look best on a system, so a standardized/reproducible approach would be best.
We're always keen to chat to as many users/experts/academics/enthusiasts as possible. Please feel free to reach me at diederik.vink@ncompass.tech and we can set up a time to meet!
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#24One vote for image inputs here. I would love a fine-tuned qwen-2-vl-72b on demand, but most of the solutions are "talk to us" level expensive. I'm assuming you beat the price or convenience of a replicate / modal solution?
Thanks for the feedback and the specific model suggestion!
Compared to the replicate/modal solutions our big focus is to ensure you don't experience rate limits. We want to ensure you get a good quality of service no matter what.
When it comes to requesting and running specific models, we won't ask you to pay extra just because there's lower demand for that specific model (which it sounds like other providers are doing). We manage scaling up and down instances for the models on your behalf to make sure you get good performance at a fair price point, so you don't have to worry about making the costs work.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#251. Why do you have a limited number of models publicly? Do you have to configure each one manually? 2. I don't see the 50% cheaper option. According to your pricing page, 16B+ models will cost $0.90, which is the same price for Together.ai and fireworks.ai
1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up within a day. There's always optimizations and tests we like to run before listing something as publically available to ensure the best experience for our users.
If a fully self-serve system is something you would like to see, we would love to hear more!
2. Could you please elaborate on the 50% cheaper option? If you're referring to the line on our website, that is due to our efficiency at scale. This efficiency benefit allows us to provide the models at the price that we do without implementing rate limits to manage our costs. Additionally, this 50% more efficient GPU utilization also benefits anyone looking to use our infrastructure for on-prem solutions.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#261. Why do you have a limited number of models publicly? Do you have to configure each one manually? 2. I don't see the 50% cheaper option. According to your pricing page, 16B+ models will cost $0.90, which is the same price for Together.ai and fireworks.ai
1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up within a day. There's always optimizations and tests we like to run before listing something as publically available to ensure the best experience for our users. If a fully self-serve system is something you would like to see, we would love…
> Reduce AI GPU Infrastructure Bills by 50%
Ok so how does #2 help me do this?
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#27Earlier quoted context omitted.
So, while time to first token is lower, throughput might also be lower in most cases?
Per user throughput might be lower at the moment yes. We're working on GPU kernel level optimizations now to fix that. But across all users on our system, the throughput is better because doing more prefills or a large number of grouped decodes has better utilization of the GPU. The idea is that this works for someone who wants to build a product that is consistent across users in terms of initial response but can tr…
I don’t really get it. Prefill saturates compute and decode saturates memory bandwidth. Why are you not doing mixed batch?
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#28Earlier quoted context omitted.
1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up within a day. There's always optimizations and tests we like to run before listing something as publically available to ensure the best experience for our users. If a fully self-serve system is something you would like to see, we would love…
> Reduce AI GPU Infrastructure Bills by 50% Ok so how does #2 help me do this?
If you deploy our solution on-prem, you would be able to handle 2x the workload on the same amount of hardware. This ensures you scale up your hardware 2x slower, giving you a ~50% reduction in your GPU Infrastructure bills.
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#29Earlier quoted context omitted.
Per user throughput might be lower at the moment yes. We're working on GPU kernel level optimizations now to fix that. But across all users on our system, the throughput is better because doing more prefills or a large number of grouped decodes has better utilization of the GPU. The idea is that this works for someone who wants to build a product that is consistent across users in terms of initial response but can tr…
I don’t really get it. Prefill saturates compute and decode saturates memory bandwidth. Why are you not doing mixed batch?
You're totally right and we are doing a mixed batch. What we changed was the priority of performing prefills over decodes.
When looking at a variety of workloads, we realized that prioritizing finishing a query (priotizing decodes) lead to underutilization of the GPU. We noticed there tended to not be enough requests that are concurrently running (because prefill wasn't prioritized) to meaningfully utilize the memory bandwidth with available decodes. This lead to a system that was unfortunately neither compute nor memory bound.
By running mixed batches that prioritize prefills we still compute some decode tokens in our spare capacity, but ensure compute is as saturated as possible. This additionally leads to a buildup of decodes, so that when we are primarily computing decode we're pushing our memory bandwidth as much as we can.
Of course there is still plenty of improvements that can be made on this front. Finding a dynamic balance between prefill and decode that allows us to have both the memory bandwidth and compute being pushed to their limits is the goal from a scheduling perspective. There are a whole host of factors such as the model architecture, input-token:output-token ratio, underlying hardware, KV-cache allocation (and many more) that all play into the pressure placed on memory and compute, so there's definitely still exploration to be done!
Re: Show HN: NCompass Technologies – yet another AI Inference API, but hear us out
#30Since you're calling out your support for underserved models, can I request you support some SOTA embeddings models? Support for embeddings is poor from other providers with only a handful of outdated models and poor latency.
Hey, great that you mentioned this. We actually had BAAI/bge-m3 on our list of models to put up in the near future to see if people had use for it over an API. It's great to hear that this is something you're looking for. If you could let us know if there was a specific model you wanted to run, we can look into getting that put up soon.
Colbert, colqwen are underserved would benefit from a latency optimized inference service