It would certainly be interesting to have a highly respected EU-native inference provider, if only to make the regulatory gods happy
Scaleway does that already but competition is always good.
Hetzner is working on LLM Inference
21–30 of 92 posts
Re: Hetzner is working on LLM Inference
#22Good to see more developments in this space. I quite like this service, which is a little further than Hetzner and has several models to choose from: https://www.infomaniak.com/en/hosting/ai-services
Important note on Infomaniak's offering - when they first launched it, I gave it a try but despite claims that it's OpenAI API compatible, even basic things like sending a base64 encoded image were broken. I reached out to their support, who replied with (verbatim): "Unfortunately, we do not provide support for our AI Service, as the solution is highly unmanaged and uses our API."
"highly unmanaged" didn't fill me with confidence, and "uses our API" is a very weird reason to give for not offering support. I wrote off ever using them for anything beyond email after that reply.
Re: Hetzner is working on LLM Inference
#23Earlier quoted context omitted.
Depends on your mindset. If you are an Engineer that does not need hand-holding and you accept terse answers from support with the gratitude to the human on the other side, then go for it. You will at least not need months of training and tough exams to be an expert in Hetzner Cloud. It is simple, but that's the point.
I mostly use Hetzner baremetal servers and not cloud, but the support there is in my experience very competent, but also VERY german and direct. I can imagine if youre neither german or super technical that this can be intimidating or considered unfriendly
Being slightly on the spectrum, I find Hetzner's "VERY german and direct" support perfect.
Re: Hetzner is working on LLM Inference
#24Potentially interesting article ruined by AI slop hallucinations like > For now, the API is fast, free, and fun to try. The next hardware announcement will tell us much more than another small model would.
what is the hallucination here? it is fast, free and fun to try. And I genuinely think that the hardware decision (if they get bigger gpus) decides if this will be a banger product or not?
nobody writes like that. the article is based on a prompt, which i would much rather read, instead of the ironed out version that the llm produced
Re: Hetzner is working on LLM Inference
#25Hetzner is very efficient hosting servers
Will this be the new division of labor?
Americans - best proprietary models
Chinese - best open weight models
Europeans - best / most efficient inference service
Re: Hetzner is working on LLM Inference
#26Earlier quoted context omitted.
what is the hallucination here? it is fast, free and fun to try. And I genuinely think that the hardware decision (if they get bigger gpus) decides if this will be a banger product or not?
nobody writes like that. the article is based on a prompt, which i would much rather read, instead of the ironed out version that the llm produced
fine, i agree that this doesnt sound like me. But the content is correct and not a hallucination!
Re: Hetzner is working on LLM Inference
#27Earlier quoted context omitted.
Infomaniak is such a shitty company, I had to use them on a previous job I worked at and dealing with them was awful.
That makes it sound like dealing with Hetzner is easy. Is the case?
I only have a couple of VPS plans with them, but my experience is good.
Re: Hetzner is working on LLM Inference
#28Hetzner is very efficient hosting servers Will this be the new division of labor? Americans - best proprietary models Chinese - best open weight models Europeans - best / most efficient inference service
We need frontier open weight models from the West.
Maybe Nvidia's Nemotron can get there.
Re: Hetzner is working on LLM Inference
#29Interesting. I could see them perhaps coming in competitive for models that fit into single cards? Less so playing in the big model serving league...climbing into that esp right now would be madness
Re: Hetzner is working on LLM Inference
#30Interesting. I could see them perhaps coming in competitive for models that fit into single cards? Less so playing in the big model serving league...climbing into that esp right now would be madness
Why do you think this would be madness? It seems like, at least in the EU, there is barely competition for the big open-weight models?