Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-Shot Text Classification on a low-end CPU-only machine?

news.ycombinator.com

Post reply on HN
Print view View original

11–13 of 13 posts

Re: Zero-Shot Text Classification on a low-end CPU-only machine?

#11

Post by backend-dev-33 »

kkielhofner wrote: post #10
backend-dev-33 wrote: post #3

Earlier quoted context omitted.

Well, looking at Triton Inference Server + OpenVINO backend [1]...uff... as you said: "significant amount of development effort". Not easy to handle when you do it first time. Is ONMX runtime + OpenVINO [2] a good idea ? Seems easier to install and to use: Pre-built Docker image and Python package... Not sure about performance (the hardware-related performance improvements - they are in OpenVINO anyway, right?). [1]…

Hah, it actually gets worse. What I was describing was the Triton ONNX backend with the OpenVINO execution accelerator[0] (not the OpenVINO backend itself). Clear as mud, right? Your issue here is model performance with the additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner. Triton does things like dynamic batching[1] where throughput is increased si…

> additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner.

@kkielhofner thanks a lot! now I realize it. I see, there is even GRPC support in Triton, so it make sense.

Re: Zero-Shot Text Classification on a low-end CPU-only machine?

#12

Post by pilotneko »

backend-dev-33 wrote: post #5
pilotneko wrote: post #4

Hugging Face does maintain a package named Text Embedding Inference (TEI) with GPU/CPU-optimized container images. While I have only used this for hosting embedding models, it does appear to support Roberta architecture classifiers (specifically sentiment analysis). https://github.com/huggingface/text-embeddings-inference You can always run a zero shot pipeline in HF with a simple Flask/FastAPI application.

thanks for Text Embedding Inference (never heard about it before) > you can always run a zero shot pipeline in HF with a simple Flask/FastAPI application. Yeah, sometimes things that are right in front of your nose, you don't see them. you mean this? https://huggingface.co/docs/api-inference/index

Sorry, life got busy and I haven’t been able to get back to you. I was referring to pipelines in the Transformers package from Hugging Face. https://huggingface.co/docs/transformers/v4.45.2/en/main_cla...

These are essentially function calls for you to run pre-trained models. If you want to continue this conversation elsewhere, feel free to shoot me an e-mail. It’s just my username @ gmail.

Re: Zero-Shot Text Classification on a low-end CPU-only machine?

#13

Post by kkielhofner »

backend-dev-33 wrote: post #11
kkielhofner wrote: post #10

Earlier quoted context omitted.

Hah, it actually gets worse. What I was describing was the Triton ONNX backend with the OpenVINO execution accelerator[0] (not the OpenVINO backend itself). Clear as mud, right? Your issue here is model performance with the additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner. Triton does things like dynamic batching[1] where throughput is increased si…

> additional challenge of offering it over a network socket across multiple requests and doing so in a performant manner. @kkielhofner thanks a lot! now I realize it. I see, there is even GRPC support in Triton, so it make sense.

Make sure to check out the existing Triton client libraries:

https://github.com/triton-inference-server/client

Post reply on HN