Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Carton – Run any ML model from any programming language

carton.run

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Show HN: Carton – Run any ML model from any programming language

#41

Post by ZeroCool2u »

Dayshine wrote: post #27
vpanyam wrote: post #12

Earlier quoted context omitted.

That's a good question! There's an FAQ entry on the homepage that touches on this, but let me know if I can improve it: > ONNX converts models while Carton wraps them. Carton uses the underlying framework (e.g. PyTorch) to actually execute a model under the hood. This is important because it makes it easy to use custom ops, TensorRT, etc without changes. For some sophisticated models, "conversion" steps (e.g. to ONNX…

ONNX runtime doesn't convert models, it runs them, and it has bindings in several languages. And most importantly it's tiny compared to the whole python package mess you get with TF or pytorch. If carton took a TF/pytorch model and just dealt with the conversion into a real runtime, somehow using custom ops for the bits that don't convert, that would be amazing though.

There's an ONNX runtime, but to use the runtime you do need to convert your model into ONNX format first. You can't just run a TF of PyTorch model using the ONNX runtime directly. (At least last time I checked.) Unfortunately this conversion process can be a pain and there needs to be an equivalent operator in ONNX for each op in your TF/Torch execution graph.

Re: Show HN: Carton – Run any ML model from any programming language

#42

Post by capableweb »

astronautas wrote: post #23

Is this the same as Nvidia's Triton?

I think this Carton project is on a lower level than Triton. With Triton you'd start the Triton server then make requests against it, while Carton is more like a library that you include in your application/library and code it with the same language you'd write your application/library.

Re: Show HN: Carton – Run any ML model from any programming language

#43

Post by capableweb »

conradev wrote: post #6

> Carton wraps your model with some metadata and puts it in a zip file Why a zip file?

It's a fairly common way of bundling multiple files into one that has large support and usually "good enough" compression.

It's hardly revolutionary to do this, here are some common examples of things that are zip files but don't label themselves as such:

- .jar

- .odt, .ods, .odp, .docx, .xlsx, .pptx

- .epub

- .apk

- .crx, .xpi

Re: Show HN: Carton – Run any ML model from any programming language

#44

Post by capableweb »

otabdeveloper4 wrote: post #20
smokel wrote: post #16

Earlier quoted context omitted.

Replying to this to explain the downvotes. We all think this. My initial thought was that this is probably a startup selling PyTorch-as-a-Service, and I did not bother to read the article. It turns out that I was wrong, and this might even be useful -- if not for the implementation, then perhaps for the idea. However, it turns out to make Hacker News a nicer space if we follow these guidelines: > Please don't post sh…

It's not a shallow dismissal. The selling point of this thing is cross-language interoperability, and while they advertise it, they don't deliver. Sorry, but if your "any language" is "Python or Javascript" your project hasn't even reached the proof of concept stage, it's just a vague idea at this point. Supporting C++ and C will be 90% of the work and the real challenge.

Maybe you and I have different understandings about what "Proof of Concept" means, but if you're supposed to deliver cross-language interoperability and have successfully delivered it to three different languages with wildly different runtimes, I think I'd consider that a successful proof of concept and since you're demonstrated that the bindings works for at least two other languages, it's more or less trivial to get it to work for N other languages, so this is clearly beyond the proof of concept stage at this point, and trying to reach a maturity stage instead.

Re: Show HN: Carton – Run any ML model from any programming language

#45

Post by huac »

astronautas wrote: post #26
r0l1 wrote: post #24

Earlier quoted context omitted.

We have a similar high performance AI stack written in Go capable to load many different models from different frameworks. This is work of several years. Just saw your comment and thought about our company internal talk to release everything under an open source license. Thanks for reminding me :) What are your use-cases?

Wow, make it open source quickly!!! :hype:. It's a classic Python REST API for model serving. But we have very low latency constraints. As such, rewriting in more high performant backend languages e.g. Go or Rust would substantially reduce resource usage (by reducing horizontal scaling need). Pre-baked model serving frameworks e.g. Nvidia's Triton aren't an option, since we have to query a feature store, and do some…

We used Triton Inference Server (with a Golang sidecar to translate requests) for model serving and a separate Go app that handled receiving the request, fetching features, sending to Triton, doing other stuff with the response, serving. This scaled to 100k QPS with pretty good performance but does require some hops.

In general writing pure Go inference libraries sucks. Not easy to do array/vector manipulation, not easy to do SIMD/CUDA acceleration, cgo is not go, etc. I wrote a fast XGBoost library at least (https://github.com/stillmatic/arboreal) - it's on par with C implementations, but doing anything more complex is going to be tricky.

Re: Show HN: Carton – Run any ML model from any programming language

#46

Post by liuliu »

otabdeveloper4 wrote: post #19

> From any [*] programming language. [*] If "any programming language" is Python or Javascript.

This is a reasonable approach for systems that allowed to load binaries (either the running artifact is a binary or semi-binary (WASM executable) or it allows to load .so / .dll from user-provided places).

It basically runs with the promise that you can package CUDA / PyTorch / Python interpreter into the host language in some way, and use it.

This is true for Android, not true for iOS, true for almost all desktop systems, somewhat true for web (packaging PyTorch + Python interpreter in WASM, the latter is easy, the former, I am unsure), probably not true for FAAS environments (such as Cloudflare worker, or AWS Lambda).

Re: Show HN: Carton – Run any ML model from any programming language

#47

Post by brap »

Just some random brain dump: Why limit to ML models?

Perhaps we can (should?) have some universal package hub, where you can package and push a "thing" from any language, and then pull and use it from any other language. With some metadata describing the input/output schema. The underlying engine can use WASM or containers or something like that.

Re: Show HN: Carton – Run any ML model from any programming language

#48

Post by liuliu »

astronautas wrote: post #22

Make it for Go, and I am sold. Running ML models in Go services is still an unsolved problem.

This seems to be a reasonable approach for Go, but you did need to carry a lot in your containerized environment (Go tends to have very lean container, and this approach requires a fat container with CUDA, PyTorch, Python etc).

Re: Show HN: Carton – Run any ML model from any programming language

#49

Post by 3vidence »

otabdeveloper4 wrote: post #20
smokel wrote: post #16

Earlier quoted context omitted.

Replying to this to explain the downvotes. We all think this. My initial thought was that this is probably a startup selling PyTorch-as-a-Service, and I did not bother to read the article. It turns out that I was wrong, and this might even be useful -- if not for the implementation, then perhaps for the idea. However, it turns out to make Hacker News a nicer space if we follow these guidelines: > Please don't post sh…

It's not a shallow dismissal. The selling point of this thing is cross-language interoperability, and while they advertise it, they don't deliver. Sorry, but if your "any language" is "Python or Javascript" your project hasn't even reached the proof of concept stage, it's just a vague idea at this point. Supporting C++ and C will be 90% of the work and the real challenge.

I gotta agree here, I don't think the process of porting this to a wide array of languages is trivial.

Additionally I would have some serious performance concerns when it comes to marshaling the data across languages boundaries.

Re: Show HN: Carton – Run any ML model from any programming language

#50

Post by mikeravkine »

brap wrote: post #47

Just some random brain dump: Why limit to ML models? Perhaps we can (should?) have some universal package hub, where you can package and push a "thing" from any language, and then pull and use it from any other language. With some metadata describing the input/output schema. The underlying engine can use WASM or containers or something like that.

..isn't this just Docker?
Post reply on HN