Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Carton – Run any ML model from any programming language

carton.run

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Show HN: Carton – Run any ML model from any programming language

#51

Post by brap »

mikeravkine wrote: post #50
brap wrote: post #47

Just some random brain dump: Why limit to ML models? Perhaps we can (should?) have some universal package hub, where you can package and push a "thing" from any language, and then pull and use it from any other language. With some metadata describing the input/output schema. The underlying engine can use WASM or containers or something like that.

..isn't this just Docker?

Well... yeah, kind of.

I guess some parts that are missing are having a schema for the CMD part, and being able to easily generate APIs for various languages from that schema

Re: Show HN: Carton – Run any ML model from any programming language

#52

Post by chriscosma »

otteromkram wrote: post #14

"...run any machine learning model from any programming language*." *As long as that language is python or rust. What I think is that this is nothing more than a resume-bolstering effort that doesn't really need to exist and probably won't once OP lands a role at whatever FAANG company they're trying to impress.

OP has already worked at both Facebook and Google, it's doubtful they need any more resume-bolstering.

Re: Show HN: Carton – Run any ML model from any programming language

#53

Post by armchairhacker »

brap wrote: post #47

Just some random brain dump: Why limit to ML models? Perhaps we can (should?) have some universal package hub, where you can package and push a "thing" from any language, and then pull and use it from any other language. With some metadata describing the input/output schema. The underlying engine can use WASM or containers or something like that.

Dynamic libraries, command-line executables, …

Re: Show HN: Carton – Run any ML model from any programming language

#55

Post by WorldMaker »

vpanyam wrote: post #12
Areibman wrote: post #7

Maybe I'm missing something here, isn't this largely achieved by ONNX already? [0] https://onnx.ai

That's a good question! There's an FAQ entry on the homepage that touches on this, but let me know if I can improve it: > ONNX converts models while Carton wraps them. Carton uses the underlying framework (e.g. PyTorch) to actually execute a model under the hood. This is important because it makes it easy to use custom ops, TensorRT, etc without changes. For some sophisticated models, "conversion" steps (e.g. to ONNX…

> This lets you use ONNX if you choose and it enables some interesting use cases (like running models in-browser with WASM)

If someone already has an ONNX model, there's already an in-browser capable ONNX runtime: https://onnxruntime.ai/docs/get-started/with-javascript.html...

(It does use some parts compiled to WASM under the hood, presumably for performance.)

Re: Show HN: Carton – Run any ML model from any programming language

#56

Post by otabdeveloper4 »

liuliu wrote: post #46
otabdeveloper4 wrote: post #19

> From any [*] programming language. [*] If "any programming language" is Python or Javascript.

This is a reasonable approach for systems that allowed to load binaries (either the running artifact is a binary or semi-binary (WASM executable) or it allows to load .so / .dll from user-provided places). It basically runs with the promise that you can package CUDA / PyTorch / Python interpreter into the host language in some way, and use it. This is true for Android, not true for iOS, true for almost all desktop sy…

It's gonna fall apart in a spectacular way when they try to marshal data across compiled language boundaries.

This is the actual hard problem in this domain, not packaging a model file in a zipfile.

Re: Show HN: Carton – Run any ML model from any programming language

#58

Post by astronautas »

huac wrote: post #45
astronautas wrote: post #26

Earlier quoted context omitted.

Wow, make it open source quickly!!! :hype:. It's a classic Python REST API for model serving. But we have very low latency constraints. As such, rewriting in more high performant backend languages e.g. Go or Rust would substantially reduce resource usage (by reducing horizontal scaling need). Pre-baked model serving frameworks e.g. Nvidia's Triton aren't an option, since we have to query a feature store, and do some…

We used Triton Inference Server (with a Golang sidecar to translate requests) for model serving and a separate Go app that handled receiving the request, fetching features, sending to Triton, doing other stuff with the response, serving. This scaled to 100k QPS with pretty good performance but does require some hops. In general writing pure Go inference libraries sucks. Not easy to do array/vector manipulation, not e…

Cool, thanks for sharing!

Re: Show HN: Carton – Run any ML model from any programming language

#59

Post by astronautas »

capableweb wrote: post #42
astronautas wrote: post #23

Is this the same as Nvidia's Triton?

I think this Carton project is on a lower level than Triton. With Triton you'd start the Triton server then make requests against it, while Carton is more like a library that you include in your application/library and code it with the same language you'd write your application/library.

True!
Post reply on HN