Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Carton – Run any ML model from any programming language

carton.run

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Show HN: Carton – Run any ML model from any programming language

#21

Post by janalsncm »

shoo wrote: post #9
conradev wrote: post #6

> Carton wraps your model with some metadata and puts it in a zip file Why a zip file?

zip-file-as-a-container-format seems pragmatic: it's a way to bundle multiple files into one file (easier to manage than scattering multiple files), it avoids introducing a new proprietary format, it can optionally be compressed, support for reading and writing the container format is already widespread. To give two examples of prior art, it worked for Quake 3 data files (.pk3) & geospatial data files (.kmz) Maybe it…

Also docx as well I believe.

Re: Show HN: Carton – Run any ML model from any programming language

#24

Post by r0l1 »

astronautas wrote: post #22

Make it for Go, and I am sold. Running ML models in Go services is still an unsolved problem.

We have a similar high performance AI stack written in Go capable to load many different models from different frameworks. This is work of several years. Just saw your comment and thought about our company internal talk to release everything under an open source license. Thanks for reminding me :) What are your use-cases?

Re: Show HN: Carton – Run any ML model from any programming language

#25

Post by smokel »

otabdeveloper4 wrote: post #20
smokel wrote: post #16

Earlier quoted context omitted.

Replying to this to explain the downvotes. We all think this. My initial thought was that this is probably a startup selling PyTorch-as-a-Service, and I did not bother to read the article. It turns out that I was wrong, and this might even be useful -- if not for the implementation, then perhaps for the idea. However, it turns out to make Hacker News a nicer space if we follow these guidelines: > Please don't post sh…

It's not a shallow dismissal. The selling point of this thing is cross-language interoperability, and while they advertise it, they don't deliver. Sorry, but if your "any language" is "Python or Javascript" your project hasn't even reached the proof of concept stage, it's just a vague idea at this point. Supporting C++ and C will be 90% of the work and the real challenge.

The shallow dismissal that I was referring to is:

> What I think is that this is nothing more than a resume-bolstering effort that doesn't really need to exist and probably won't once OP lands a role at whatever FAANG company they're trying to impress.

The title of the post might be click-bait, but there is an obvious asterisk on the homepage of Carton, and even at a quick glance it is obvious that only very few languages are supported. The claim is so obviously false, that I don't mind. I would not expect support for INTERCAL or Awk.

Yes, it does not deliver, but that does not warrant the personal attack. The author of Carton actually already had internships at Google and Facebook, and currently works at Uber.

Re: Show HN: Carton – Run any ML model from any programming language

#26

Post by astronautas »

r0l1 wrote: post #24
astronautas wrote: post #22

Make it for Go, and I am sold. Running ML models in Go services is still an unsolved problem.

We have a similar high performance AI stack written in Go capable to load many different models from different frameworks. This is work of several years. Just saw your comment and thought about our company internal talk to release everything under an open source license. Thanks for reminding me :) What are your use-cases?

Wow, make it open source quickly!!! :hype:. It's a classic Python REST API for model serving. But we have very low latency constraints. As such, rewriting in more high performant backend languages e.g. Go or Rust would substantially reduce resource usage (by reducing horizontal scaling need). Pre-baked model serving frameworks e.g. Nvidia's Triton aren't an option, since we have to query a feature store, and do some input feature tracking in between. Go seemed like an efficient, developer friendly choice, but there aren't any well maintained model inference libraries in Go up to this day...

Re: Show HN: Carton – Run any ML model from any programming language

#27

Post by Dayshine »

vpanyam wrote: post #12
Areibman wrote: post #7

Maybe I'm missing something here, isn't this largely achieved by ONNX already? [0] https://onnx.ai

That's a good question! There's an FAQ entry on the homepage that touches on this, but let me know if I can improve it: > ONNX converts models while Carton wraps them. Carton uses the underlying framework (e.g. PyTorch) to actually execute a model under the hood. This is important because it makes it easy to use custom ops, TensorRT, etc without changes. For some sophisticated models, "conversion" steps (e.g. to ONNX…

ONNX runtime doesn't convert models, it runs them, and it has bindings in several languages. And most importantly it's tiny compared to the whole python package mess you get with TF or pytorch.

If carton took a TF/pytorch model and just dealt with the conversion into a real runtime, somehow using custom ops for the bits that don't convert, that would be amazing though.

Re: Show HN: Carton – Run any ML model from any programming language

#30

Post by ramoz »

astronautas wrote: post #26
r0l1 wrote: post #24

Earlier quoted context omitted.

We have a similar high performance AI stack written in Go capable to load many different models from different frameworks. This is work of several years. Just saw your comment and thought about our company internal talk to release everything under an open source license. Thanks for reminding me :) What are your use-cases?

Wow, make it open source quickly!!! :hype:. It's a classic Python REST API for model serving. But we have very low latency constraints. As such, rewriting in more high performant backend languages e.g. Go or Rust would substantially reduce resource usage (by reducing horizontal scaling need). Pre-baked model serving frameworks e.g. Nvidia's Triton aren't an option, since we have to query a feature store, and do some…

I’ve also ran models in Go, transformers even T5. There wasn’t that much overhead maybe some annoying compilation stuff but nothing crazy.

This was tensorflow btw which has Go bindings support.

It is a smart & worthwhile move, we also needed to drop python for performance/cost gains.

Post reply on HN