Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Carton – Run any ML model from any programming language

carton.run

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Show HN: Carton – Run any ML model from any programming language

#31

Post by astronautas »

ramoz wrote: post #30
astronautas wrote: post #26

Earlier quoted context omitted.

Wow, make it open source quickly!!! :hype:. It's a classic Python REST API for model serving. But we have very low latency constraints. As such, rewriting in more high performant backend languages e.g. Go or Rust would substantially reduce resource usage (by reducing horizontal scaling need). Pre-baked model serving frameworks e.g. Nvidia's Triton aren't an option, since we have to query a feature store, and do some…

I’ve also ran models in Go, transformers even T5. There wasn’t that much overhead maybe some annoying compilation stuff but nothing crazy. This was tensorflow btw which has Go bindings support. It is a smart & worthwhile move, we also needed to drop python for performance/cost gains.

eh, awesome! Seems this one, right? https://github.com/galeone/tfgo. Quite many stars.

Re: Show HN: Carton – Run any ML model from any programming language

#37

Post by ramoz »

astronautas wrote: post #31
ramoz wrote: post #30

Earlier quoted context omitted.

I’ve also ran models in Go, transformers even T5. There wasn’t that much overhead maybe some annoying compilation stuff but nothing crazy. This was tensorflow btw which has Go bindings support. It is a smart & worthwhile move, we also needed to drop python for performance/cost gains.

eh, awesome! Seems this one, right? https://github.com/galeone/tfgo . Quite many stars.

I think just native https://pkg.go.dev/github.com/tensorflow/tensorflow/tensorfl... but tfgo looks interesting.

Actually the docs around this weren’t great. Took the train-in-python & inference-in-go approach. And only for versions greater than tf2

Re: Show HN: Carton – Run any ML model from any programming language

#38

Post by jarym »

This looks interesting - I use OONX to call my PyTorch models from .NET but so far it’s meant I’ve not been able to test out JAX based libraries since they don’t have ONNX export and it has also meant I had to write C# boilerplate code to preprocess my input data into the form required by the model.

Potentially this could be a lot better but I’d be curious what speed overhead the IPC layer adds. At least with ONNX you get a nice speed bump :)

Re: Show HN: Carton – Run any ML model from any programming language

#39

Post by astronautas »

ramoz wrote: post #37
astronautas wrote: post #31

Earlier quoted context omitted.

eh, awesome! Seems this one, right? https://github.com/galeone/tfgo . Quite many stars.

I think just native https://pkg.go.dev/github.com/tensorflow/tensorflow/tensorfl... but tfgo looks interesting. Actually the docs around this weren’t great. Took the train-in-python & inference-in-go approach. And only for versions greater than tf2

Write a blog post then about this! I can tell you it is hardly a solved problem.

Re: Show HN: Carton – Run any ML model from any programming language

#40

Post by Nischalj10 »

carbocation wrote: post #17
Nischalj10 wrote: post #15

is this ancillary to what [these guys]( https://github.com/unifyai/ivy ) are trying to do?

That seems different to me. OP is talking about using ML models outside of python (well, in python, too). That link seems to be talking about using ML models across frameworks (pytorch, tensorflow, jax, etc) in python.

got it. went through both of the codebases. what you say is the case. thanks!
Post reply on HN