Show HN: Carton – Run any ML model from any programming language

The goal of Carton is to let you use a single interface to run any machine learning model from any programming language.

It’s currently difficult to integrate models that use different technologies (e.g. TensorRT, Ludwig, TorchScript, JAX, GGML, etc) into your application, especially if you’re not using Python. Even if you learn the details of integrating each of these frameworks, running multiple frameworks in one process can cause hard-to-debug crashes.

Ideally, the ML framework a model was developed in should just be an implementation detail. Carton lets you decouple your application from specific ML frameworks so you can focus on the problem you actually want to solve.

At a high level, the way Carton works is by running models in their own processes and using an IPC system to communicate back and forth with low overhead. Carton is primarily implemented in Rust, with bindings to other languages. There are lots more details linked in the architecture doc below.

Importantly, Carton uses your model’s original underlying framework (e.g. PyTorch) under the hood to actually execute the model. This is meaningful because it makes Carton composable with other technologies. For example, it’s easy to use custom ops, TensorRT, etc without changes. This lets you keep up with cutting-edge advances, but decouples them from your application.

I’ve been working on Carton for almost a year now and I’m excited to open source it today!

Some useful links:

* Website, docs, quickstart - https://carton.run

* Explore existing models - https://carton.pub

* Repo - https://github.com/VivekPanyam/carton

* Architecture - https://github.com/VivekPanyam/carton/blob/main/ARCHITECTURE...

Please let me know what you think!

Show HN: Carton – Run any ML model from any programming language
carton.run

Yes, that's a use case Carton supports.

For exmaple, if your model contains arbitrary Python code, you'd pack it using [1] and then you could load it from another language using [2]. In this case, Carton transparently spins up an isolated Python interpreter under the hood to run your model (even if the rest of your application is in another language).

You can take it one step further if you're using certain DL frameworks. For example, you can create a TorchScript model in Python [3] and then use it from any programming language Carton supports without requiring python at runtime (i.e. your model runs completely in native code).

[1] https://carton.run/docs/packing/python

[2] https://carton.run/docs/loading

[3] https://carton.run/docs/packing/torchscript

Seems almost too good to be true, but I really hope it's not. How does it handle things like CUDA dependencies? Can it somehow make those portable too? Or is GPU acceleration not quite there yet?

Maybe I'm missing something here, isn't this largely achieved by ONNX already?

[0] https://onnx.ai

zip-file-as-a-container-format seems pragmatic: it's a way to bundle multiple files into one file (easier to manage than scattering multiple files), it avoids introducing a new proprietary format, it can optionally be compressed, support for reading and writing the container format is already widespread.

To give two examples of prior art, it worked for Quake 3 data files (.pk3) & geospatial data files (.kmz)

Maybe it's not the best choice but it doesn't seem like a bad one.