Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bauplan – Git-for-data pipelines on object storage

docs.bauplanlabs.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Bauplan – Git-for-data pipelines on object storage

#21

Post by sbpayne »

jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

I have really enjoyed the conversations I have had with Jacopo and Ciro over the years. They have really revisited a lot of assumptions behind commonly used tools/infrastructure in the data space and build something that really has a much better developer experience.

So excited to see them take this step!

Re: Bauplan – Git-for-data pipelines on object storage

#23

Post by pablomendes »

jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

In what kinds of workloads or usage patterns do you see the biggest performance gains vs traditional FaaS + storage stacks?

Re: Bauplan – Git-for-data pipelines on object storage

#24

Post by jtagliabuetooso »

pablomendes wrote: post #23
jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

In what kinds of workloads or usage patterns do you see the biggest performance gains vs traditional FaaS + storage stacks?

In a nutshell, data and AI workloads require fast re-building and vertical scaling:

1) you should not need to redeploy a Lambda if you you're running January and February vs only January now. In the same vein, you should not need to redeploy a lambda if you upgrade from pandas to polars: rebuilding functions is 15x faster than lambda, 7x snowpark (-> https://arxiv.org/pdf/2410.17465)

2) the only way (even in popular orchestrators, e.g. Airflow, not just FaaS) to pass data around in DAGs is through object storage, which is slow and costly: we use Arrow as intermediate data format and over the wire, with a bunch of optimizations in caching and zero-copy sharing to make the development loop extra-fast, and the usage of compute efficient!

Our current customers run near real-time analytics pipelines (Kafka -> S3 / Iceberg -> Bauplan run -> Bauplan query), DS / AI workloads and WAP for data ingestion.

Re: Bauplan – Git-for-data pipelines on object storage

#25

Post by jtagliabuetooso »

sbpayne wrote: post #21
jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

I have really enjoyed the conversations I have had with Jacopo and Ciro over the years. They have really revisited a lot of assumptions behind commonly used tools/infrastructure in the data space and build something that really has a much better developer experience. So excited to see them take this step!

Thanks @sbpayne <3

Re: Bauplan – Git-for-data pipelines on object storage

#26

Post by jtagliabuetooso »

anentropic wrote: post #17
zenlikethat wrote: post #10

Earlier quoted context omitted.

> or is this a platform that maybe runs on k8s and provides its own serverless compute resources? This one, although it’s a custom orchestration system, not Kubernetes. (there are some similarities but our system is really optimized for data workloads) We manage Iceberg for easy data versioning, take care of data caching and Python modules, etc., and you just write some Python and SQL and exec it over your data catal…

So there's no self-hosted option? I think currently the docs are lacking some context if you arrive there via a link rather than via your SaaS home page

Thanks a lot for the feedback, point taken!

Wrt deployment: the system has a control plane (on Bauplan AWS, never see any data, just auth and metadata), and data planes for customers (single tenant, private link, Soc2 compliant and all that).

If by hosting you mean "move the data plane to my cloud", that is entirely possible but not as recommended as the managed offering: in the end, the only dependency we have are off-the-shelf VMs in which we install our binary - and your bucket of course, but that is yours.

If you mean "installing the control plane on my cloud", that is not in the cards at the moment, unless a very special deployment is needed.

My suggestion - before complex deployment discussion - is always super simple: try it for free on public datasets and decide if you like it; running the quick start takes three minutes, just send over your email for access.

If you do like it, we can have a discussion on deployment, which has never been a blocker before.

Post reply on HN