Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bauplan – Git-for-data pipelines on object storage

docs.bauplanlabs.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Bauplan – Git-for-data pipelines on object storage

#11

Post by zenlikethat »

esafak wrote: post #9
jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

It is a service, not an open source tool, as far as I can tell. Do you intend to stay that way? What is the business model and pricing? I am a bit concerned that you want users to swap out both their storage and workflow orchestrator. It's hard enough to convince users to drop one. How does it compare to DuckDB or Polars for medium data?

Yep, staying service.

RE: workflow orchestrators. You can use the Bauplan SDK to query, launch jobs and get results from within your existing platform, we don’t want to replace entirely if it’s doesn’t fit for you, just to augment.

RE: DuckDB and Polars. It literally uses DuckDB under the hood but with two huge upgrades: one, we plug into your data catalog for really efficient scanning even on massive data lake houses, before it hits the DuckDB step. Two, we do efficient data caching. Query results and intermediate scans and stuff can be reused across runs.

More details here: https://www.bauplanlabs.com/blog/blending-duckdb-and-iceberg...

As for Polars, you can use Polars itself within your Python models easily by specifying it in a pip decorator. We install all requested packages within Python modules.

Re: Bauplan – Git-for-data pipelines on object storage

#12

Post by barabbababoon »

esafak wrote: post #9
jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

It is a service, not an open source tool, as far as I can tell. Do you intend to stay that way? What is the business model and pricing? I am a bit concerned that you want users to swap out both their storage and workflow orchestrator. It's hard enough to convince users to drop one. How does it compare to DuckDB or Polars for medium data?

- Yes. it is a service and at least the runner will stay like that for the time being.

- We are not quite live yet, but the pricing model is based on compute capacity and it is divided in tiers (e.g. small=50GB for concurrent scans=$1500/month, large can get up to a TB). infinite queries, infinte jobs, infinite users. The idea is to have a very clear pricing with no sudden increases due to volume.

- You do not have to swap your storage - our runner comes to your S3 bucket and your data never ever have to be anywhere else that is not your S3.

- You do not have to swap your orchestrator either. Most of our clients are actually using it with their existing orchestrator. You call the platform's APIs, including run from your Airflow/Prefect/Temporal tasks https://www.prefect.io/blog/prefect-on-the-lakehouse-write-a...

Does it help?

Re: Bauplan – Git-for-data pipelines on object storage

#13

Post by zenlikethat »

korijn wrote: post #5
jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

How does this compare to dbt? Seems like it can do the same?

Some similarities, but Bauplan offers:

1. Great Python support. Piping something from a structured data catalog into Python is trivial, and so is persisting results. With materialization, you never need to recompute something in Python twice if you don’t want to — you can store it in your data catalog forever.

Also, you can request anything Python package you want, and even have different Python versions and packages in different workflow steps.

2. Catalog integration. Safely make changes and run experiments in branches.

3. Efficient caching and data re-use. We do a ton of tricks behind to scenes to avoid recomputing or rescanning things that have already been done, and pass data between steps with Arrow zero copy tables. This means your DAGs run a lot faster because the amount of time spent shuffling bytes around is minimal.

Re: Bauplan – Git-for-data pipelines on object storage

#14

Post by zenlikethat »

dijksterhuis wrote: post #4
jtagliabuetooso wrote: post #2

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizatio…

the big question i have is — where is the code executed? “the cloud”? who’s cloud? my cloud? your environment on AWS? the paper briefly mentions “bring your own cloud” in 4.5 but the docs page doesn’t seem to have any information on doing that (or at least none that i can find).

The code you execute on your data currently runs in a per-customer AWS account managed by us. We leave the door open for BYOC based on the architecture we’ve designed, but due to lean startup life, that’s not an option yet. We’d definitely be down to chat about it

Re: Bauplan – Git-for-data pipelines on object storage

#15

Post by barabbababoon »

laminarflow027 wrote: post #6
korijn wrote: post #5

Earlier quoted context omitted.

How does this compare to dbt? Seems like it can do the same?

To me they seem like the pythonic version of dbt! Instead of yaml, you write Python code. That, and a lot of on-the-fly computations to generate an optimized workflow plan.

Plenty of stuff in common with dbt's philosophy. One big thing though, dbt does not run your compute or manage your lake. It orchestrate your code and pushes it down to a runtime (e.g. 90% of the time Snowflake).

This IS a runtime.

You import bauplan, write your functions and run them in straight into the cloud - you don't need anything more. When you want to make a pipeline you chain the functions together, and the system manages the dependencies, the containerization, the runtime, and gives you a git-like abstractions over runs, tables and pipelines.

Re: Bauplan – Git-for-data pipelines on object storage

#16

Post by tech_ken »

The Git-like approach to data versioning seems really promising to me, but I'm wondering what those merge operations are expected to look like in practice. In a coding environment, I'd review the PR basically line-by-line to check for code quality, engineering soundness, etc. But in the data case it's not clear to me that a line-by-line review would be possible, or even useful; and I'm also curious about what (if any) tooling is provided to support it?

For example: I saw the YouTube video demo someone linked here where they had an example of a quarterly report pipeline. Say that I'm one of two analysts tasked with producing that report, and my coworker would like to land a bunch of changes. Say in their data branch, the topline report numbers are different from `main` by X%. Clearly it's due to some change in the pipeline, but it seems like I will still have to fire up a notebook and copy+paste chunks of the pipeline to see step-by-step where things are different. Is there another recommended workflow (or even better: provided tooling) for determining which deltas in the pipeline contributed to the X% difference?

Re: Bauplan – Git-for-data pipelines on object storage

#17

Post by anentropic »

zenlikethat wrote: post #10
anentropic wrote: post #8

Earlier quoted context omitted.

I am very interested in this but have some questions after a quick look It mentions "Serverless pipelines. Run fast, stateless Python functions in the cloud." on the home page... but it took me a while of clicking around looking for exactly what the deployment model is e.g. is it the cloud provider's own "serverless functions"? or is this a platform that maybe runs on k8s and provides its own serverless compute resou…

> or is this a platform that maybe runs on k8s and provides its own serverless compute resources? This one, although it’s a custom orchestration system, not Kubernetes. (there are some similarities but our system is really optimized for data workloads) We manage Iceberg for easy data versioning, take care of data caching and Python modules, etc., and you just write some Python and SQL and exec it over your data catal…

So there's no self-hosted option?

I think currently the docs are lacking some context if you arrive there via a link rather than via your SaaS home page

Re: Bauplan – Git-for-data pipelines on object storage

#18

Post by buremba »

Looks interesting! Bauplan seems like a mix of an orchestration engine and a data warehouse. It's similar to Motherduck as it runs DuckDB on managed EC2, with more data engineer-focused branching and Python support similar to SQLMesh.

It's interesting that most vendors compute in their own managed account instead of BYOC though. I understand it's hard to manage compute on the customer cloud for vendors, but I was under the impression that it's a no-go for most enterprise companies. Maybe I'm wrong?

Re: Bauplan – Git-for-data pipelines on object storage

#20

Post by zenlikethat »

tech_ken wrote: post #16

The Git-like approach to data versioning seems really promising to me, but I'm wondering what those merge operations are expected to look like in practice. In a coding environment, I'd review the PR basically line-by-line to check for code quality, engineering soundness, etc. But in the data case it's not clear to me that a line-by-line review would be possible, or even useful; and I'm also curious about what (if any…

That’s a great question. Diffing is one area we’ve thought a bit about but still need to dedicate more cycles to. One thing I would be curious about is, what are you doing in these notebooks to check? For what it’s worth, could possibly have an intermediate Python model that does some calculation to look at differences and materializes the results to a table, which you could then query directly for further insight.

One thing we do have support for “expectations” — model-like Python steps that check data quality, and can flag it if the pipeline violates them.

Post reply on HN