41–46 of 46 posts
I'd love to see a 10 minute YouTube video of the capabilities of this product.
For real-world enterprise grade deployment stories, just use our blog or reach out to any of us to learn more (jacopo.tagliabue@bauplanlabs.com).
Earlier quoted context omitted.
So there's no self-hosted option? I think currently the docs are lacking some context if you arrive there via a link rather than via your SaaS home page
Thanks a lot for the feedback, point taken! Wrt deployment: the system has a control plane (on Bauplan AWS, never see any data, just auth and metadata), and data planes for customers (single tenant, private link, Soc2 compliant and all that). If by hosting you mean "move the data plane to my cloud", that is entirely possible but not as recommended as the managed offering: in the end, the only dependency we have are o…
Just trying to understand how it all fits together
Earlier quoted context omitted.
Thanks a lot for the feedback, point taken! Wrt deployment: the system has a control plane (on Bauplan AWS, never see any data, just auth and metadata), and data planes for customers (single tenant, private link, Soc2 compliant and all that). If by hosting you mean "move the data plane to my cloud", that is entirely possible but not as recommended as the managed offering: in the end, the only dependency we have are o…
so like the Lambda funcs in the examples - do I deploy those myself to my own infra? or they have to be defined using Serverless framework and get deployed to Bauplan-controlled infra? are they in the control plane or the data plane? Just trying to understand how it all fits together
So, the AWS lambda in the data product example is a bit of a red herring, and it's used as the outer process to create branches and launch bauplan pipelines through the Python client (https://github.com/BauplanLabs/data-products-with-bauplan/bl...).
It can be your laptop, an Airflow task, a prefect flow or a step function or a cron job on a VM - it's the "host" process (for the data product we picked lambda because it's the easiest way for people to "run small Python stuff every 5 minutes" - this is a prefect example: https://www.prefect.io/blog/prefect-on-the-lakehouse-write-a...).
When you interact with the Bauplan lakehouse, all the compute happen on bauplan, nothing happens in the lambda: think of launching a Snowflake query from a lambda - the client is in the lambda but all the work is done in the SF cloud. Unlike many (all?) other lakehouses, Bauplan is code-first, so you can program the entire branching and merging patterns with a few lines of code, offloading the runtime to the platform.
The platform itself runs on standard EC2, which contains the dockerized functions needed for execution - typically we manage Ec2 in single tenant, private link, soc2 compliant account we own for simplicity, but nothing prevents the VMs to be somewhere else (given connectivity is ok etc.). It is our philosophy that you should not worry about the infra part of it, so even in case of BYOC we will be in charge of managing that.
Does it help clarify the mental model?
Earlier quoted context omitted.
so like the Lambda funcs in the examples - do I deploy those myself to my own infra? or they have to be defined using Serverless framework and get deployed to Bauplan-controlled infra? are they in the control plane or the data plane? Just trying to understand how it all fits together
Sorry for the confusing example. So, the AWS lambda in the data product example is a bit of a red herring, and it's used as the outer process to create branches and launch bauplan pipelines through the Python client ( https://github.com/BauplanLabs/data-products-with-bauplan/bl... ). It can be your laptop, an Airflow task, a prefect flow or a step function or a cron job on a VM - it's the "host" process (for the data…
Earlier quoted context omitted.
> For what it’s worth, could possibly have an intermediate Python model that...materializes the results to a table I think this is kind of the answer I was looking for, and in other systems I've actually manually implemented things like this with a "temp materialize" operator that's enabled by a "debug_run=True" flag. With the NB thing basically I'm trying to "step inside" the data pipeline, like how an IDE debugger…
"In the notebook I'll typically try to replicate (as close as possible) the state of the data inside some intermediate step, and will then manually mutate the pipeline between the original and branch versions to determine how the pipeline changes relate to the data changes." You can automate many changes / tests by materializing the parent(s) of the target table, and use the SDK to produce variations of a pipeline pr…
Earlier quoted context omitted.
"In the notebook I'll typically try to replicate (as close as possible) the state of the data inside some intermediate step, and will then manually mutate the pipeline between the original and branch versions to determine how the pipeline changes relate to the data changes." You can automate many changes / tests by materializing the parent(s) of the target table, and use the SDK to produce variations of a pipeline pr…
Wow that's super cool, thanks for explaining! I will definitely keep an eye out; having that level of certifiability and replayability at a pipeline level is something my team has been having too much angst about. It would be incredible to be able to abstract it away as cleanly as we do code management.
The full auditability is already here today though, so we are always happy to hear your feedback on our public sandbox (you can join for free from our website, and reach out to us at anytime for good and bad feedback!)