Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an open-source data pipeline tool in Go

github.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Show HN: I built an open-source data pipeline tool in Go

#41

Post by karakanb »

halfcat wrote: post #29

I always thought Hamilton [1] does a good job of giving enough visual hooks that draw you in. I also noticed this pattern where library authors sometimes do a bit extra in terms of discussing and even promoting their competitors, and it makes me trust them more. A “heres why ours is better and everyone else sucks …” section always comes across as the infomercial character who is having quite a hard time peeling an ap…

hey, thanks a lot for sharing your thoughts.

I like the comparison page in Hamilton, and in their examples they operate in the asset level, whereas Bruin crosses the asset level into the orchestrator level as well, effectively bridging the gap there. What Bruin does is beyond a single asset that might be a group of functions, it is basically being able to build and run pipelines of that.

In terms of distributed execution, it is in our roadmap to support running distributed workloads as simple as possible, and Postgres as a pluggable queue backend is one of the options as well. Currently, Bruin is meant as a single-node CLI tool that will do the orchestration and the execution within the same machine.

Re: Show HN: I built an open-source data pipeline tool in Go

#43

Post by peterm4 »

I'd absolutely love to love this.

Using dbt at $JOB, and building a custom dbt adapter for our legacy data repos, I've slowly developed a difficult relationship dbt's internals and externals. Struggling with the way it (python) handles concurrency, threading, timeouts with long running (4hr+ jobs), and the like. Not to mention inconsistencies with the way it handles Jinja in config files vs SQL files. Also it's lack of ingestion handling and VSCode/editor support, which it seems like Bruin considers very well! Since starting poking around on the inside of dbt I've felt like Go or Rust would be a far more suitable platform for a pipeline building tool, and this looks to be going in a great direction, so congrats on the launch and best of luck with your cloud offering.

That being said, I tried starting the example bruin pipeline with duckdb on a current data project, and I'm having no luck getting the connection to appear with `bruin connections list` so nothing will run. So looks like I'm going to have to stick with dbt for now. Might be worth adding some more documentation around the .bruin.yml file; dbt has great documentation listing the purpose and layout of each file in the folder which is very helpful when trying to set things up.

Re: Show HN: I built an open-source data pipeline tool in Go

#44

Post by baris1432 »

peterm4 wrote: post #43

I'd absolutely love to love this. Using dbt at $JOB, and building a custom dbt adapter for our legacy data repos, I've slowly developed a difficult relationship dbt's internals and externals. Struggling with the way it (python) handles concurrency, threading, timeouts with long running (4hr+ jobs), and the like. Not to mention inconsistencies with the way it handles Jinja in config files vs SQL files. Also it's lack…

thanks for the feedback! I’ve put together a basic tutorial https://bruin-data.github.io/bruin/getting-started/tutorials... that shows how to configure .bruin.yml and build a pipeline using DuckDB—it might help get things running.

Your point on .bruin.yml documentation is spot on, and we’ll make improving that a priority. If you’re still running into issues, please don’t hesitate to reach out—I’d be happy to help debug this with you directly. Thanks again for giving Bruin a try!

Re: Show HN: I built an open-source data pipeline tool in Go

#45

Post by karakanb »

peterm4 wrote: post #43

I'd absolutely love to love this. Using dbt at $JOB, and building a custom dbt adapter for our legacy data repos, I've slowly developed a difficult relationship dbt's internals and externals. Struggling with the way it (python) handles concurrency, threading, timeouts with long running (4hr+ jobs), and the like. Not to mention inconsistencies with the way it handles Jinja in config files vs SQL files. Also it's lack…

hey, thanks a ton for sharing your thoughts, I appreciate that!

I am sorry to hear that it didn't work, we do have a dedicated page for duckdb specifically here: https://bruin-data.github.io/bruin/platforms/duckdb.html

Would this help with it? I'd love to see how we can improve if you'd like to share your thoughts on that. Please feel free to join our slack community as well, we can talk directly there too.

Re: Show HN: I built an open-source data pipeline tool in Go

#46

Post by karakanb »

gigatexal wrote: post #42

Ingestion with DLT likely would have given you more connections to things. Still very cool. I saw you talking about this on LinkedIn.

thanks! we already use DLT under the hood with ingestr, so some of our connectors already come from there. is that what you meant?

Re: Show HN: I built an open-source data pipeline tool in Go

#47

Post by karakanb »

Multrex wrote: post #23

Why there is not MySQL integration? Will you plan to add it? MySQL is very popular.

while MySQL is very popular, it is very rare to see it in analytical/ML usecases, that's why we haven't added it yet. There's nothing from a technical POV that prevents us from adding, it just hasn't been a priority, I am happy to pull it up if that would help your usecases.

Re: Show HN: I built an open-source data pipeline tool in Go

#48

Post by gigatexal »

karakanb wrote: post #46
gigatexal wrote: post #42

Ingestion with DLT likely would have given you more connections to things. Still very cool. I saw you talking about this on LinkedIn.

thanks! we already use DLT under the hood with ingestr, so some of our connectors already come from there. is that what you meant?

Oh didn’t know that. The last time I looked at ingestr it had fewer — seemingly — connections than DLT advertises.

Re: Show HN: I built an open-source data pipeline tool in Go

#49

Post by tinodb »

karakanb wrote: post #10
kyt wrote: post #5

Why use this over Meltano?

great question! Meltano, if I am not wrong, only does data ingestion (Extract & Load), whereas we go further into the pipeline such as transformation with SQL and Python, ML pipelines, data quality, and more. I guess a more comparable alternative would be Meltano + dbt + Great Expectations + Airflow (for Python stuff), whereas Bruin does all of them at once. In that sense, Bruin's alternative would be a stack rather…

What’s the equivalent of meltano’s (singer) taps in Bruin? There’s already hundreds out there that no one is wanting to duplicate of course :)
Post reply on HN