Is dlt part of bruin-stack?
depends on what you mean by that, but we do use dlt through ingestr (https://github.com/bruin-data/ingestr), which is used inside Bruin CLI.
Show HN: I built an open-source data pipeline tool in Go
21–30 of 49 posts
Re: Show HN: I built an open-source data pipeline tool in Go
#22Earlier quoted context omitted.
good point, thanks. I'll definitely add some more details about the comparison between different tools. I agree with you 100% on the language part, I think it is an interesting detail for a data tool to be built in Go, but we have a lot more than that, a couple of things we do there is: - everything is local-first: native Python support, local VS Code extension, isolated local environments, etc - very quick iteration…
My #1 feedback would be to expand on the documentation. I really want to know how this is going to benefit me before I start putting in a lot of effort to switch to using it. That means I need to see why it is better than ${EXISTING_TOOL}. I also need to know that it is actually compatible with my existing data pipeline. For example, we have many single tenant databases that are replicated to a central warehouse. Dur…
I would love to add a dedicated section on this, and would love to learn a bit more from you in this. Do you have any particular example tools that you would compare Bruin in your mind that you would like to understand the difference better?
Re: Show HN: I built an open-source data pipeline tool in Go
#23Why there is not MySQL integration? Will you plan to add it? MySQL is very popular.
Re: Show HN: I built an open-source data pipeline tool in Go
#24Hi Burak, thanks for posting! We're looking for a tool in this space and i'll take a look.
Does Bruin support specifying and visualizing DAGs? I didn't see that in the documentation via a quick look, but I thought to ask because you may use different terminology that can be a substitute.
Re: Show HN: I built an open-source data pipeline tool in Go
#25Congrats Burak, I can tell a lot of work has gone into this. If I may recommend, a comparison of this project with similar other/state-of-the-art projects would be really good to have in your documentation set for others to understand how your approach differs from them.
Re: Show HN: I built an open-source data pipeline tool in Go
#26"Interesting, congrats! I've felt the same challenges but ended up using custom Python with dbt and DuckDB. I'll take a look!"
Re: Show HN: I built an open-source data pipeline tool in Go
#27Hi Burak, thanks for posting! We're looking for a tool in this space and i'll take a look. Does Bruin support specifying and visualizing DAGs? I didn't see that in the documentation via a quick look, but I thought to ask because you may use different terminology that can be a substitute.
> specifying and visualizing DAGs
Do you mean like Airflow or Pachyderm? I am also very interested in new tooling in this space that has these features.
Re: Show HN: I built an open-source data pipeline tool in Go
#28This looks cool! How would this compare to Benthos?
Re: Show HN: I built an open-source data pipeline tool in Go
#29I always thought Hamilton [1] does a good job of giving enough visual hooks that draw you in.
I also noticed this pattern where library authors sometimes do a bit extra in terms of discussing and even promoting their competitors, and it makes me trust them more. A “heres why ours is better and everyone else sucks …” section always comes across as the infomercial character who is having quite a hard time peeling an apple to the point you wonder if this the first time they’ve used hands.
One thing wish for is a tool that’s essentially just Celery that doesn’t require a message broker (and can just use a database), and which is supported on Windows. There’s always a handful of edge cases where we’re pulling data from an old 32-bit system on Windows. And basically every system has some not-quite-ergonomic workaround that’s as much work as if you’d just built it yourself.
It seems like it’s just sending a JSON message over a queue or HTTP API and the worker receives it and runs the task. Maybe it’s way harder than I’m envisioning (but I don’t think so because I’ve already written most of it).
I guess that’s one thing I’m not clear on with Bruin, can I run workers if different physical locations and have them carry out the tasks in the right order? Or is this more of a centralized thing (meaning even if its K8s or Dask or Ray, those are all run in a cluster which happens to be distributed, but they’re all machines sitting in the same subnet, which isn’t the definition of a “distributed task” I’m going for.
Re: Show HN: I built an open-source data pipeline tool in Go
#30Hi Burak, thanks for posting! We're looking for a tool in this space and i'll take a look. Does Bruin support specifying and visualizing DAGs? I didn't see that in the documentation via a quick look, but I thought to ask because you may use different terminology that can be a substitute.
> specifying and visualizing DAGs Do you mean like Airflow or Pachyderm? I am also very interested in new tooling in this space that has these features.
yes that's what i'm thinking about.