How does it compete with something like Apache Flink?
Show HN: Gaia – Build pipelines in any programming language
41–50 of 69 posts
Re: Show HN: Gaia – Build pipelines in any programming language
#42I may have misunderstood the readme, but what I got was
> integration of services is hard because not everything uses grpc.
> this assumes everything uses grpc
Re: Show HN: Gaia – Build pipelines in any programming language
#43Build powerful pipelines in any programming language we support?
perhaps a cultural thing but I find people gushing on themselves suspicious and distasteful.
yet another but what about X
Re: Show HN: Gaia – Build pipelines in any programming language
#44Looks like a great idea. What's the roadmap for any programming language SDKs in your "Develop Pipelines" section, such as for Rust?
We have just added the support for C++ and currently looking at Rust but it's a bit tricky. There is no official gRPC SDK for Rust but I think we will manage it soon. :-)
Re: Show HN: Gaia – Build pipelines in any programming language
#45YAML in devop is essentially being used as a declarative language. They do have their place in domain specific applications, and can often be quite powerful. While being able to write in any language gives you flexibility, I do wonder about maintainability down the line. How would this be addressed? Or it doesn't matter as this sort of code tends to be thrown away?
The issue with YAML is that the abstraction ceiling is way too low for its use cases. How do you maintain a 500 line YAML file? You have no tools to simplify it apart from super basic replacements
Meanwhile if I give you a 500 line python function you have a lot of tools at your disposal to refactor to improve maintainability (you could actually write basic tests for your CI pipeline config instead of having failures when the config gets pushed out)
Re: Show HN: Gaia – Build pipelines in any programming language
#46like aws stepflow?
Re: Show HN: Gaia – Build pipelines in any programming language
#47I agree with other commenters that it would be really helpful to understand exactly what you mean by "pipeline", and how they can help/improve a project or workflow. What problem does this solve, and how does it solve that problem better than other solutions? Also, very minor nitpick, but your programming language icons shouldn't be in tags. They appear to be clickable, but do nothing, which gives the appearance that…
Looks like it runs a series of (small) self-contained applications written in any language as long as it implements the GRPC interface in a specific order and with consolidated state tracking and logs.
The individual steps are jobs and the whole thing is a pipeline. Since it executes arbitrary code, it has the flexibility to do anything from ETL to CI.
Re: Show HN: Gaia – Build pipelines in any programming language
#48Earlier quoted context omitted.
Thanks for the feedback. You are absolutely right. I think we have answered those questions on our github readme page ( https://github.com/gaia-pipeline/gaia ) but somehow missed that on our webpage. We will work on that! Edit: Oh and thanks. I will fix the links of the language icons. :-)
> but somehow missed that on our webpage. Every web page I've ever seen that looks like yours has had the same problem.
You are right. Often I'm so drown in what I'm doing that I lose the overview. I will do better in the future!
Re: Show HN: Gaia – Build pipelines in any programming language
#49This looks great. To echo other folks’ sentiment - I think you mean CI-like pipelines specifically, though it could be extended to do some other stuff. You probably want to list some concrete use cases on the main site (e.g. CI, cron jobs, ETL?). I’m curious what differences/trade offs there are in Gaia vs something like Argo ( https://github.com/argoproj/argo ) or Buildkite ( https://buildkite.com/ ). It looks like…
Really good idea. Website definitively needs something like that. I will work on this!
Argo is pretty cool but I think it's different. Imagine developing an automation workflow with 20-30 different steps. In Argo you basically have to develop 20-30 different applications, compile them, build a separate docker image and push all these images to a registry.
In my opinion that's too much overhead and a configuration management nightmare as well. Additionally, what if one step needs to share information with another one? What if when this information is not trivial like a binary for example?
Re: Show HN: Gaia – Build pipelines in any programming language
#50I agree with other commenters that it would be really helpful to understand exactly what you mean by "pipeline", and how they can help/improve a project or workflow. What problem does this solve, and how does it solve that problem better than other solutions? Also, very minor nitpick, but your programming language icons shouldn't be in tags. They appear to be clickable, but do nothing, which gives the appearance that…
Looks like it runs a series of (small) self-contained applications written in any language as long as it implements the GRPC interface in a specific order and with consolidated state tracking and logs. The individual steps are jobs and the whole thing is a pipeline. Since it executes arbitrary code, it has the flexibility to do anything from ETL to CI.
Pretty good description. Seems like I still have a lot to learn on how to present an idea right. :-)