Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Gaia – Build pipelines in any programming language

gaia-pipeline.io

Post reply on HN
Print view View original

21–30 of 69 posts

Re: Show HN: Gaia – Build pipelines in any programming language

#21

Post by AnIdiotOnTheNet »

michelvocks wrote: post #14
mtrpcic wrote: post #8

I agree with other commenters that it would be really helpful to understand exactly what you mean by "pipeline", and how they can help/improve a project or workflow. What problem does this solve, and how does it solve that problem better than other solutions? Also, very minor nitpick, but your programming language icons shouldn't be in tags. They appear to be clickable, but do nothing, which gives the appearance that…

Thanks for the feedback. You are absolutely right. I think we have answered those questions on our github readme page ( https://github.com/gaia-pipeline/gaia ) but somehow missed that on our webpage. We will work on that! Edit: Oh and thanks. I will fix the links of the language icons. :-)

> but somehow missed that on our webpage.

Every web page I've ever seen that looks like yours has had the same problem.

Re: Show HN: Gaia – Build pipelines in any programming language

#24

Post by certifiedloud »

otherflavors wrote: post #20

"any programming language" is * Golang * Java * Python * C++ ?

Those are the languages for which they provide an SDK.

To quote the github readme: "any programming language as long as gRPC is supported"

You can use gaia without an official sdk. You just need to handle the gRPC integration yourself.

Re: Show HN: Gaia – Build pipelines in any programming language

#25

Post by peter303 »

I remember people doing this for seismc data analysis piggy-backed UNIX pipes 40 years ago. They would send and transform JSON-like metadata down the pipe while the massive seismic data would remain in disk files or shared core memory. I think variants of these packages are the most widely used seismic freeware to this day.

Re: Show HN: Gaia – Build pipelines in any programming language

#26

Post by weberc2 »

NightMKoder wrote: post #18

This looks great. To echo other folks’ sentiment - I think you mean CI-like pipelines specifically, though it could be extended to do some other stuff. You probably want to list some concrete use cases on the main site (e.g. CI, cron jobs, ETL?). I’m curious what differences/trade offs there are in Gaia vs something like Argo ( https://github.com/argoproj/argo ) or Buildkite ( https://buildkite.com/ ). It looks like…

Some differences at a glance: argo appears to take a dependency on k8s and buildkite appears to be closed source.

Re: Show HN: Gaia – Build pipelines in any programming language

#27

Post by bpicolo »

michelvocks wrote: post #15
bsenftner wrote: post #13

So, a task/process queue? Is that what this is? How does this vary from the quite mature task/process queues used by media production companies? (Examples include Deadline, and dozens of tested-by-production proprietary queues at every production studio.)

Hey bsenftner. Could you provide a link or an example of what you mean? :-) I googled "Deadline" but what I've found looks not correct. I usually compare Gaia with tools like Jenkins and Spinnaker. For example, many people use Jenkins Pipeline ( https://jenkins.io/solutions/pipeline/ ) which allows you to write CI/CD tasks in Groovy. In my opinion, Gaia fulfills this job way better because it doesn't force you to use…

> Gaia fulfills this job way better because it doesn't force you to use a specific language

CI typically seems much more dependent on bash/unixy sorts of tools to get things done. This seems to not really support that workflow, requiring code to define pipelines instead.

If it's intended to do CI, how do you deal with CI-style tasks, like shuffling around files between pipeline steps? Or the corollary, what does this do that makes it easier to do CI in practice than with typical unix command based workflows? Inherently, it seems like "Create a golang script that can start a subprocess that runs a test suite" is more overhead than "run a test suite".

At first glance, it looks like more of a competitor to, say, AWS step functions, but it doesn't sound like that's what you're targeting.

Re: Show HN: Gaia – Build pipelines in any programming language

#28

Post by peterkelly »

I really don't get it.

From looking through the front page of the site, I see a bunch of code examples, but no explanation of how this differs from using Unix with a bunch of programs that each read from stdin and write to stdout.

I'm sure there's actually something useful about this product, but the website really doesn't give any indication of what that useful thing is.

Re: Show HN: Gaia – Build pipelines in any programming language

#29

Post by bsenftner »

michelvocks wrote: post #15
bsenftner wrote: post #13

So, a task/process queue? Is that what this is? How does this vary from the quite mature task/process queues used by media production companies? (Examples include Deadline, and dozens of tested-by-production proprietary queues at every production studio.)

Hey bsenftner. Could you provide a link or an example of what you mean? :-) I googled "Deadline" but what I've found looks not correct. I usually compare Gaia with tools like Jenkins and Spinnaker. For example, many people use Jenkins Pipeline ( https://jenkins.io/solutions/pipeline/ ) which allows you to write CI/CD tasks in Groovy. In my opinion, Gaia fulfills this job way better because it doesn't force you to use…

Here's Deadline: https://deadline.thinkboxsoftware.com/ I used it in the past, along with the render queues at a few VFX studios. Very feature rich process scheduling and scaling managers are in heavy use by media productions.

Re: Show HN: Gaia – Build pipelines in any programming language

#30

Post by lalaithion »

michelvocks wrote: post #10
rwmj wrote: post #4

Raises the question "what is a pipeline"? I'm assuming you don't mean the "prog | sort" kind of pipeline, but I was no clearer after reading it what you do mean. Also, make the page display something without 2 different Javascript sources having to be enabled.

Thanks for the feedback. I will add that! To answer your question, a pipeline is a real application which consists of at least one function (we call it job). You can compare it with a Jenkins Pipeline ( https://jenkins.io/solutions/pipeline/ ). Therefore, a pipeline is one flow of automation task which consists of one or more steps.

"a pipeline is a real application which consists of at least one function"

So.... every application is a pipeline? This really clears very little up for someone not already familiar with the concept. The Jenkins link is about testing and deploying code. Is that the intended use case for this tool as well?

Post reply on HN