GitLab is working on a tool just for data teams
about.gitlab.com
GitLab is working on a tool just for data teams
1–10 of 98 posts
Re: GitLab is working on a tool just for data teams
#2https://quiltdata.com/ ticks a lot of boxes in this space for me.
Re: GitLab is working on a tool just for data teams
#3Be interested to know all the competitors in this space. https://data.world/ is one I am most familiar with.
Re: GitLab is working on a tool just for data teams
#4Taylor from GitLab here! Happy to answer any questions about what we're doing.
Re: GitLab is working on a tool just for data teams
#5Be interested to know all the competitors in this space. https://data.world/ is one I am most familiar with.
Some of the alternatives are listed on in this table in the readme: https://gitlab.com/meltano/meltano/blob/master/README.md#dat...
Re: GitLab is working on a tool just for data teams
#6Be interested to know all the competitors in this space. https://data.world/ is one I am most familiar with.
One major difference will be the complete data life cycle vs providing just one part of it. Just like we do in GitLab except for data teams instead of software development teams.
Re: GitLab is working on a tool just for data teams
#7https://quiltdata.com/ ticks a lot of boxes in this space for me.
Is that project more for versioning data like https://docs.dotmesh.com/tutorials/subdots/ or http://www.pachyderm.io/ ?
Re: GitLab is working on a tool just for data teams
#8This looks like an amalgamation of 8+ open source projects or industries with products put forth by companies that have dozens of employees and worked on their products for years.
It also doesn't even categorize the products they compete with correctly[0].
Why not contribute some of your resources to one of the many active open source libraries already trying to solve some of these problems, and focus your engineering efforts on your core product?
[0] Fivetran is only considered "Orchestrate" but is actually competes directly with Alooma in the Extract and Load. Also, there are DOZENS of company in that space. https://gitlab.com/meltano/meltano/blob/master/README.md#dat...
Re: GitLab is working on a tool just for data teams
#9This looks like an amalgamation of 8+ open source projects or industries with products put forth by companies that have dozens of employees and worked on their products for years. It also doesn't even categorize the products they compete with correctly[0]. Why not contribute some of your resources to one of the many active open source libraries already trying to solve some of these problems, and focus your engineerin…
What we're doing different is making one product that does the whole lifecycle instead of having to string tools together. It took us many months to string our toolset together and we felt there had to be a better way. Just like GitLab we try to leverage existing open source projects wherever possible.
I agree Fivetran also belongs in extract and load and updated it https://gitlab.com/meltano/meltano/commit/1df9813f5ab42c4479... Do you think it should be removed from Orchestrate? Any other suggestions for proprietary products in that category?
Re: GitLab is working on a tool just for data teams
#10This looks like an amalgamation of 8+ open source projects or industries with products put forth by companies that have dozens of employees and worked on their products for years. It also doesn't even categorize the products they compete with correctly[0]. Why not contribute some of your resources to one of the many active open source libraries already trying to solve some of these problems, and focus your engineerin…
What we're doing different is making one product that does the whole lifecycle instead of having to string tools together. It took us many months to string our toolset together and we felt there had to be a better way. Just like GitLab we try to leverage existing open source projects wherever possible. I agree Fivetran also belongs in extract and load and updated it https://gitlab.com/meltano/meltano/commit/1df9813f5…
As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off.
Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools are probably going to be the better choice.
Maybe you can build all these tools better than the hundreds of companies with thousands of employees and millions of dollars. It just seems like the odds that you build the best of each is so unlikely.
I would have been more impressed if your team had designed some API that other tools/platforms could plug in to coordinate a lot of the above jobs with your CI system. There is a SERIOUS need for that and I've had a lot of conversations with companies about what that would look like.
To answer your quest, no, Fivetran does not currently belong in the orchestration area, IMO. I've heard they are soon to release some sort of orchestration tooling to compete with dbt, but it isn't the type of orchestration you get with Airflow.