GitLab is working on a tool just for data teams • Hacker News Board

Re: GitLab is working on a tool just for data teams

#11

Post by slap_shot » Thu, Aug 02, 2018, 12:09 AM UTC

hn_throwaway_99 wrote: post #3

Be interested to know all the competitors in this space. https://data.world/ is one I am most familiar with.

This projects competes with too many industries to really give a succinct answer, but here's just Extraction/Loading and Analyze:

Extraction/Loading Dell Boomi SAP SAS Pentaho Domo Oracle IBM Microsoft Informatica Talend JitterBit SnapLogic Mulesoft SyncSort Information Builders Actian Attunity Datameer Alteryx Striim Treasure Data Cask StreamSets Snowplow DataTorrent Astronomer Panoply Apache Nifi Stitch Data FlyData Bedrock Data Alooma ETLeap Fivetran Xplenty MethodMill Celigo TerraSky DBSync Youredi Scribe Civis Analytics DataScience Dataloader.io datorama Astera

Analyze Microsostrategy GoodData Sisense Looker Power BI Wagon Birst Tableau Qlik Domo Hue Mode Chartio Periscope Pentaho

The amount of hype and BS in the Notebook space would require me to spend some time combing through that again.

Re: GitLab is working on a tool just for data teams

#12

Post by joshlambert » Thu, Aug 02, 2018, 12:10 AM UTC

slap_shot wrote: post #10

sytse wrote: post #9

Earlier quoted context omitted.

What we're doing different is making one product that does the whole lifecycle instead of having to string tools together. It took us many months to string our toolset together and we felt there had to be a better way. Just like GitLab we try to leverage existing open source projects wherever possible. I agree Fivetran also belongs in extract and load and updated it https://gitlab.com/meltano/meltano/commit/1df9813f5…

As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off. Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools…

slap_shot one of our major goals is to provide a solution that startups and small companies can utilize to start putting their data to work.

It shouldn't take weeks of effort, a data engineer, multiple proprietary solutions, and tens of thousands of dollars to answer key questions like CAC or the efficiency of a given marketing campaign.

We're hoping to lower the barrier to entry in both cost and effort, by providing an open source pre-packaged solution.

Re: GitLab is working on a tool just for data teams

#13

Post by slap_shot » Thu, Aug 02, 2018, 12:19 AM UTC

joshlambert wrote: post #12

slap_shot wrote: post #10

Earlier quoted context omitted.

As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off. Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools…

slap_shot one of our major goals is to provide a solution that startups and small companies can utilize to start putting their data to work. It shouldn't take weeks of effort, a data engineer, multiple proprietary solutions, and tens of thousands of dollars to answer key questions like CAC or the efficiency of a given marketing campaign. We're hoping to lower the barrier to entry in both cost and effort, by providing…

[Obligatory: Someone is downvoting you, but it isn't me. I upvoted you]

Yeah, I get that. The analytics space is very complex and companies, even ones with good engineering teams, don't have the internal knowledge or resources to typically put all this together.

In addition to working in this space, my copmany helps companies set up their analytics stack.

We typically set them up with one cloud-based data integration tool (the one with the most # of integrations they need at the best price), dbt, and one BI tool (usually Looker or Periscope, in that order). All in, that takes us a few weeks to get them set up and going.

I applaud your effort. I just struggle to understand why you accept punting on transformations (and using dbt (amazing library, by the way - great choice)), but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there.

I'm eager to see where this goes.

Re: GitLab is working on a tool just for data teams

#14

Post by cheghook » Thu, Aug 02, 2018, 12:20 AM UTC

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to follow on new issues created on GitLab.com and decide if it's safe release and even then he still reports 4-5 issues to GitLab support after every update. We were expecting it to be an easy `yum update` like a normal package but it's just getting worse update after update. It's so bad that my manager asked me to look into GitHub + another CI/CD solution.

Re: GitLab is working on a tool just for data teams

#15

Post by sciurus » Thu, Aug 02, 2018, 12:22 AM UTC

slap_shot wrote: post #10

sytse wrote: post #9

Earlier quoted context omitted.

What we're doing different is making one product that does the whole lifecycle instead of having to string tools together. It took us many months to string our toolset together and we felt there had to be a better way. Just like GitLab we try to leverage existing open source projects wherever possible. I agree Fivetran also belongs in extract and load and updated it https://gitlab.com/meltano/meltano/commit/1df9813f5…

As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off. Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools…

> designed some API that other tools/platforms could plug in to coordinate a lot of the above jobs with your CI system

That's GitHub's strategy. Don't choose solutions for their customers. Be a platform other tools can plug into.

Gitlab's strategy is to cobble together a bunch of open source software (including their own) to provide a solution out of the box. It's not necessarily the best one for you, but it's certainly less effort for you.

Re: GitLab is working on a tool just for data teams

#16

Post by veritas3241 » Thu, Aug 02, 2018, 12:25 AM UTC

slap_shot wrote: post #10

sytse wrote: post #9

Earlier quoted context omitted.

What we're doing different is making one product that does the whole lifecycle instead of having to string tools together. It took us many months to string our toolset together and we felt there had to be a better way. Just like GitLab we try to leverage existing open source projects wherever possible. I agree Fivetran also belongs in extract and load and updated it https://gitlab.com/meltano/meltano/commit/1df9813f5…

As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off. Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools…

On the analytics side, we're using GitLab CI as our orchestration tool. We're pushing it to its limits and trying to find ways to make it better for us (i.e. data teams) and for GitLab more generally.

I'd love to learn more about what you'd like to see CI be able to do from a dataops perspective.

Re: GitLab is working on a tool just for data teams

#17

Post by sytse » Thu, Aug 02, 2018, 12:32 AM UTC

slap_shot wrote: post #13

joshlambert wrote: post #12

Earlier quoted context omitted.

slap_shot one of our major goals is to provide a solution that startups and small companies can utilize to start putting their data to work. It shouldn't take weeks of effort, a data engineer, multiple proprietary solutions, and tens of thousands of dollars to answer key questions like CAC or the efficiency of a given marketing campaign. We're hoping to lower the barrier to entry in both cost and effort, by providing…

[Obligatory: Someone is downvoting you, but it isn't me. I upvoted you] Yeah, I get that. The analytics space is very complex and companies, even ones with good engineering teams, don't have the internal knowledge or resources to typically put all this together. In addition to working in this space, my copmany helps companies set up their analytics stack. We typically set them up with one cloud-based data integration…

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there."

I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot.

BTW Do you want to do a livestreamed video call to discuss further in the 30 next minutes? You have a lot of knowledge. If so please email me and comment here.

Update: He did email and livestream will happen on https://www.youtube.com/watch?v=F8tEDq3K_pE

Re: GitLab is working on a tool just for data teams

#18

Post by sytse » Thu, Aug 02, 2018, 12:34 AM UTC

cheghook wrote: post #14

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to fo…

I'm sorry to heard your experience with GitLab hasn't been smooth. We have more people then ever working on the core of GitLab. And the number of reported issues per customer are going down. But every problem is one too many. Please email me at sytse@gitlab.com if you're open to a call about your situation.

Re: GitLab is working on a tool just for data teams

#19

Post by slap_shot » Thu, Aug 02, 2018, 12:38 AM UTC

sytse wrote: post #17

slap_shot wrote: post #13

Earlier quoted context omitted.

[Obligatory: Someone is downvoting you, but it isn't me. I upvoted you] Yeah, I get that. The analytics space is very complex and companies, even ones with good engineering teams, don't have the internal knowledge or resources to typically put all this together. In addition to working in this space, my copmany helps companies set up their analytics stack. We typically set them up with one cloud-based data integration…

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there." I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot. BTW Do you want to do a livestreamed video…

Sure - just shot you an email at website@yourhandle.com

Re: GitLab is working on a tool just for data teams

#20

Post by wmccullough » Thu, Aug 02, 2018, 12:44 AM UTC

sytse wrote: post #18

cheghook wrote: post #14

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to fo…

I'm sorry to heard your experience with GitLab hasn't been smooth. We have more people then ever working on the core of GitLab. And the number of reported issues per customer are going down. But every problem is one too many. Please email me at sytse@gitlab.com if you're open to a call about your situation.

I appreciate how dedicated GitLab is to continuously improving the product. Thinking about moving my projects from Bitbucket to GitLab for that reason.