GitLab is working on a tool just for data teams

Post by fipple » Thu, Aug 02, 2018, 2:36 AM UTC

slap_shot wrote: post #8

This looks like an amalgamation of 8+ open source projects or industries with products put forth by companies that have dozens of employees and worked on their products for years. It also doesn't even categorize the products they compete with correctly[0]. Why not contribute some of your resources to one of the many active open source libraries already trying to solve some of these problems, and focus your engineerin…

Post by drewbanin » Thu, Aug 02, 2018, 2:42 AM UTC

slap_shot wrote: post #19

sytse wrote: post #17

Earlier quoted context omitted.

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there." I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot. BTW Do you want to do a livestreamed video…

Sure - just shot you an email at website@yourhandle.com

Post by sytse » Thu, Aug 02, 2018, 2:58 AM UTC

mendelk wrote: post #33

jakecodes wrote: post #25

Earlier quoted context omitted.

What a great interview. @slap_shot, you had great questions and you are so well spoken. Really appreciate the feedback. We're all taking notes here. Hope you will keep an eye on our issue tracker for Meltano and give us your feedback as things come up.

I have no horse in this race, but this is so cool that one minute you're exchanging comments on HN and the next you're livestreaming a conversation on the topic! What a world :)

Post by sytse » Thu, Aug 02, 2018, 3:00 AM UTC

Ajedi32 wrote: post #35

fipple wrote: post #32

Earlier quoted context omitted.

You could say the same about Github/Gitlab themselves... that they mash together git and JIRA and .plan etc.

_Especially_ GitLab. Basically their entire product seems to be about building a whole bunch of separate tools and integrating them seamlessly into each other. GitLab has a built-in CI system, a deployment pipeline with Kubernetes integration, a built-in Docker container registry, performance monitoring tools for deployed applications, automated static analysis tools, etc. Describing it as "an amalgamation of 8+ open…

Post by sandra666 » Thu, Aug 02, 2018, 3:58 AM UTC

veritas3241 wrote: post #16

slap_shot wrote: post #10

Earlier quoted context omitted.

As someone who works very, very closely in this industry, I would just be very careful how much of this you think you want to bite off. Consider how you trust using dbt more than rolling your own transformation tool. Why wouldn't this apply to the rest of your stack? The 10+ companies that offer data extraction and loading are likely a better choice. Again with Analytics - the dozens of companies that offer BI tools…

On the analytics side, we're using GitLab CI as our orchestration tool. We're pushing it to its limits and trying to find ways to make it better for us (i.e. data teams) and for GitLab more generally. I'd love to learn more about what you'd like to see CI be able to do from a dataops perspective.