Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab is working on a tool just for data teams

about.gitlab.com

Post reply on HN
Print view View original

71–80 of 98 posts

Re: GitLab is working on a tool just for data teams

#71

Post by veritas3241 »

MarHoff wrote: post #55
georgewfraser wrote: post #46

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is: - Studying each source to figure out the right data model - Chasing down a million weird corner cases - Working around dumb bugs in the data sources This is the kind of problem where paying for software really works better. When people build data pip…

I think the point is to provide a set of tools for people that build data pipelines. Period. The software being open source don't reflect in any way WHO will use this tool. Depending on the success of this project, it might be that you could switch your team to this new tool at some point. Personally I work as a "lone wolf" (to my own complains) because I'm in a small company that can't afford a huge team. Most of my…

The scope of Meltano isn't limited to just data pipelines, though that is the first major part of it.

One thing that gets me really excited about it is the way we want to build version control in from the start. To give you an example of where that's really powerful - we have a bunch of dashboards in Looker. Right now, figuring out what Looks/Dashboards rely on a given field is very challenging. If I change a column in my extraction, right now I can fairly easily propagate it to my final transformed table (thanks to dbt!) and even to the LookML. But knowing what in Looker is going to change / break if I change the LookML is way harder.

But if everything was defined in code from extraction, loading, transformation, modeling, _and_ visualization, that'd be really powerful from my perspective.

The Meltano team has several user personas that they're looking at focusing on, data engineers are definitely one of them, but data analyst/BI users are as well, and we want the product to work well for the whole data team.

Re: GitLab is working on a tool just for data teams

#72

Post by veritas3241 »

js8 wrote: post #50
georgewfraser wrote: post #46

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is: - Studying each source to figure out the right data model - Chasing down a million weird corner cases - Working around dumb bugs in the data sources This is the kind of problem where paying for software really works better. When people build data pip…

Can't agree more. IMHO, if you want to make a dent in the space, figure out better debugging tools! In particular, tools that explain how a certain (specific) value was calculated in the system, tools that let you bisect the source data in some way and let you focus on the source data that are likely to have a problem, tools that help you figure out that certain intermediate value in calculations is an outlier, tools…

You're talking about more debugging tools within the transformation steps of a pipeline, right? dbt is helping with that via data tests (see https://gitlab.com/meltano/analytics/tree/master/elt/dbt/tes...) as an example.

I'd love for a more robust way to test data pipelines and the data within them generally. I was at DataEngConf earlier this year and many people were talking about this problem exactly. One way we're trying to address it a bit is by using the Review Apps feature on Merge Requests within GitLab. Right now, when you open an MR on our repo it will create a clone of the data warehouse that's completely isolated from production. This, obviously, can't scale once the DW is beyond a certain size, but I think there are ways to keep this sort of practice going.

Re: GitLab is working on a tool just for data teams

#73

Post by numbsafari »

sytse wrote: post #9
slap_shot wrote: post #8

This looks like an amalgamation of 8+ open source projects or industries with products put forth by companies that have dozens of employees and worked on their products for years. It also doesn't even categorize the products they compete with correctly[0]. Why not contribute some of your resources to one of the many active open source libraries already trying to solve some of these problems, and focus your engineerin…

What we're doing different is making one product that does the whole lifecycle instead of having to string tools together. It took us many months to string our toolset together and we felt there had to be a better way. Just like GitLab we try to leverage existing open source projects wherever possible. I agree Fivetran also belongs in extract and load and updated it https://gitlab.com/meltano/meltano/commit/1df9813f5…

Yo. Just keep doing what you're doing. I dig it.

I'm not 100% with all the tools you are using, but stringing together random SaaS tools and having to survey a random number of open source tools in order to assemble a sensible platform makes way less sense.

At the very least, what we end up with is a group of folks working together in the open to surface some of the limitations and challenges and attempt to work out some of the alternative solutions to the problems that arise in this space.

So, I applaud your effort. Ignore the salesmen and the haters.

Re: GitLab is working on a tool just for data teams

#74

Post by NegatioN »

tamersalama wrote: post #45

Is there some resemblance with Floydhub http://floydhub.com/ ?

Does anyone have a comprehensive list of similar offerings to floydhub? or OSS alternatives?

I think this market is not being served properly, most of them seem to still require most of the heavy lifting to be done by the ML practitioner.

I suppose I would even be okay with a service that just saves all my graphs from tensorboard for later reviewing.

Re: GitLab is working on a tool just for data teams

#75

Post by slap_shot »

MarHoff wrote: post #56
sytse wrote: post #17

Earlier quoted context omitted.

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there." I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot. BTW Do you want to do a livestreamed video…

Non english speaker here you mentioned an OSS solution called "Inbulk" or something like that during the conversation. Could you spell it I'm pretty interested in finding out more about that project but google return a lot of unrelated result because of the name I guess...

https://github.com/embulk/embulk

Re: GitLab is working on a tool just for data teams

#76

Post by sytse »

cheghook wrote: post #49
sytse wrote: post #18

Earlier quoted context omitted.

I'm sorry to heard your experience with GitLab hasn't been smooth. We have more people then ever working on the core of GitLab. And the number of reported issues per customer are going down. But every problem is one too many. Please email me at sytse@gitlab.com if you're open to a call about your situation.

> And the number of reported issues per customer are going down. This doesn't mean anything, maybe the customers are simply tired of reporting issues. For example last year we didn't do any updates for 6 months because we were afraid it'd break something and we were too busy to be willing to spend the time reporting problems. We also don't report issues that are already open on gitlab.com, reporting the issue means y…

Can you maybe share the plot you made and/or the code you made it with?

As GitLab gets more popular I'm not surprised the number of issues grows.

We are measuring a lot of metrics on GitLab.com. And we are shipping a lot of performance improvements to improve those metrics. https://about.gitlab.com/handbook/engineering/performance/#p...

For example a really big MR had a time to first byte of 15 second. It now is 3 seconds. https://www.dropbox.com/s/ymo28t2v4i4jl4x/Screenshot%202018-...

Re: GitLab is working on a tool just for data teams

#77

Post by sytse »

akerro wrote: post #69
cheghook wrote: post #49

Earlier quoted context omitted.

> And the number of reported issues per customer are going down. This doesn't mean anything, maybe the customers are simply tired of reporting issues. For example last year we didn't do any updates for 6 months because we were afraid it'd break something and we were too busy to be willing to spend the time reporting problems. We also don't report issues that are already open on gitlab.com, reporting the issue means y…

>maybe the customers are simply tired of reporting issues. For example last year we didn't do any updates for 6 months because we were afraid it'd break something and we were too busy to be willing to spend the time reporting problems. We just stopped upgrading GitLab over 2 years ago, we're on 8.9

We're correct metrics of how frequently customers upgrade and as far as I can tell it is way above industry average.

I'm sorry to hear you experienced to much breakage. Can you maybe point to a regression or two that stayed open too long or that caused you a lot of trouble so we can learn from it?

Re: GitLab is working on a tool just for data teams

#79

Post by sytse »

gregoriol wrote: post #63
sytse wrote: post #29

Earlier quoted context omitted.

We did https://about.gitlab.com/2017/10/11/from-dev-to-devops/ when we where at 50% of our current number of engineers. So far so good.

Really no, look at all the comments here (and this is only from techies): you have lost us, we don't know anymore what you are doing, or even trying to do.

We are trying to make a single application that covers the whole DevOps lifecycle, from planning your change up to monitoring its effect.

We're doing it because we believe there are emergent benefits to having the lifecycle in a single application https://about.gitlab.com/handbook/product/single-application...

Re: GitLab is working on a tool just for data teams

#80

Post by sytse »

merb wrote: post #51
hunter23 wrote: post #43

Earlier quoted context omitted.

Agreed - our company moved to Gitlab about the same time (2.5 years ago) and it's very clear from their updates that their focus has splintered in different directions. Our company has recently moved to more Microsoft products so I am pushing our CTO to move to Github. If the CEO is following this, please improve basic user stories like: * As a user, I want to easily know who has approved my merge request. Note the w…

Loading a Merge Request with 168 changes, basically breaks a 4cpu's 4gb instance on gitlab, so yes, "the most basic areas of the core product is almost unusable". And while some integration is good... A lot of recent stuff is just "we try to grab the easy money"

Yep, load times of large merge requests was a big problem. In 11.1 we launched a refactor of merge requests to solve this https://about.gitlab.com/2018/07/22/gitlab-11-1-released/#me...

That got the time down for the worst case we measure from 15 seconds to 3 seconds, see https://news.ycombinator.com/item?id=17671300

Post reply on HN