Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab is working on a tool just for data teams

about.gitlab.com

Post reply on HN
Print view View original

51–60 of 98 posts

Re: GitLab is working on a tool just for data teams

#51

Post by merb »

hunter23 wrote: post #43
cheghook wrote: post #14

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to fo…

Agreed - our company moved to Gitlab about the same time (2.5 years ago) and it's very clear from their updates that their focus has splintered in different directions. Our company has recently moved to more Microsoft products so I am pushing our CTO to move to Github. If the CEO is following this, please improve basic user stories like: * As a user, I want to easily know who has approved my merge request. Note the w…

Loading a Merge Request with 168 changes, basically breaks a 4cpu's 4gb instance on gitlab, so yes, "the most basic areas of the core product is almost unusable".

And while some integration is good... A lot of recent stuff is just "we try to grab the easy money"

Re: GitLab is working on a tool just for data teams

#53

Post by danpalmer »

Reading this I was concerned that it would be written in Ruby. While Ruby is a reasonable language for server development, it has almost no data science community when compared with some other ecosystems.

I was very glad to see this is Python! Python has some of the best data tools out there, and a mature ecosystem for solving all the engineering problems that go along with a great data stack.

Re: GitLab is working on a tool just for data teams

#55

Post by MarHoff »

georgewfraser wrote: post #46

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is: - Studying each source to figure out the right data model - Chasing down a million weird corner cases - Working around dumb bugs in the data sources This is the kind of problem where paying for software really works better. When people build data pip…

I think the point is to provide a set of tools for people that build data pipelines. Period. The software being open source don't reflect in any way WHO will use this tool. Depending on the success of this project, it might be that you could switch your team to this new tool at some point.

Personally I work as a "lone wolf" (to my own complains) because I'm in a small company that can't afford a huge team. Most of my (ETL) Transforms are done in SQL which happen to be pretty standardized as opposed to many ETL products I've seen so far.

This solution is probably far from being ready, but I find this approach quite interesting, because it look like a code based ETL that use SQL for transform (so I might be biased). Overall this might result in a more maintainable/versionable data pipeline model than GUI-first ETL which usually generate spaghetti code. Because you are usually forced to regularly adapt data-pipeline to unstable external inputs, being able to easily diff ETL process would be a blessing.

Re: GitLab is working on a tool just for data teams

#56

Post by MarHoff »

sytse wrote: post #17
slap_shot wrote: post #13

Earlier quoted context omitted.

[Obligatory: Someone is downvoting you, but it isn't me. I upvoted you] Yeah, I get that. The analytics space is very complex and companies, even ones with good engineering teams, don't have the internal knowledge or resources to typically put all this together. In addition to working in this space, my copmany helps companies set up their analytics stack. We typically set them up with one cloud-based data integration…

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there." I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot. BTW Do you want to do a livestreamed video…

Non english speaker here you mentioned an OSS solution called "Inbulk" or something like that during the conversation. Could you spell it I'm pretty interested in finding out more about that project but google return a lot of unrelated result because of the name I guess...

Re: GitLab is working on a tool just for data teams

#58

Post by jakecodes »

georgewfraser wrote: post #46

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is: - Studying each source to figure out the right data model - Chasing down a million weird corner cases - Working around dumb bugs in the data sources This is the kind of problem where paying for software really works better. When people build data pip…

Thanks for the great advice! Obviously your years of experience, with trial and error, advice is greatly appreciated.

The idea is to give users a set of default extractors (which are the ones we use internally, so they are battle tested), along with loaders, transformers etc. With documentation on how to build their own. For our MVP, and possibly into the future, it will work similar to Wordpress plugins where you have an extractor directory that you place your extractor which is written following our protocol, and the UI will recognize it and give you choices of extractors to run, same for loaders, and so on.

We do not want to be chasing down every last corner case, for extractors (except for our own) because that's just not a good long term solution, needing constant maintenance (as we've seen already). With user contributions, I believe it can work.

Re: GitLab is working on a tool just for data teams

#60

Post by jakecodes »

cheghook wrote: post #14

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to fo…

It is truly a bummer that you feel you are receiving half assed updates and features with constant problems.

The size of the GitLab is constantly growing and Meltano is adding to GitLabs capabilities, not subtracting. We've hired 2 very awesome Python developers for Meltano specifically. They each have tons of experience in the ELT space.

All this to say, that no one at GitLab has turned their eyes away from GitLab, it's the opposite. This business is here to help GitLab as our first customer. Rather than having GitLab struggle to get it's data tools together, and make business decisions based on that data, we've devoted a whole team to provide a solution while helping the community at the same time.

Post reply on HN