Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab is working on a tool just for data teams

about.gitlab.com

Post reply on HN
Print view View original

41–50 of 98 posts

Re: GitLab is working on a tool just for data teams

#41

Post by chasewright »

slap_shot wrote: post #11
hn_throwaway_99 wrote: post #3

Be interested to know all the competitors in this space. https://data.world/ is one I am most familiar with.

This projects competes with too many industries to really give a succinct answer, but here's just Extraction/Loading and Analyze: Extraction/Loading Dell Boomi SAP SAS Pentaho Domo Oracle IBM Microsoft Informatica Talend JitterBit SnapLogic Mulesoft SyncSort Information Builders Actian Attunity Datameer Alteryx Striim Treasure Data Cask StreamSets Snowplow DataTorrent Astronomer Panoply Apache Nifi Stitch Data FlyDat…

slap_shot, I agree and as I disclaimer I also work at GitLab. There is no shortage of data tools in the space today. A majority of my career has been spent in the data & analytics space and I've talked / worked with at least 60% of the companies you mentioned. At the end of the day, these are the questions I've asked over and over again.

1. Do we have enough money / budget for a tool like this? 2. Can we derive enough insights from this product fast enough to make a good ROI? 3. Does this tool use a proprietary language that no one wants to learn or can I code in a language that is relevant? 4. In all honesty, can I get insights faster in a spreadsheet than these tools? 5. What is the learning curve? 6. Can I answer the business question that was originally asked?

Open to more discussions around the topic as it is a lot harder to answer than a few philosophical questions, but it certainly resonates with many data & analytics professionals. A nice goal would be to have project where you can stand up a business, turn your data pipelines on, ingest the data, and view the insights needed to make a business decision all within a short timeframe of when a business goes live.

Re: GitLab is working on a tool just for data teams

#42

Post by sytse »

slap_shot wrote: post #39
jakecodes wrote: post #25

Earlier quoted context omitted.

What a great interview. @slap_shot, you had great questions and you are so well spoken. Really appreciate the feedback. We're all taking notes here. Hope you will keep an eye on our issue tracker for Meltano and give us your feedback as things come up.

My pleasure! Will watch the repo closely and contribute if I can. Very curious to see where this goes. Also, I had a coffee at 5PM with someone, which is way too late to be drinking coffee, and it is evident in how quickly I'm talking >.<

I liked the fast pace! But normally I do watch YouTube on 150% :) thanks for the chat

Re: GitLab is working on a tool just for data teams

#43

Post by hunter23 »

cheghook wrote: post #14

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to fo…

Agreed - our company moved to Gitlab about the same time (2.5 years ago) and it's very clear from their updates that their focus has splintered in different directions. Our company has recently moved to more Microsoft products so I am pushing our CTO to move to Github.

If the CEO is following this, please improve basic user stories like:

* As a user, I want to easily know who has approved my merge request. Note the word "easily". The UI lists the people who did not approve next to label "Approved" and the people who did approve next to the label "Approved by". Makes absolutely no sense

* As a user, I want to see all the merge requests that I need to review because I am listed as an approved (it boggles my mind that this doesn't exist)

* As a user, I want to be notified by todos that only have any pending actions on them

* As a user, I want to disapprove a merge request

There are so many basic areas of the core product that are almost unusable. All of our engineers who have to regularly switch between github and gitlab prefer the github ui.

Re: GitLab is working on a tool just for data teams

#44

Post by mycelium »

slap_shot wrote: post #19
sytse wrote: post #17

Earlier quoted context omitted.

"but then try to tackle something like integrations or BI tools. The complexity of both of those is massive and there are great open source efforts already out there." I would love to hear your suggestion for a great open source BI tool. We tried Superset and Metabase but both didn't came close to what we could do with Looker. That is why we're giving Meltano Analyze a shot. BTW Do you want to do a livestreamed video…

Sure - just shot you an email at website@yourhandle.com

Thanks to both of you for your time doing that discussion!

@slap_shot and anyone else — I'm curious if you have thoughts on, or even have heard of the Ballerina language? It's a programming language for doing data integration work, built by the ESB/integration consultancy WSO2. It seems to have a lot of eng resources sunk into it but surprisingly little fanfare.

The CEO's interview with the Software Engineering Daily podcast was great: https://softwareengineeringdaily.com/2018/07/12/ballerina-la...

The language site tends toward buzzword-salad, but clearly has had a lot of love and thought put into it: https://ballerina.io/philosophy/

Re: GitLab is working on a tool just for data teams

#46

Post by georgewfraser »

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is:

  - Studying each source to figure out the right data model
  - Chasing down a million weird corner cases
  - Working around dumb bugs in the data sources
This is the kind of problem where paying for software really works better. When people build data pipelines in-house, they tend to hack at it until it works for their use case and then stop. When we build data pipelines, we map out every feature of the data source, implement the whole thing at once, and then put it through a beta period with multiple real users. This is easy to do when you have a tight-knit dev team; much harder for a group of part-time open-source contributors.

Re: GitLab is working on a tool just for data teams

#48

Post by MechanicalTwerk »

georgewfraser wrote: post #46

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is: - Studying each source to figure out the right data model - Chasing down a million weird corner cases - Working around dumb bugs in the data sources This is the kind of problem where paying for software really works better. When people build data pip…

I kind of agree with this. To take an example outside of ETL/DW/BI, when I first saw Zapier I was skeptical of how many APIs they could support because I'd seen a decent amount of open source ESBs like Mulesoft run out of steam after a certain number of connectors. Zapier, being proprietary from day one (albeit less featureful than a full blown enterprise ESB) has done better than I expected. Still, they only support 100 or so datasources and the types of data/objects/triggers/whatever they support is limited at times. IMO at some point both open source and proprietary models fall apart in the face of the long tail. Amazon has tackled the long tail of ecommerce but that's an enormous market that allows them to employ hundreds of thousands of people to tackle that long tail. Tackling the long tail of connectors (whether it's for ESBs/SaaS integration or ETL/DW/BI) is just too expensive compared to the size of the markets that are willing to take a shot at it.

Re: GitLab is working on a tool just for data teams

#49

Post by cheghook »

sytse wrote: post #18
cheghook wrote: post #14

I can't understand why GitLab thinks they have to embark on a new project every so often instead of focusing on their current product and features. There is just a lot to work on, so many of the current features/products are half assed. At my place we moved to GitLab 2.5 years ago and updates where smoother back then but the past few months we had to hire a new sys admin for our build machines and GitLab server to fo…

I'm sorry to heard your experience with GitLab hasn't been smooth. We have more people then ever working on the core of GitLab. And the number of reported issues per customer are going down. But every problem is one too many. Please email me at sytse@gitlab.com if you're open to a call about your situation.

> And the number of reported issues per customer are going down.

This doesn't mean anything, maybe the customers are simply tired of reporting issues. For example last year we didn't do any updates for 6 months because we were afraid it'd break something and we were too busy to be willing to spend the time reporting problems.

We also don't report issues that are already open on gitlab.com, reporting the issue means your customer is willing to spend time reporting, following up and testing your bug. This is your job, not the customer's. At the moment we are only reporting issues that are either blocking us from work or slowing down our development. The majority of issues we are facing are performance problems.

I just wrote a script to plot the number of issues on gitlab-ce over time and percentage of open/close issues, and the overall period they have been open for, you are accumulating issues with: `backend`, `UX`, `technical debt`, `performance`, `CI/CD`, ... labels, a lot of them don't have a Milestone and have been open for a long time.

I am not sure how emailing you would help us, it's not like the problems are not reported or you don't already know about them. It just appears that the priority of GitLab, as a company, is not shipping a quality product anymore.

EDIT: I work in the aerospace industry and one of the stages of our pipelines is to run stress test on our product. I would suggest you to run a stress test on an instance of GitLab, this would be an amazing place to start looking for performance problems.

Re: GitLab is working on a tool just for data teams

#50

Post by js8 »

georgewfraser wrote: post #46

Data pipelines are not a great subject for an open-source project. We've been building these for the last 3+ years at Fivetran, and I can tell you that the challenge is: - Studying each source to figure out the right data model - Chasing down a million weird corner cases - Working around dumb bugs in the data sources This is the kind of problem where paying for software really works better. When people build data pip…

Can't agree more.

IMHO, if you want to make a dent in the space, figure out better debugging tools!

In particular, tools that explain how a certain (specific) value was calculated in the system, tools that let you bisect the source data in some way and let you focus on the source data that are likely to have a problem, tools that help you figure out that certain intermediate value in calculations is an outlier, tools that let you test certain assumptions about data over the whole pipeline..

Post reply on HN