Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I improve our data infrastructure?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Ask HN: How do I improve our data infrastructure?

#101

Post by mremes »

remilouf wrote: post #95
mremes wrote: post #92

Earlier quoted context omitted.

Don’t worry, I’m having my battles convincing my clients (both business and DS/DEs) that this is a viable paradigm. Here’s a nice-looking z-value recipe by Silota that I just googled up: http://www.silota.com/docs/recipes/sql-z-score.html

Thanks! Do you have any tips on convincing people that SQL is a good paradigm?

I'd just go and write out the technical architecture, defining what are the inputs (the raw data) and what are the outputs (matrices for training, testing etc. etc.) on different intervals (usually, data scientists want the previous days' data processed into some format, A/B test results and such) and how are you going to instrument those transformations. It's not just SQL but the DB where that SQL would be run and orchestration (for example with Apache Airflow), and for concrete ETL tasks (nodes in a processing graph) using a combination of open-source modules (usually in Python) and Bash scripts.

It takes time to get experienced in explaining and mapping these things to the domain.

Re: Ask HN: How do I improve our data infrastructure?

#102

Post by mremes »

remilouf wrote: post #80
shadowmint wrote: post #31

It sounds like you already have an idea of what you want to do, but I think you should pause and think more deeply about what you have, vs. what you want. What I would want in your situation is: - All the data in one place. - An easy way to explore the data. - A single source of truth for transformed data. - Metadata to explain the data model (ie. documentation). What you're proposing does some of those things, but i…

You changed my perspective a little bit by asking the right questions. > Moves from an architecture that is clustered for scale (ie. spark) to one that only scales vertically I did a quick estimate of the volume, and we won't reach 1Tb before > 5 years. We're not in a line of business where the number of clients can increase dramatically so it's fairly predictable. I don't want to design for imaginary scaling issues.…

I disagree with the author of the parent comment in regards of using SQL and using Spark instead. I actually first wrote my "SQL advocation" as a reply to this comment but decided to leave leave this view for what it is and write my own "rant" against complicating "big" data transformations with Spark or EMR (Hadoop Pig) or vendor-locked Spark-instrumentations like AWS Glue.

But I agreed with the parent comment's author about pretty much anything until the third bullet point of the second list. I'd like to get more reasoning behind his SQL hate.

Re: Ask HN: How do I improve our data infrastructure?

#103

Post by qxmat »

remilouf wrote: post #74
qxmat wrote: post #20

We have a similar issue. Generally speaking, your work must fit within a value stream - that is, to support your job/function you must do something that's rewarding (someone's got to pay the bills). There are a lot of interesting agile principles at play here but ultimately they revolve around delivery which must occur regardless of how complete, incomplete, fast, slow and/or viable your data is. Delivery is valuable…

Thank you for your answer, it was eye-opening. The story is that they opened that department a year ago and needed buy-in quickly so needed to jump-start it by hiring consultants. Whether hiring consultants was a good idea or not, I understand how things ended up where there are. A mix of inexperience and huge pressure to deliver. Now that we have buy in the pressure isn't so bad. I have a startup background and this…

Your manager sounds great.

Once you're able to outline the delivery/business/product case consider getting your QA function to champion (author?) the technical solution with you.

It's good to separate the two initially (a well documented problem leads to more options than a quick fix) but at some point you'll want to engineer buy-in. What you've described will almost certainly be a pain point for testers - this makes them great advocates for change and grants QA much needed ownership.

Feel free to dm me if you want to discuss this further.

Re: Ask HN: How do I improve our data infrastructure?

#104

Post by aarbor989 »

remilouf wrote: post #100
sologoub wrote: post #72

Earlier quoted context omitted.

That’s the beauty of BQ - it scales well, but it works just fine in smaller use cases. It doesn’t get simpler than SQL. Another item to consider is that BQ now has ML (simpler) models built in, further reducing the complexity of your pipeline: https://cloud.google.com/bigquery/docs/bigqueryml-intro If you are not on GCP, then I’d consider AWS Athena for querying the parquet files, but you still have to structure thes…

I will consider that. How about Redshift?

We had Redshift for our 23TB+ dataset and it worked great. The downside is it can get pricy, so do a cost analysis before you commit. Also know that views in redshift are not materialized so it’s more efficient to create physical tables of the views - which then adds maintenance overhead. The last thing I’ll add is that you’ll need to experiment with compression settings for your data. For us, a combination of ZSTD and bytedict was all we needed

Re: Ask HN: How do I improve our data infrastructure?

#106

Post by aeroevan »

riku_iki wrote: post #56
aeroevan wrote: post #51

Earlier quoted context omitted.

It's 100GB compressed. Parquet does a very good job of compressing most data so that's where the estimate of 10x (so 1TB) uncompressed was mentioned as a rule of thumb. Parquet also supports much better access mechanisms, like being able to deserialize a single column without having to read in entire rows. But like you mentioned, 1TB of data in a traditional database isn't that bad.

you can run sql DB over compressed filesystem, and some DBs allow you to compress tables too > like being able to deserialize a single column without having to read in entire rows. and it reads filesystem's whole page anyway

Sorry for the late reply, but parquet is a columnar format so if it's big enough data, you should have multiple pages/blocks of data in a single column for a specific row group, and then be able to seek to the next row group and sequentially read the next set of blocks.

Re: Ask HN: How do I improve our data infrastructure?

#107

Post by kwillets »

eVoLInTHRo wrote: post #79

Like others have said, the technical infrastructure is usually a manifestation of the people processes of the corporation. I think it's valuable to kinda ignore the technical stuff initially, and instead first understand the requirements of your customers. It's totally possible that the current system, as weird as it is, might satisfy your customers' requirements best. Unlikely, but it's possible. But given that they…

99% perspiration, 1% inspiration, and the rest is ML.

Re: Ask HN: How do I improve our data infrastructure?

#108

Post by kwillets »

lgsilver wrote: post #41

Really frustrating. I went through this process recently. The data was a couple orders of magnitude bigger and so I tend to agree that maybe just straight to Redshift / Bigquery would probably work best, but here were our steps: 1.) Insure that ingestion / S3 jobs were stabilized (in our case, the legacy were in Informatica, and maintenance took up all the teams' time). We moved to Luigi for this, but Airflow is grea…

Mode missed the boat by not including ETL capabilities. I worked for a time on the tool that Mode was based on, and its ETL capability was the hidden hand that got Data Scientists to build and maintain the data pipeline.

Re: Ask HN: How do I improve our data infrastructure?

#109

Post by sreekanthr »

remilouf wrote: post #73
sreekanthr wrote: post #61

I cannot answer your question without full understanding of what is the current usage of your data infrastructure. Few pointers - Who are users of the platform? If it is only used by data science team then you can rip apart the solution and work towards a more logical infrastructure where all you are doing is cleansing, normalizing and deriving features and these become your central feature repository which your team…

Thank you for taking the time to answer thoroughly ! > If it is only used by data science team then you can rip apart the solution and work towards a more logical infrastructure where all you are doing is cleansing, normalizing and deriving features and these become your central feature repository which your team can pull and build models. It is only used by data scientists. What do you mean by a feature repository?…

For your problem, I would suggest you to take a look at streamsets. They have an ETL plus data drift system in place, which is really interesting.

Ref: https://streamsets.com/

>Besides the brittleness of the process, I found that people are reluctant to analyse the data because it takes an unreasonable amount of time.

Is this because of the bad queries or way the data is organized?

>It is only used by data scientists. What do you mean by a feature repository? How would you organize it so people can push new features? This sounds very interesting.It is only used by data scientists. What do you mean by a feature repository? How would you organize it so people can push new features? This sounds very interesting.

Can you take a look at Feast by Go-Jek: https://github.com/gojek/feast There are similar projects by different big players in market, this should get you started on idea which I was talking about.

PS: Sorry, was traveling that is why there was a delay in answering your question.

Re: Ask HN: How do I improve our data infrastructure?

#110

Post by royalharsh95 »

We built a data warehousing pipeline which handles 40 million events per day and we use BigQuery. We don't use any other tools like Kafka or Apache Beam. But we use redis very extensively. It's simple system but handles everything. You don't require a complex solution always. But yes you can minimize the complexity with tools like Kafka.
Post reply on HN