Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

351–360 of 475 posts

Re: Big data is dead (2023)

#351

Post by xLaszlo »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

6TB - Snowflake

Why?

That's the boring solution. If you don't have a use case, what kind of queries you would run then opt for maximum flexibility with the minimum setup of a managed solution.

If cost is prohibitive on the long run, you can figure out a more tailored solution based on the revealed preferences.

Fiddling with CSVs is the DWH version of the legendary "Dropbox HN commenter".

Re: Big data is dead (2023)

#352

Post by vegetablepotpie »

newaccount74 wrote: post #31
davedx wrote: post #19

Earlier quoted context omitted.

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all. I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.

The flip side of that is that you end up with spaghetti code that is expensive to add features to and is expensive to clean up when you are successful. Then people in the business implement workarounds to handle special cases that are undocumented and hidden.

Re: Big data is dead (2023)

#353

Post by andoando »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

Its great if the interviewer actually takes time to sort out the questions you have, cause seemingly simple questions to you have a lot of assumptions you made.

I had an interview "design an app store". I tried asking, ok an app store has a ton of components, which part of the app store are you asking exactly? The response I got was "Have you ever used an app store? Design an app store". Umm ok.

Re: Big data is dead (2023)

#354

Post by oivey »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

Engineering for scalability here is the single server solution that you throw away later when scale is needed. The price is so small (in this case) for the simple solution that you should basically always start with it.

Re: Big data is dead (2023)

#355

Post by Dylan16807 »

whamlastxmas wrote: post #234
kmarc wrote: post #209

Earlier quoted context omitted.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

The fancy cluster is probably slower for most tasks than one big machine storing everything in RAM. It's not like a fast food burger.

Re: Big data is dead (2023)

#356

Post by Dylan16807 »

xethos wrote: post #300
crowcroft wrote: post #230

Earlier quoted context omitted.

I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.

Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either). [0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... [1] https://www.tomshardware.com/reviews/ram-speed…

Several gigabytes per second, plus RAM caching, is probably enough though. Latency can be very important, but there exist some very low latency enterprise flash drives.

Re: Big data is dead (2023)

#357

Post by Dylan16807 »

randomtoast wrote: post #118
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

No, if you want SQL you install postgresql on the single machine.

Why would use use bigquery just to get SQL?

Re: Big data is dead (2023)

#358

Post by esafak »

kjkjadksj wrote: post #143
randomtoast wrote: post #118

Earlier quoted context omitted.

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

For someone who is comfortable with sql we are talking minutes to hours to figure out awk well enough to see how its used or use it.

I have been using sql for decades and I am not comfortable with awk or intend to become so. There are better tools.

Re: Big data is dead (2023)

#359

Post by beryilma »

Renaud wrote: post #223
thesz wrote: post #188

Earlier quoted context omitted.

Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…

Parquet is not a database, it's a storage format that allows efficient column reads so you can get just the data you need without having to parse and read the whole file. Most tools can run queries across parquet files. Like everything, it has its strengths and weaknesses, but in most cases, it has better trade-offs over CSV if you have more than a few thousand rows.

> Parquet is not a database.

This is not emphasized often enough. Parquet is useless for anything that requires writing back computed results as in data used by signal processing applications.

Re: Big data is dead (2023)

#360

Post by kmarc »

SkipperCat wrote: post #347
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

That makes total sense if you're archiving the data, but what happens when you want to have 10,000 people have access to read/update the data concurrently. Then you start to need some fairly complex solutions.

This thread blew up a lot, and some unfriendly commenters made many assumptions about this innocent story.

You didn't, and indeed you have a point (missing specification of expected queries), so I expand it as a response here.

Among the MANY requirements I shared with the candidate, only one was the 6TiB. Another one was that it was going to be serving as part of the backend of an internal banking knowledge base, with at maximum 100 request a day (definitely not 10k people using it).

To all the upset data infrastructure wizards here: calm down. It was a banking startup, with an experimental project, and we needed the sober thinker generalist, who can deliver solutions to real *small scale* problems, and not the one who was the winner on the buzzword bingo.

HTH.

Post reply on HN