Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

261–270 of 475 posts

Re: Big data is dead (2023)

#261

Post by hipadev23 »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Huh? How are you proposing loading a 6TB CSV into memory multiple times? And then processing with awk, which generally streams one a line at a time.

Obviously we can get boxes with multiple terabytes of RAM for $50-200/hr on-demand but nobody is doing that and then also using awk. They’re loading the data into clickhouse or duckdb (at which point the ram requirement is probably 64-128GB)

I feel like this is an anecdotal story that has mixed up sizes and tools for dramatic effect.

Re: Big data is dead (2023)

#262

Post by palata »

hnfong wrote: post #257
whamlastxmas wrote: post #234

Earlier quoted context omitted.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

I see it more like "it's 11pm and a family member suddenly wants to eat a steak at home, what would you do?" The person who says "I'm going drive back to the restaurant and take my professional equipment home to cook the steak" is probably offering the wrong answer. I'm obviously not a professional cook, but presumably the ability to improvise with whatever tools you currently have is a desirable skill.

Hmm I would say that the equivalent to your 11pm question is more something like "your sister wants to backup her holiday pictures on the cloud, how do you design it?". The person who says "I ask her 10 millions to build a data center" is probably offering the wrong answer :-).

Re: Big data is dead (2023)

#263

Post by hnfong »

fijiaarone wrote: post #113
__alexs wrote: post #85

Earlier quoted context omitted.

A moderately powerful desktop processor has memory bandwidth of over 50TB/s so yeah it'll take a couple of minutes sure.

The slow part of using awk is waiting for the disk to spin over the magnetic head. And most laptops have 4 CPU cores these days, and a multiprocess operating system, so you don’t have to wait for random access on a spinning plate to find every bit in order, you can simply have multiple awk commands running in parallel. Awk is most certainly a better user interface than whatever custom BrandQL you have to use in a tex…

I haven't been using spinning disks for perf critical tasks for a looong time... but if I recall correctly, using multiple processes to access the data is usually counter-productive since the disk has to keep repositioning its read heads to serve the different processes reading from different positions.

Ideally if the data is laid out optimally on the spinning disk, a single process reading the data would result in a mostly-sequential read with much less time wasted on read head repositioning seeks.

In the odd case where the HDD throughput is greater than a single-threaded CPU processing for whatever reason (eg. you're using a slow language and complicated processing logic?), you can use one optimized process to just read the raw data, and distribute the CPU processing to some other worker pool.

Re: Big data is dead (2023)

#264

Post by breckognize »

99.9%+ of data sets fit on an SSD, and 99%+ fit in memory. [1]

This is the thesis for https://rowzero.io. We provide a real spreadsheet interface on top of these data sets, which gives you all the richness and interactivity that affords.

In the rare cases you need more than that, you can hire engineers. The rest of the time, a spreadsheet is all you need.

[1] I made these up.

Re: Big data is dead (2023)

#265

Post by pdimitar »

apwell23 wrote: post #252
pdimitar wrote: post #249

Earlier quoted context omitted.

The classic putting words in people's mouths technique it is then. The good old straw man. If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]". I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.

> that people don't try simpler and proven solutions FIRST Well why don't people do that according to you ? Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.

I don't know why and this is why I said it's mind-blowing. Because to me trying stuff that can work on most laptops comes naturally in my head as the first viable solution.

As for interviews, sure, they have all sorts of traps. It really depends on the format and the role. Since I already disclaimed that I am not actual data scientist and just a seasoned dev who can make some magic happen without a dedicated data team (if/when the need arises) then I wouldn't even be in a data scientist interview in the first place. ¯\_(ツ)_/¯

Re: Big data is dead (2023)

#266

Post by palata »

kmarc wrote: post #209
ufo wrote: post #186

Earlier quoted context omitted.

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Sure, but as you said yourself: it's a trick question. How often does the employee have to answer trick questions without having any time to think in the actual job?

As an interviewer, why not asking: "how would you do that in a setup that doesn't have much data and doesn't need to scale, and then how would you do it if it had a ton of data and a big need to scale?". There is no trick here, do you feel you lose information about the interviewee?

Re: Big data is dead (2023)

#267

Post by pdimitar »

zaphar wrote: post #215
apwell23 wrote: post #206

Earlier quoted context omitted.

> Your entire post reeks of "I'm smarter than you" unlike "blows my mind" ? > As stated the question didn't require any of what you outline here. Right. OP mentioned it was "tricky question" . What makes it tricky is that all those attributes are implicitly assumed. I wouldn't interview at google and tell them my "stack" is "load it on your laptop". I would never say that in an interview even if I think that's the ri…

"blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not. You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.

Honest question: was "blows my mind" so offensive? Thought it was quite obvious I meant that "it blows my mind people don't try the simpler stuff first, especially having in mind that it works for much bigger percentage than cloud providers would have you believe"?

I guess it wasn't but even if so, it would be legitimately baffling how people manage to project so much negativity in three words that are slightly tongue-in-cheek casual comment on the state of affairs in an area whose value is not always clear (in my observations, only after you start having 20+ data sources it starts to pay off to have dedicated data team; I've been in teams only 3-4 devs and we still managed to have 15-ish data dashboards for the executives without too much cursing).

An anecdote, surely, but what isn't?

Re: Big data is dead (2023)

#269

Post by dartos »

coffeebeqn wrote: post #260
jancsika wrote: post #247

Earlier quoted context omitted.

That's great, but it's really just desiderata about you and your personal situation. E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that da…

This is the crux of it. Another interviewer would’ve marked “run on a local machine with a big SSD” - as: this fool doesn’t know enough about distributed systems and just runs toy projects on one machine

That is what I think interviewers think when I don’t immediately bring up kubernetes and sqs in an architecture interview

Re: Big data is dead (2023)

#270

Post by the_real_cher »

coldtea wrote: post #196
the_real_cher wrote: post #155

Earlier quoted context omitted.

I mean if you're doing data science the data is not always organized and of course you would want multi-processing. 1 TB of memory is like 5 grand from a quick Google search then you probably need specialized motherboards.

> I mean if you're doing data science the data is not always organized and of course you would want multi-processing Not necessarily - I might not want it or need it. It's a few TB, it can be on a fast HD, on an even faster SSD, or even in memory. I can crunch them quite fast even with basic linear scripts/tools. And organized could just mean some massaging or just having them in csv format. This is already the same…

Yeah I agree about over engineering.

Im just trying to understand the parent to my original comment.

How would running awk for analysis on 6TB of data work quickly and efficiently?

They say it would go into memory but its not clear to me how that would work as would still have paging and thrashing issues if the data didnt have often used sections of the data.

am I overthinking it and they were they just referring to buying a big ass Ram machine?

Post reply on HN