Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

211–220 of 475 posts

Re: Big data is dead (2023)

#211

Post by cheptsov »

A good clickbait title. One should credit the author for that.

As to the topic, IMO, there is a contradiction. The only way to handle big data is to divide it into chunks that aren’t expensive to query. In that sense, no data is "big" as long as it’s handled properly.

Also, about big data being only a problem for 1 percent of companies: it's a ridiculous argument implying that big data was supposed to be a problem for everyone.

I personally don’t see the point behind the article, with all due respect to the author.

I also see many awk experts here who have never been in charge of building enterprise data pipelines.

Re: Big data is dead (2023)

#212

Post by banku_brougham »

dventimi wrote: post #37

Question for the Big Data folks: where do sampling and statistics fit into this, if at all? Unless you're summing to the penny, why would you ever need to aggregate a large volume of data (the population) rather than a small volume of data (a sample)? I'm not saying there isn't a reason. I just don't know what it is. Any thoughts from people who have genuine experience in this realm?

There is a problem on website data, where new features are only touching a subset of customers and you need results for every single one.

You wont be partitioned for this case, but the compute you need is just filtering out this set.

But sampling wont get what you want especially if you are doing QC at the business team level about whether the CX is behaving as expected.

Re: Big data is dead (2023)

#213

Post by marginalia_nu »

thunky wrote: post #135
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

> requirements of "6 TiB of data" How could anyone answer this without knowing how the data is to be used (query patterns, concurrent readers, writes/updates, latency, etc)? Awk may be right for some scenarios, but without specifics it can't be a correct answer.

Those are very appropriate follow up questions I think. If someone tasks you to deal with 6 TiB of data, it is very appropriate to ask enough questions until you can provide a good solution, far better than to assume the questions are unknowable and blindly architect for all use cases.

Re: Big data is dead (2023)

#214

Post by apwell23 »

koverstreet wrote: post #174
apwell23 wrote: post #138

Earlier quoted context omitted.

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

this is how you know when someone takes themself too seriously buddy, you're just rolling off buzzwords and lording it over other people

buddy you suffer from NIH syndrome upset that no one wants your 'hacks'.

Re: Big data is dead (2023)

#215

Post by zaphar »

apwell23 wrote: post #206
zaphar wrote: post #156

Earlier quoted context omitted.

What about his answer prevents any of that? As stated the question didn't require any of what you outline here. ZFS will probably do a better job of protecting your data than almost any other filesystem out there so it's not a bad foundation to start with if you want to protect data. Your entire post reeks of "I'm smarter than you" smugness while at the same time revealing no useful information or approaches. Near as…

> Your entire post reeks of "I'm smarter than you" unlike "blows my mind" ? > As stated the question didn't require any of what you outline here. Right. OP mentioned it was "tricky question" . What makes it tricky is that all those attributes are implicitly assumed. I wouldn't interview at google and tell them my "stack" is "load it on your laptop". I would never say that in an interview even if I think that's the ri…

"blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not.

You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.

Re: Big data is dead (2023)

#216

Post by HelloNurse »

photonthug wrote: post #183
apwell23 wrote: post #138

Earlier quoted context omitted.

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with. No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment. You risk comi…

Abstractly, "safety, auditablity, access control, knowledge sharing" are about people reading and writing files: simplifying away complicated management systems improves security. The operating system should be good enough.

Re: Big data is dead (2023)

#217

Post by rgrieselhuber »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

This is a great test / question. More generally, it tests knowledge with basic linux tooling and mindset as well as experience level with data sizes. 6TiB really isn't that much data these days, depending on context and storage format, etc. of course.

Re: Big data is dead (2023)

#218

Post by marcosdumay »

apwell23 wrote: post #122
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

What kind of business just has a static set of 6TiB data that people are loading on their laptops. You tricked candidates with your nonsensical scenario. Hate smartass interviewers like this that are trying some gotcha to feel smug about themselves. Most candidates don't feel comfortable telling ppl 'just load on your laptops' even if they think thats sensible. They want to present a 'professional solution', esp when…

> What kind of business just has a static set of 6TiB data that people are loading on their laptops.

Most business have static sets of data that people load on their PCs. (Why do you assume laptops?)

The only weird part of that question is that 6TiB is so big it's not realistic.

Re: Big data is dead (2023)

#219

Post by 6510 »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I dont know anything but when doing that I always end up next Thursday having the same with 4TB and the next with 17 at which point I regret picking a solution that fit so exactly.

Re: Big data is dead (2023)

#220

Post by deadbabe »

doubloon wrote: post #160
deadbabe wrote: post #94

Something similar will happen with generative AI someday. AI scientists will propose all sorts of elaborate complex solutions to problems using LLMs, and the dismissive responsive will be “Your problem is solvable with a couple if statements.” Most people just don’t have problems that require AI.

Thats one of the first things Andrew Ng said in his old ML course

This is why I personally just can’t find motivation to even pay attention to most AI developments. It’s a toy, it does some neat things, but there’s no problem I’ve heard of or encountered where LLM style AI was the only tool for the job, or even the best tool. The main use seems to be content creation and manipulation at scale, which the vast majority of companies simply don’t have to deal with.

Similarly, a lot of companies talk about how they have tons of data, but there’s never any real application or game changing insight from it. Just a couple neat tricks and product managers patting themselves on the back.

Setting up a good database is probably the peak of a typical company’s tech journey.

Post reply on HN