Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

321–330 of 475 posts

Re: Big data is dead (2023)

#321

Post by data-ottawa »

kjkjadksj wrote: post #141
mattbillenstein wrote: post #76

Earlier quoted context omitted.

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

Hes hiring data scientists not building a service though. This might realistically be a one off analysis for those 6tb. At which point you are happy your data scientists has returned statistical information instead of spending another week making sure the pipeline works if someone puts a greek character in a field.

Even if I'm doing a one off, depending on the task it can be easier/faster/more reliable to load 6TiB into a big query table than waiting hours for some task to complete and fiddling with parallelism and memory management.

It's a couple hundred bucks a month and $36 to query the entire dataset, after partitioning thats not terrible.

Re: Big data is dead (2023)

#322

Post by citizenpaul »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

The funny thing is that is exactly the place I want to work at. I've only found one company so far and the owner sold during the pandemic. So far my experience is that amount of companies/people that want what you describe is incredibly low.

I wrote a comment on here the other day that some place I was trying to do work for was using $11k USD a month on a BigQuery DB that had 375MB of source data. My advice was basically you need to hire a data scientist that knows what they are doing. They were not interested and would rather just band-aid the situation for a "cheap" employee. Despite the fact their GCP bill could pay for a skilled employee.

As I've seen it for the last year job hunting most places don't want good people. They want replaceable people.

Re: Big data is dead (2023)

#323

Post by drubio »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

It's almost a law "all technical discussions devolve into interview mind games", this industry has a serious interview/hiring problem.

Re: Big data is dead (2023)

#324

Post by [deleted] »

__alexs wrote: post #85
mattbillenstein wrote: post #76

Earlier quoted context omitted.

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

A moderately powerful desktop processor has memory bandwidth of over 50TB/s so yeah it'll take a couple of minutes sure.

[deleted]

Re: Big data is dead (2023)

#325

Post by bewaretheirs »

Androider wrote: post #203
fijiaarone wrote: post #113

Earlier quoted context omitted.

The slow part of using awk is waiting for the disk to spin over the magnetic head. And most laptops have 4 CPU cores these days, and a multiprocess operating system, so you don’t have to wait for random access on a spinning plate to find every bit in order, you can simply have multiple awk commands running in parallel. Awk is most certainly a better user interface than whatever custom BrandQL you have to use in a tex…

> The slow part of using awk is waiting for the disk to spin over the magnetic head. If we're talking about 6 TB of data: - You can upgrade to 8 TB of storage on a 16-inch MacBook Pro for $2,200, and the lowest spec has 12 CPU cores. With up to 400 GB/s of memory bandwidth, it's truly a case of "your big data problem easily fits on my laptop". - Contemporary motherboards have 4 to 5 M.2 slots, so you could today buil…

You can go further.

There are relatively cheap adapter boards which let you stick 4 M.2 drives in a single PCIe x16 slot; you can usually configure a x16 slot to be bifurcated (quadfurcated) as 4 x (x4).

To pick a motherboard at quasi-random:

Tyan HX S8050. Two M.2 on the motherboard.

20 M.2 drives in quadfurcated adapter cards in the 5 PCIe x16 slots

And you can connect another 6 NVMe x4 devices to the MCIO ports.

You might also be able to hook up another 2 to the SFF-8643 connectors.

This gives you a grand total of 28-30 x4 NVME devices on one not particularly exotic motherboard, using most of the 128 regular PCIe lanes available from the CPU socket.

Re: Big data is dead (2023)

#326

Post by pdimitar »

zaphar wrote: post #314
pdimitar wrote: post #267

Earlier quoted context omitted.

Honest question: was "blows my mind" so offensive? Thought it was quite obvious I meant that "it blows my mind people don't try the simpler stuff first, especially having in mind that it works for much bigger percentage than cloud providers would have you believe"? I guess it wasn't but even if so, it would be legitimately baffling how people manage to project so much negativity in three words that are slightly tongu…

I generally don't find that sort of thing offensive when combined with useful alternative approaches like your post provided. However the phrase does come with a connotation that you are surprised by a lack of knowledge or skill in others. That can be taken as smug or elitist by someone in the wrong frame of mind.

Thank you, that's helpful.

Re: Big data is dead (2023)

#327

Post by layer8 »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

You shouldn’t guess what they expect, you should say what you think is right, and why. Do you want to work at a company where you would fail an interview due to making a correct technical assessment? And even if the guess is right, as an interviewer I would be more impressed by an applicant that will give justified reasons for a different answer than what I expected.

Re: Big data is dead (2023)

#328

Post by palata »

hirsin wrote: post #320
palata wrote: post #266

Earlier quoted context omitted.

Sure, but as you said yourself: it's a trick question. How often does the employee have to answer trick questions without having any time to think in the actual job? As an interviewer, why not asking: "how would you do that in a setup that doesn't have much data and doesn't need to scale, and then how would you do it if it had a ton of data and a big need to scale?". There is no trick here, do you feel you lose infor…

Trick questions (although not known as such at the time) are the basis of most of the work we do? XY problem is a thing for a reason, and I cannot count the number of times my teams and I have ratholed on something complex only to realize we were solving for the wrong problem, i.e. A trick question. As a sibling puts it though, it's a matter of level. Senior/staff and above? Yeah, that's mostly what you do. Lower tha…

> are the basis of most of the work we do?

I don't know about you, but in my work, I always have more than 3 seconds to find a solution. I can slowly think about the problem, sleep on it, read about it, try stuff, think about it while running, etc. I usually do at least some of those for new problems.

Then of course there is a bunch of stuff that is not challenging and for which I can start coding right away.

In an interview, those trick questions will just show you who already has experience with the problem you mentioned and who doesn't. It doesn't say at all (IMO) how good the interviewee is at tackling challenging problem. The question then is: do you want to hire someone who is good at solving challenging problems, or someone who already knows how to solve the one problem you are hiring them for?

Re: Big data is dead (2023)

#329

Post by nairboon »

maayank wrote: post #33
zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

What do you mean by 'variety'?

it doesn't fit the relational model, e.g. you have some tables, but also tons of different types of images, video, sounds, raw text, etc.

Re: Big data is dead (2023)

#330

Post by geraldwhen »

kmarc wrote: post #66
mcny wrote: post #63

Earlier quoted context omitted.

I think that’s the way we were taught in college / grad school. If the premise of the class is relational databases, the professor says, for the purpose of this course, assume the data does not fit in memory. Additionally, assume that some normalization is necessary and a hard requirement. Problem is most students don’t listen to the first part “for the purpose of this course”. The professor does not elaborate becaus…

FWIW if they were juniors, I would've continued the interview and direct them with further questions, and observer their flow of thinking to decide if they are good candidates to pursue further. But no, this particular person had been working professionally for decades (in fact, he was much older than me).

Yeah. I don’t even bother asking juniors this. At that level I expect that training will be part of the job, so it’s not a useful screener.
Post reply on HN