Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

401–410 of 475 posts

Re: Big data is dead (2023)

#401

Post by boh »

rqtwteye wrote: post #244
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a system IT had built for us with Hadoop, API gateways, multiple developers and hundreds of thousands of yearly cost. I told him that at our scale (now and any foreseeable future) I could easily run the whole thing on a USB drive attached to his laptop and a few pyt…

That's the tech sector in a nutshell. Very few innovations actually matter to non-tech companies. Most companies could survive on Windows 98 software.

Re: Big data is dead (2023)

#402

Post by zeroq »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Reminds me an old story from Steve Yegge:

I gave him the "find the phone numbers in 50,000 html files" question, and he decided to write a huge program with an ad-hoc state machine. When I asked how long it would take to write the program, he said he'd have to hit N files, with M lines per file, so... I interrupted him: no, WRITE. How long to WRITE the program? Oh. He estimated it at 5 days of work. At this point I was 50% ready to throw him out.

Re: Big data is dead (2023)

#403

Post by gen220 »

1vuio0pswjnm7 wrote: post #399
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

"... or a $199 enterprise HDD" External or internal? Any examples? "... it could be loaded (multimple times) to memory" All 6TiB at once, or loaded in chunks?

https://diskprices.com yields https://www.amazon.com/dp/B0C363Y5BQ, fwiw. (16TB for $129.99 at time of writing)

Re: Big data is dead (2023)

#404

Post by gen220 »

apwell23 wrote: post #122
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

What kind of business just has a static set of 6TiB data that people are loading on their laptops. You tricked candidates with your nonsensical scenario. Hate smartass interviewers like this that are trying some gotcha to feel smug about themselves. Most candidates don't feel comfortable telling ppl 'just load on your laptops' even if they think thats sensible. They want to present a 'professional solution', esp when…

Maybe you don't realize, but a well-done interview isn't a test, it's a conversation. You can absolutely ask clarifying questions to help shape your answer (this is something I always remind people of in interviews, personally, before and after giving the question).

You don't get extra credit points for mind-reading, if anything you get more esteem for requirements-gathering, which would lead you towards either a professional solution or a laptop solution: whichever fits the business needs.

It might be a totally unreasonable question if it's provided context-free as a form on a screen, but it is a perfectly reasonable conversation-starter in an interview.

Re: Big data is dead (2023)

#405

Post by marginalia_nu »

ninkendo wrote: post #400
marginalia_nu wrote: post #86

Earlier quoted context omitted.

To be fair, mmap doesn't put your data in RAM, it presents it as though it was in RAM and has the OS deal with whether or not it actually is.

Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram. It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.

Really depends on the shape of the data. mmap can be suboptimal in many cases.

For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.

Re: Big data is dead (2023)

#406

Post by fuzztester »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

Architecting for scalability doesn't mean disregarding resource costs, though - rather, the opposite. In fact, resource costs are even more important at higher scale, because more resources cost more.

Re: Big data is dead (2023)

#407

Post by jmalicki »

bee_rider wrote: post #302
whamlastxmas wrote: post #234

Earlier quoted context omitted.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

I’m not sure if you are referencing it intentionally or not, but some chefs (Gordon Ramsey for one) will ask an interviewee to make some scrambled eggs; something not super niche or specialized but enough to see what their technique is. It is a sort of “interview hack” example that’s been used to emphasize the idea of a simple unspecialized skill-test that went around a while ago. I guess upcoming chefs probably prac…

This is sort of the chef equivalent of fizzbuzz or "reverse a binary tree" - there's no gimmicks, something "everyone" should know how to do, it's nothing fancy, just the basics of "can you iterate over data structures and write for loops competently" - or in this case "can you not under/over cook the eggs and can you deliver them in the style you say you're going to.

Re: Big data is dead (2023)

#408

Post by mewpmewp2 »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

On the other hand if salaries are at 300k then 10k compared to that is not a huge cost. If a scalable tool can make you even 10 percent more effective it would be worth 30k.

Re: Big data is dead (2023)

#409

Post by 1vuio0pswjnm7 »

gen220 wrote: post #403
1vuio0pswjnm7 wrote: post #399

Earlier quoted context omitted.

"... or a $199 enterprise HDD" External or internal? Any examples? "... it could be loaded (multimple times) to memory" All 6TiB at once, or loaded in chunks?

https://diskprices.com yields https://www.amazon.com/dp/B0C363Y5BQ , fwiw. (16TB for $129.99 at time of writing)

That 1st site is great. Many thanks.

Re: Big data is dead (2023)

#410

Post by sien »

rqtwteye wrote: post #244
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a system IT had built for us with Hadoop, API gateways, multiple developers and hundreds of thousands of yearly cost. I told him that at our scale (now and any foreseeable future) I could easily run the whole thing on a USB drive attached to his laptop and a few pyt…

The flipside of this is that people at some places are fully aware of this and get very suspicious of consultants offering to handle their 'big data' challenges with loads of highly proprietary stuff.

Data organisation can be more of an issue, but the general issue with this is often a lack of internal discipline on the data owners to carefully manage their data. But that isn't nearly as attractive for management. Putting 'brought in new cloud vendor / technology' looks better than 'improved data organisation' on a CV even if the new vendor was a waste of money.

Post reply on HN