Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

171–180 of 475 posts

Re: Big data is dead (2023)

#171

Post by EdwardDiego »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

If you were hiring me for a data engineering role and asked me how to store and query 6 TiB, I'd say you don't need my skills, you've probably got a Postgres person already.

Re: Big data is dead (2023)

#172

Post by the_real_cher »

fijiaarone wrote: post #125
coldtea wrote: post #102

Earlier quoted context omitted.

> How would six terabytes fit into memory? What device do you have in mind? I've seen places use 2TB RAM servers, and that was years ago, and it isn't even that expensive (can get those for about $5K or so). Currently HP allows "up to 48 DIMM slots which support up to 6 TB for 2933 MT/s DDR4 HPE SmartMemory". Close enough to fit the OS, the userland, and 6 TiB of data with some light compression. > It seems like it w…

“How would six terabytes fit into memory?” A better question would be: Why would anyone stream 6 terabytes of data over the internet? In 2010 the answer was: because we can’t fit that much data in a single computer, and we can’t get accounting or security to approve a $10k purchase order to build a local cluster, so we need to pay Amazon the same amount every month to give our ever expanding DevOps team something to…

Well yeah streaming to the cloud to work around budget issues is a while nother convo haha.

Re: Big data is dead (2023)

#173

Post by marcosdumay »

filleokus wrote: post #133
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I think I've written about it here before, but I imported ≈1 TB of logs into DuckDB (which compressed it to fit in RAM of my laptop) and was done with my analysis before the data science team had even ingested everything into their spark cluster. (On the other hand, I wouldn't really want the average business analyst walking around with all our customer data on their laptops all the time. And by the time you have a p…

> And by the time you have ... the Big Data Solution™ probably have a lower TCO...

I doubt it. The common Big Data Solutions manage to have a very high TCO, where the least relevant share is spent on hardware and software. Most of its cost comes from reliability engineering and UI issues (because managing that "proper ACL" that doesn't fit your business is a hell of a problem that nobody will get right).

Re: Big data is dead (2023)

#174

Post by koverstreet »

apwell23 wrote: post #138
pdimitar wrote: post #62

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

this is how you know when someone takes themself too seriously

buddy, you're just rolling off buzzwords and lording it over other people

Re: Big data is dead (2023)

#175

Post by acomjean »

mcny wrote: post #63
geraldwhen wrote: post #46

Earlier quoted context omitted.

I ask a similar question on screens. Almost no one gives a good answer. They describe elaborate architectures for data that fits in memory, handily.

I think that’s the way we were taught in college / grad school. If the premise of the class is relational databases, the professor says, for the purpose of this course, assume the data does not fit in memory. Additionally, assume that some normalization is necessary and a hard requirement. Problem is most students don’t listen to the first part “for the purpose of this course”. The professor does not elaborate becaus…

I took a Hadoop class. We learned hadoop and were told by the instructor we probably wouldn’t’t need it, and learned some other Java processing techniques (streams etc)

Re: Big data is dead (2023)

#176

Post by sph »

mrtimo wrote: post #168
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Third consecutive time in 86 days that you mention .parquet files. I am out of my element here, but it's a bit weird

Re: Big data is dead (2023)

#177

Post by emptyfile »

palata wrote: post #167
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

[dead]

Re: Big data is dead (2023)

#178

Post by tomrod »

apwell23 wrote: post #122
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

What kind of business just has a static set of 6TiB data that people are loading on their laptops. You tricked candidates with your nonsensical scenario. Hate smartass interviewers like this that are trying some gotcha to feel smug about themselves. Most candidates don't feel comfortable telling ppl 'just load on your laptops' even if they think thats sensible. They want to present a 'professional solution', esp when…

I disagree with your take. Your surly rejoinder aside, the parent commenter identifies an area where senior level knowledge and process appropriately assess a problem. Not every job interview is satisfying checklist of prior experience or training, but rather assessing how well that skillset will fit the needed domain.

In my view, it's an appropriate question.

Re: Big data is dead (2023)

#179

Post by pyrale »

mattbillenstein wrote: post #76
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

Once you understand that 6tb fits on a hard drive, you can just as well put it in a run-of-the-mill pg instance, which metabase will reference just as easily. Hell, metabase is fine with even a csv file...

Re: Big data is dead (2023)

#180

Post by fifilura »

sph wrote: post #176
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Third consecutive time in 86 days that you mention .parquet files. I am out of my element here, but it's a bit weird

FWIW I am the same. I tend to recommend BigQuery and AWS/Athena in various posts. Many times paired with Parquet.

But it is because it makes a lot of things much simpler, and that a lot of people have not realized that. Tooling is moving fast in this space, it is not 2004 anymore.

His arguments are still valid and 86 days is a pretty long time.

Post reply on HN