Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

361–370 of 475 posts

Re: Big data is dead (2023)

#361

Post by palata »

theamk wrote: post #343
palata wrote: post #328

Earlier quoted context omitted.

> are the basis of most of the work we do? I don't know about you, but in my work, I always have more than 3 seconds to find a solution. I can slowly think about the problem, sleep on it, read about it, try stuff, think about it while running, etc. I usually do at least some of those for new problems. Then of course there is a bunch of stuff that is not challenging and for which I can start coding right away. In an i…

If the interviewer expects you to answer entire design question in 3 seconds, that interview is pretty broken. Those questions should take longish time (minutes to tens of minutes), and should let candidate showcase their thought process.

I meant that the interviewer expects you to start answering after 3 seconds. Of course you can elaborate over (tens of) minutes. But that's very far from actual work, where you have time to think before you start solving a problem.

You may say "yeah but you just have to think out loud, that's what the interviewer wants". But again that's not how I work. If the interviewer wants to see me design a system, they should watch me read documentation for hours, then think about it while running, and read again, draw a quick thing, etc.

Re: Big data is dead (2023)

#362

Post by anon84873628 »

blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

>The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc.

I think that was exactly their point. If new architectures were actually necessary, we would have seen a greater rise in Mongo and the like. But we didn't, because the existing systems were perfectly adequate.

Re: Big data is dead (2023)

#364

Post by nostrademons »

jjgreen wrote: post #288
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Please promote the use of .parquet files! apt-cache search parquet Maybe later

It's more like "sudo pip install pandas" and then Pandas comes with Parquet support.

Re: Big data is dead (2023)

#365

Post by nostrademons »

ddalex wrote: post #185
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Why is .parquet better than protobuf?

Parquet ~= Dremel, for those who are up on their Google stack.

Dremel was pretty revolutionary when it came out in 2006 - you could run ad-hoc analyses in seconds that previously would've taken a couple days of coding & execution time. Parquet is awesome for the same reasons.

Re: Big data is dead (2023)

#366

Post by kwillets »

lokimedes wrote: post #49

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise,…

Back in the 80's and 90's NASA built a National Aerodynamic Simulator, which was a big Cray or similar that could crunch FEA simulations (probably a low-range graphics card nowadays). IIRC they found that the queue for that was as long or longer than it took to run jobs on cheaper hardware; MPP systems such as Beowulf grew out of those efforts.

Re: Big data is dead (2023)

#367

Post by nostrademons »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I would've said "Pandas with Parquet files". If you're hiring a DS it's implied that you want to do some sort of aggregate or summary statistics, which is exactly what Pandas is good for, while awk + shell scripts would require a lot of clumsy number munging. And Parquet is an order of magnitude more storage efficient than CSV, and will let you query very quickly.

Re: Big data is dead (2023)

#368

Post by citizen_friend »

nevi-me wrote: post #73
pdimitar wrote: post #62

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

To be fair on candidates, CLI programs create technical debt the moment they're written. A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.

Configuring debugged, optimized software, with a shell script is orders of magnitude cheaper than developing novel software.

Re: Big data is dead (2023)

#369

Post by citizen_friend »

randomtoast wrote: post #118
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

sqlite cli

Re: Big data is dead (2023)

#370

Post by citizen_friend »

SkipperCat wrote: post #347
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

That makes total sense if you're archiving the data, but what happens when you want to have 10,000 people have access to read/update the data concurrently. Then you start to need some fairly complex solutions.

This load is well handled by a Postgres instance and 15-25k thrown at hardware.
Post reply on HN