Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

451–460 of 475 posts

Re: Big data is dead (2023)

#451

Post by edogrider »

snakeyjake wrote: post #344
zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

>Big Data today is not: I don't have enough storage or compute. It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities. That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices. We don't usually fill up the RAIDs so we gen…

Why not Minio? Yes, the network will still be the bottleneck but you can use whatever he works best and it is remarkably simple.

Re: Big data is dead (2023)

#452

Post by the_real_cher »

Terr_ wrote: post #397
the_real_cher wrote: post #172

Earlier quoted context omitted.

Well yeah streaming to the cloud to work around budget issues is a while nother convo haha.

I'm having flashbacks to some new outside-hire CEO making flim-flam about capex-vs-opex in order to justify sending business towards a contracting firm they happened to know.

Straight to jail

Re: Big data is dead (2023)

#454

Post by alentred »

mrtimo wrote: post #168
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Agreed. The abstractions on top of parquet are quite immature yet, though, and lots of software assumes that if you use Parquet - you also use Hive, Spark and stuff.

Take Apache Iceberg for example. It is essentially a specification to how to store parquet files for efficient use and exploration of data, but the only implementation... depends on Apache Spark!

Re: Big data is dead (2023)

#455

Post by orhmeh09 »

orhmeh09 wrote: post #430
fifilura wrote: post #427

Earlier quoted context omitted.

Possibly, but it seems like overkill for the type of analysis that the OP expected the interviewee to do with awk. SQL should be fine for that. Actually, I have a feeling that the awk solution will struggle if there are many unique keys. For example if they in that dataset have a million customers and want to extract the top 10. Then there is an intermediate map stage that will be storage or memory consuming. It is l…

I agree completely for this scale. I did want to point out that it's fairly easy these days to do the kinds of things one would do on a cluster, which I learned just a few months ago myself :)

quick addition: there are modules (eg cloudknot) for Python that make it possible to run a Python callable that launches an AWS Batch environment and job with a single method, which you could do anywhere that runs Python.

Re: Big data is dead (2023)

#456

Post by yourapostasy »

marcosdumay wrote: post #173
filleokus wrote: post #133

Earlier quoted context omitted.

I think I've written about it here before, but I imported ≈1 TB of logs into DuckDB (which compressed it to fit in RAM of my laptop) and was done with my analysis before the data science team had even ingested everything into their spark cluster. (On the other hand, I wouldn't really want the average business analyst walking around with all our customer data on their laptops all the time. And by the time you have a p…

> And by the time you have ... the Big Data Solution™ probably have a lower TCO... I doubt it. The common Big Data Solutions manage to have a very high TCO, where the least relevant share is spent on hardware and software. Most of its cost comes from reliability engineering and UI issues (because managing that "proper ACL" that doesn't fit your business is a hell of a problem that nobody will get right).

> ...managing that "proper ACL" that doesn't fit your business is a hell of a problem that nobody will get right...

I'm not sure there is a way to get this right unless there is a programmatic integration into the org chart, and ability to describe and parse in a declarative language the organizational rules of who has access to what, when, under what auth, etc. It has otherwise been for me an exercise in watching massive amounts of toil manually interpreting between the SOT of the org chart and all the other applications mediated by many manual approval policies and procedures. And at every client I've posed this to, I've always been denied that programmatic access for integration.

A lot of sites try to avoid this by designing ACL's around certain activity or data domains because those are more stable than organizations, but this breaks down when you get to the fine-grained levels of the ACL's so we get capped benefits from this approach.

I'd love to hear how others solve this in large (10K+ staff) organizations that frequently change around teams.

Re: Big data is dead (2023)

#457

Post by kentm »

antisthenes wrote: post #296
jancsika wrote: post #247

Earlier quoted context omitted.

That's great, but it's really just desiderata about you and your personal situation. E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that da…

> E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. If people in high stakes environments interpret hedging as a smell - run from that company as fast as you can. Hedging is a natural adult reasoning process. Do you really want to work with someone who doesn't understand that?

Agreed. I don't really understand the mindset of someone who would consider this sort of hedging a smell. A candidate being able to take vague requirements and clearly state different solutions for different scenarios is an excellent candidate. That would be considered a positive signal for myself and pretty much all the interviewers I've worked with.

Re: Big data is dead (2023)

#458

Post by yapret »

snakeyjake wrote: post #344
zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

>Big Data today is not: I don't have enough storage or compute. It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities. That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices. We don't usually fill up the RAIDs so we gen…

> We can only fly every other day because it takes a day between flights to offload the data via fiber onto storage servers

What’s preventing you from buying a second set of SSD drives and swapping them into planes end of day?

The planes would then be able to fly every day while the previous day’s data is being offloaded on the ground, condensing the two weeks into one. Swaps could even be quite easy if you replace full enclosures rather than each drive individually.

I'm probably missing something, though!

Re: Big data is dead (2023)

#459

Post by riku_iki »

swyx wrote: post #411
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

> They are 10x smaller than CSV. So 600GB instead of 6TB. how? lossless compression? under what scenario? vague headlines like this just beg more questions

Likely this assumes that parquet has internal compression applied, and CSV is uncompressed.

Re: Big data is dead (2023)

#460

Post by riku_iki »

seabass-labrax wrote: post #315
jjgreen wrote: post #288

Earlier quoted context omitted.

Please promote the use of .parquet files! apt-cache search parquet Maybe later

Parquet is a file format , not a piece of software. 'apt install csv' doesn't make any sense either.

apt search would return tons of libparquet-java/c/python packages if it was popular.
Post reply on HN