Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

371–380 of 475 posts

Re: Big data is dead (2023)

#371

Post by citizen_friend »

vegetablepotpie wrote: post #352
newaccount74 wrote: post #31

Earlier quoted context omitted.

The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all. I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.

The flip side of that is that you end up with spaghetti code that is expensive to add features to and is expensive to clean up when you are successful. Then people in the business implement workarounds to handle special cases that are undocumented and hidden.

No it doesn't. Simple and targeted solutions are not bad code. For example, start with a single postgres instance on a single machine, rather than Hadoop clusters and Kubernetes. Once that is maxed out, you will have time and money to solve bigger problems.

Re: Big data is dead (2023)

#372

Post by mrtimo »

sph wrote: post #176
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Third consecutive time in 86 days that you mention .parquet files. I am out of my element here, but it's a bit weird

I've downloaded many csv files that were mal-formatted (extra commas or tabs etc.), or had dates in non-standard formats. Parquet format probably would not have had these issues!

Re: Big data is dead (2023)

#373

Post by citizen_friend »

Spooky23 wrote: post #106
davedx wrote: post #19

Earlier quoted context omitted.

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

The exception is when you have people with skills in particular tools. The suggestion upthread to use awk is awesome if you’re a bunch of Linux grey beards. But if you have access to people with particular skills or domain knowledge… spending extra cash on silly infrastructure is (within reason) way cheaper than having that employee be less productive.

Nope, if every person does things completely differently, that's just a lack of technical leadership. Leaders pick an approach with tradeoffs that meet organizational goals and help their team to follow it.

Re: Big data is dead (2023)

#374

Post by thesz »

maxnevermind wrote: post #334
thesz wrote: post #188

Earlier quoted context omitted.

Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…

> 7.2 millions row groups Why would you need 7.2 mil row groups? Row group size when stored in HDFS is usually equal to HDFS bock size by default, which is 128MB 7.2 mil * 128MB ~ 1PB You have a single parquet file 1PB in size?

Parquet is not HDFS. It is a static format, not a B-tree in disguise like HDFS.

You can have compressed Parquet columns with 8192 entries being a couple of tens bytes in size. 600 columns in a row group is then 12K bytes or so, leading us to 100GB file, not a petabyte. Four orders of magnitude of difference between your assessment and mine.

Re: Big data is dead (2023)

#375

Post by citizen_friend »

threeseed wrote: post #29
brtkdotse wrote: post #11

Earlier quoted context omitted.

> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success

HN is the worst echo chamber around. Obsessed with this "you must use PostgreSQL for every use case" nonsense. And that anyone who actually has unique data needs is simply doing it for their resume or are over-engineering.

Nobody is saying this.

> who actually has unique data needs

We are saying this is probably not true, and you just want to play with toys rather than ship working systems.

Google search, cannot be built in Postgres.

Re: Big data is dead (2023)

#377

Post by atomicnumber3 »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

It's really hard because I've failed interviews by pitching "ok we start with postgres, and when that starts to fall over we throw more hardware at it, then when that fails we throw read replicas in, then we IPO, then we can spend all our money and time doing distributed system stuff".

Whereas the "right answer" (I had a man on the inside) was to describe some wild tall and wide event based distributed system. For some nominal request volume that was nowhere near the limits of postgres. And they didn't even care if you solved the actual hard distributed system problems that would arise like distributed transactions etc.

Anyway, I said I failed the interview, really they failed my filter because if they want me to ignore pragmaticism and blindly regurgitate a YouTube video on "system design" FAANG interview prep, then I don't want to work there anyway.

Re: Big data is dead (2023)

#378

Post by geysersam »

riku_iki wrote: post #313
wenc wrote: post #311

Earlier quoted context omitted.

Yes, it does that, assuming you read in the entire CSV, which works for CSVs that fit in memory. With Parquet you almost never read in the entire dataset and it's fast on all the projections, joins, etc. while living on disk.

> which works for CSVs that fit in memory. what? Why CSV is required to fit in memory in this case? I tested CSVs which are far larger than memory, and it works just fine.

The entire csv doesn't have to fit in memory, but the entire csv has to pass through memory at some point during the processing.

The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network.

Re: Big data is dead (2023)

#379

Post by riku_iki »

geysersam wrote: post #378
riku_iki wrote: post #313

Earlier quoted context omitted.

> which works for CSVs that fit in memory. what? Why CSV is required to fit in memory in this case? I tested CSVs which are far larger than memory, and it works just fine.

The entire csv doesn't have to fit in memory, but the entire csv has to pass through memory at some point during the processing. The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network.

> The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network.

this makes sense, and what I hoped to have. But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs.

Re: Big data is dead (2023)

#380

Post by thesz »

imiric wrote: post #222
thesz wrote: post #188

Earlier quoted context omitted.

Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…

What format would you recommend instead?

I do not know a good one.

A former colleague of mine is now working on a memory-mapped log-structured merge tree implementation and it can be a good alternative. LSM provides elasticity, one can store as much data as one needs, it is static, thus it can be compressed as well as Parquet-stored data, memory mapping and implicit indexing of data do not require additional data structures.

Something like LevelDB and/or RocksDB can provide most of that, especially when used in covering index [1] mode.

[1] https://www.sqlite.org/queryplanner.html#_covering_indexes

Post reply on HN