Big data is dead (2023) • Hacker News Board

Re: Big data is dead (2023)

#161

Post by [deleted] » Mon, May 27, 2024, 1:32 PM UTC

nevi-me wrote: post #73

pdimitar wrote: post #62

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

To be fair on candidates, CLI programs create technical debt the moment they're written. A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.

[deleted]

Re: Big data is dead (2023)

#162

Post by blagie » Mon, May 27, 2024, 1:33 PM UTC

CuriouslyC wrote: post #88

davedx wrote: post #19

Earlier quoted context omitted.

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

There's writing code to handle every eventuality, and there's considering 3-4 places you MIGHT pivot and making sure you aren't making those pivots harder than they need to be.

This is exactly what I try to do and what I've seen successful systems do.

Laying out adjacent markets, potential pivots, likely product features, etc. is a weekend-long exercise. That can help define both where the architecture needs to be flexible, and just as importantly, *where it does not*.

Over-engineering happens when you plan / architect for things which are unlikely to happen.

Re: Big data is dead (2023)

#163

Post by __MatrixMan__ » Mon, May 27, 2024, 1:35 PM UTC

nevi-me wrote: post #73

pdimitar wrote: post #62

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

To be fair on candidates, CLI programs create technical debt the moment they're written. A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.

True but it's typically less debt than anything involving a gui, pricetag, or separate server.

Re: Big data is dead (2023)

#164

Post by vishnugupta » Mon, May 27, 2024, 1:36 PM UTC

zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

I first heard of this 3 V's in a Michael Stonebreaker's talk[1]. For the uninitiated he's a legend in DBMS space, Turing award winner[2].

Highly recommend this and related talks by him, most of them are in YouTube.

[1] https://www.youtube.com/watch?v=KRcecxdGxvQ

[2] https://amturing.acm.org/award_winners/stonebraker_1172121.c...

Re: Big data is dead (2023)

#165

Post by blagie » Mon, May 27, 2024, 1:36 PM UTC

amenhotep wrote: post #34

notachatbot1234 wrote: post #15

Earlier quoted context omitted.

> - Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB) Nitpick but I cannot help myself: 8 bits are not even enough for a unique integer ID per person, that would require 8 bytes per person and then we are at 60GB already. I agree with pretty much anything else you said, just this stood out as wrong and Duty Calls.

Sure it is. You just need a one to one function from person to [0, eight billion]. Use that as your array index and you're golden. 8 GB is overkill, really, you could pack some boolean datum like "is over 18" into bits within the bytes and store your database in a single gigabyte. Writing your mapping function would be tricky! But definitely theoretically possible.

I'm old enough to have built systems with similar techniques. We don't do that much anymore since we don't need to, but it's not rocket science.

We had spell checkers before computers had enough memory to fit all words. They'd probabilistically find almost all incorrect words (but not suggest corrections). It worked fine.

Re: Big data is dead (2023)

#166

Post by blagie » Mon, May 27, 2024, 1:37 PM UTC

underwater wrote: post #20

blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later I think that in practice that’s counterproductive. A startup has a limited runway. If your engineers are spending your money on something that doesn’t pay off for years then they’re increasing the chance you’ll fail before it matters.

You're confusing planning with implementation.

Planning is a weekend, or at most a few weeks.

Re: Big data is dead (2023)

#167

Post by palata » Mon, May 27, 2024, 1:38 PM UTC

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack.

I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questions.

I was recently in a technical interview with an interviewer roughly my age and my experience, and I messed up. That's the game, I get it. But the interviewer got judgemental towards my (admittedly bad) answers. I am absolutely certain that were the roles inverted, I could choose a topic I know better than him and get him in a similarly bad position. But in this case, he was in the dominant position and he chose to make me feel bad.

My point, I guess, is this: when you are the interviewer, be extra careful not to abuse your dominant position, because it is probably counter-productive for your company (and it is just not nice for the human being in front of you).

Re: Big data is dead (2023)

#168

Post by mrtimo » Mon, May 27, 2024, 1:38 PM UTC

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

.parquet files are completely underrated, many people still do not know about the format!

.parquet preserves data types (unlike CSV)

They are 10x smaller than CSV. So 600GB instead of 6TB.

They are 50x faster to read than CSV

They are an "open standard" from Apache Foundation

Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it!

Please promote the use of .parquet files! Make .parquet files available for download everywhere .csv is available!

Re: Big data is dead (2023)

#169

Post by RodgerTheGreat » Mon, May 27, 2024, 1:38 PM UTC

randomtoast wrote: post #118

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

With the exception of regexes- which any programmer or data analyst ought to develop some familiarity with anyway- you can describe the entirety of AWK on a few sheets of paper. It's a versatile, performant, and enduring data-handling tool that is already installed on all your servers. You would be hard-pressed to find a better investment in technical training.

Re: Big data is dead (2023)

#170

Post by blagie » Mon, May 27, 2024, 1:40 PM UTC

threeseed wrote: post #23

blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> nothing Mongo does which postgresql doesn't do better a) It has a built-in and supported horizontal scalability / HA solution. b) For some use cases e.g. star schemas it has significantly better performance. > Big data solutions aren't nosql Almost all big data storage solutions are NoSQL.

> Almost all big data storage solutions are NoSQL.

Most I've seen aren't. NoSQL means non-relational database. Most big data solutions I've seen will not use a database at all. An example is hadoop.

Once you have a database, SQL makes a lot of sense. There are big data SQL solutions, mostly in the form of columnar read-optimized databases.

On the above, a little bit of relational can make a huge performance difference, in the form of, for example, a big table with compact data with indexes into small data tables. That can be algorithmically a lot more performant than the same thing without relations.