Live data from Hacker News

Big data is dead (2023)

motherduck.com

151–160 of 475 posts

Re: Big data is dead (2023)

#151
post #51
post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

You have 6 TiB of ram?

We are decomming our 5-year old 4TB systems this year and could have been ordered with more

Re: Big data is dead (2023)

#152
post #132

Earlier quoted context omitted.

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

About $20/month for chatgpt or similar copilot, which really they should reach for independently anyhow.

And since the data scientist cannot verify the very complex AWK output that should be 100% compatible with his SQL query, he relies on the GPT output for business-critical analysis.

Re: Big data is dead (2023)

#153

Earlier quoted context omitted.

How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.

There are machines that can fit that and more: https://yourdatafitsinram.net/ I'm not advocating that this is generally a good or bad idea, or even economical, but it's possible.

I'm trying to understand what the person I'm replying to had in mind when they said fit six terabytes in memory and search with awk.

is this what they were referring to just by a big ass Ram machine?

Re: Big data is dead (2023)

#154
post #19
post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

> That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because people never just make plans, they usually implement too.

That's not my experience at all.

Architecture != implementation

Architecture astronauts will try to solve the world's problems in v0. That's very different from having an architectural vision and building a subset of it to solve problems for the next 3 months. Let me illustrate:

* Agile Idiot: We'll stick it all in PostgreSQL, however it fits, and meet our 3-month milestone. [Everything crashes-and-burns on success]

* Architecture Astronaut: We'll stick it all in a high-performance KVS [Business goes under before v0 is shipped]

* Success: We have one table which will grow to petabytes if we reach scale. We'll stick it all in postgresql for now, but maintain a clean KVS abstraction for that one table. If we hit success, we'll migrate to [insert high-performance KVS]. All the other stuff will stay in postgresql.

The trick is to have a pathway to success while meeting short-term milestones. That's not just software architecture. That's business strategy (clean beachhead, large ultimate market), and every other piece of designing a successful startup. There should be a detailed 3-month plan, a long-term vision, and a rough set of connecting steps.

Re: Big data is dead (2023)

#155

Earlier quoted context omitted.

How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.

> How would six terabytes fit into memory? What device do you have in mind? I've seen places use 2TB RAM servers, and that was years ago, and it isn't even that expensive (can get those for about $5K or so). Currently HP allows "up to 48 DIMM slots which support up to 6 TB for 2933 MT/s DDR4 HPE SmartMemory". Close enough to fit the OS, the userland, and 6 TiB of data with some light compression. > It seems like it w…

I mean if you're doing data science the data is not always organized and of course you would want multi-processing.

1 TB of memory is like 5 grand from a quick Google search then you probably need specialized motherboards.

Re: Big data is dead (2023)

#156

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

What about his answer prevents any of that? As stated the question didn't require any of what you outline here. ZFS will probably do a better job of protecting your data than almost any other filesystem out there so it's not a bad foundation to start with if you want to protect data.

Your entire post reeks of "I'm smarter than you" smugness while at the same time revealing no useful information or approaches. Near as I can tell no one should trust you with any data.

Re: Big data is dead (2023)

#157

Earlier quoted context omitted.

No no, I didn't mean that at all. I meant a script using well-known CLI programs. Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.

Well your little CLI-query is suddenly in production and then... it easily escalates.

I already said I never managed a data lake and simply got stuff when it was needed but if you need to criticize then by all means, go wild.

Re: Big data is dead (2023)

#158
post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> nothing Mongo does which postgresql doesn't do better a) It has a built-in and supported horizontal scalability / HA solution. b) For some use cases e.g. star schemas it has significantly better performance. > Big data solutions aren't nosql Almost all big data storage solutions are NoSQL.

Anyone reading the above comment ask yourself. Do you actually think when interest rates dropped to zero they suddenly inverted a system that was better than SQL. “Horizontal scaling” I’m sorry I don’t speak marketing language, what is that? I’ve only been doing this for decades.

Re: Big data is dead (2023)

#159

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

I already qualified my statement quite well by stating my background but if it makes you feel better then sure, show me the door. :)

I was never a data scientist, just a guy who helped whenever it was necessary.

Re: Big data is dead (2023)

#160

Something similar will happen with generative AI someday. AI scientists will propose all sorts of elaborate complex solutions to problems using LLMs, and the dismissive responsive will be “Your problem is solvable with a couple if statements.” Most people just don’t have problems that require AI.

Thats one of the first things Andrew Ng said in his old ML course
Post reply on HN