When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
You have 6 TiB of ram?
Big data is dead (2023)
81–90 of 475 posts
Re: Big data is dead (2023)
#82Earlier quoted context omitted.
It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.
Assuming you're serious for a moment, I don't think AI is really a practical tool for working with big data. - The "hallucination" factor means every result an AI tells you about big data is suspect. I'm sure some of you who really understand AI more than the average person can "um akshually" me on this and tell me how it's possible to configure ChatGPT to absolutely be honest 100% of the time but given the current s…
> The "hallucination" factor means every result an AI tells you about big data is suspect.
AI / ML means more than just LLM chat output, even if that's the current hype cycle of the last couple of years. ML can be used to build a perfectly serviceable classifier, or predictor, or outlier detector.
It suffers from the lack of explainability that's always plagued AI / ML, especially as you start looking at deeper neural networks where you're more and more heavily reliant on their ability to approximate arbitrary functions as you add more layers.
> you're using this heavy, low-performance general-purpose tool to solve a problem which can be solved much more performatively by using tools which have been designed from the beginning to handle data management and analysis
You are not wrong here, but one challenge is that sometimes even your domain experts do not know how to solve the problem, and applying traditional statistical methods without understanding the space is a great way of identifying spurious correlations. (To be fair, this applies in equal measure to ML methods.)
Re: Big data is dead (2023)
#83When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
Problem is possibly that most people with that sort of hands-on intuition for data don't see themselves as data scientists and wouldn't apply for such a position.
It's a specialist role, and most people with the skills you seek are generalists.
Re: Big data is dead (2023)
#84> How many workloads need more than 24TB of RAM or 445 CPU cores?
Electron?
Re: Big data is dead (2023)
#85When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…
A moderately powerful desktop processor has memory bandwidth of over 50TB/s so yeah it'll take a couple of minutes sure.
Re: Big data is dead (2023)
#86Re: Big data is dead (2023)
#87When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
I agree that keeping data local is great and should be the first option when possible. It works great on 10GB or even 100GB, but after that starts to matter what you optimize for because you start seeing execution bottlenecks. To mitigate these bottlenecks you get fancy hardware (e.g oracle appliance) or you scale out (and get TCO/performance gains from separating storage and compute - which is how Snowflake sold 3x…
> I agree that keeping data local is great and should be the first option when possible. It works great on 10GB or even 100GB, but after that starts to matter what you optimize for because you start seeing execution bottlenecks.
The point of the article is 99.99% of businesses never pass even the 10 Gb point though.
Re: Big data is dead (2023)
#88Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…
> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…
There's writing code to handle every eventuality, and there's considering 3-4 places you MIGHT pivot and making sure you aren't making those pivots harder than they need to be.
Re: Big data is dead (2023)
#89Earlier quoted context omitted.
You have 6 TiB of ram?
https://yourdatafitsinram.net/
Was posted as https://news.ycombinator.com/item?id=9581862 in 2015
Re: Big data is dead (2023)
#90When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
[dead]