Big memory is eating big data.
Big data is dead (2023)
251–260 of 475 posts
Re: Big data is dead (2023)
#252Earlier quoted context omitted.
> No need to act smug and superior I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .
The classic putting words in people's mouths technique it is then. The good old straw man. If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]". I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.
> that people don't try simpler and proven solutions FIRST
Well why don't people do that according to you ?
Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.
Re: Big data is dead (2023)
#253Earlier quoted context omitted.
I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…
And here we see this strange thing that data science people does in forgetting that 6TB is small change for any SQL server worth it's salt. Just dump it into Oracle, postgre, mssql, or mysql and be amazed by the kind of things you can do with 30year old data analysis technology on an modern computer.
you wouldn't have been a 'winner' per OP. real answer is loading it on their phones not on sqlserver or whatever.
Re: Big data is dead (2023)
#254When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
This is a great test / question. More generally, it tests knowledge with basic linux tooling and mindset as well as experience level with data sizes. 6TiB really isn't that much data these days, depending on context and storage format, etc. of course.
It could be a great question if you clarify the goals. As it stands it’s “here’s a problem, but secretly I have hidden constraints in my head you must guess correctly”.
The OPs desired solution could have been found from probably some of those other candidates if asked “here is the challenge, solve in most McGuyver way possible”. Because if you change the second part, the correct answer changes.
“Here is a challenge, solve in the most accurate, verifiable way possible”
“Here is a challenge, solve in a way that enables collaboration”
“Here is a challenge, 6TiB but always changing”
^ These are data science questions much more than the question he was asking. The answer in this case is that you’re not actually looking for a data scientist.
Re: Big data is dead (2023)
#255When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
> The winner of course was the guy who understood that 6TiB is what 6 of us in the room could store on our smart phones, or a $199 enterprise HDD (or three of them for redundancy), and it could be loaded (multiple times) to memory as CSV and simply run awk scripts on it.
If it's not a very write heavy workload but you'd still want to be able to look things up, wouldn't something like SQLite be a good choice, up to 281 TB: https://www.sqlite.org/limits.html
It even has basic JSON support, if you're up against some freeform JSON and not all of your data neatly fits into a schema: https://sqlite.org/json1.html
A step up from that would be PostgreSQL running in a container: giving you the support for all sorts of workloads, more advanced extensions for pretty much anything you might ever want to do, from geospatial data with PostGIS, to something like pgvector, timescaledb etc., while still having a plethora of drivers and still not making your drown in complexity and having no issues with a few dozen/hundred TB of data.
Either of those would be something that most people on the market know, neither will make anyone want to pull their hair out and they'll give you the benefit of both quick data writes/retrieval, as well as querying. Not that everything needs or can even work with a relational database, but it's still an okay tool to reach for past trivial file storage needs. Plus, you have to build a bit less of whatever functionality you might need around the data you store, in addition to there even being nice options for transparent compression.
Re: Big data is dead (2023)
#256Previous years I would have completely agreed with this post. Nowadays with the AI and ML craze I'm not so sure. I've seen plenty of companies using using vast amounts of data to train ML models to incorporate to their products. Definitely more data that can be handle by a traditional DB, and well into Big Data territory.
This isn't a value judgement about whether that's a good idea, just an observation from talking with many tech companies doing ML. This definitely feels like a bubble that will burst in due time, but for now ML is turbocharging Big Data.
Re: Big data is dead (2023)
#257Earlier quoted context omitted.
FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.
Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools
I see it more like "it's 11pm and a family member suddenly wants to eat a steak at home, what would you do?"
The person who says "I'm going drive back to the restaurant and take my professional equipment home to cook the steak" is probably offering the wrong answer.
I'm obviously not a professional cook, but presumably the ability to improvise with whatever tools you currently have is a desirable skill.
Re: Big data is dead (2023)
#258Earlier quoted context omitted.
From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"
FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.
I once killed the deployment of a big data team in a large bank when I laid out in excruciating details exactly what they'd have to deal with during an interview.
Last I heard theyd promoted one unix guy on the inside to baby sit a bunch of chron jobs on the biggest server they could find.
Re: Big data is dead (2023)
#259[deleted]
Re: Big data is dead (2023)
#260Earlier quoted context omitted.
FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.
That's great, but it's really just desiderata about you and your personal situation. E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that da…
This is the crux of it. Another interviewer would’ve marked “run on a local machine with a big SSD” - as: this fool doesn’t know enough about distributed systems and just runs toy projects on one machine