You're never going to get a fang job with an approach like that. And basically most developers are working towards that goal.
Big data is dead (2023)
181–190 of 475 posts
Re: Big data is dead (2023)
#182When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
And how many data scientists are familiar with using awk scripts? If you’re the only one then you’ll have failed at scaling the data science team.
Re: Big data is dead (2023)
#183Earlier quoted context omitted.
Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…
> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.
> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.
No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment.
You risk coming across as a person who feels threatened by simple solutions, perhaps someone who wants to spend $500k in vendor subscriptions every year for simple and/or imaginary problems... exactly the type of thing TFA talks about.
But I'll ask the question.. why do you think safety, auditablity, access control, and knowledge sharing are incompatible with CLI tools and a specific choice of file system? What's your preferred alternative? Are you sticking with that alternative regardless of how often the work load runs, how often it changes, and whether the data fits in memory or requires a cluster?
Re: Big data is dead (2023)
#184Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…
> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…
Another way to say that is that "planning for success" is prematurely optimizing for scale.
Scaling up will bring its own challenges, with many of them difficult to foresee.
Re: Big data is dead (2023)
#185When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…
Why is .parquet better than protobuf?
Re: Big data is dead (2023)
#186When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…
From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"
Re: Big data is dead (2023)
#187Is there a solid definition of big data nowadays?
It seems to conflate somewhat with SV companies completely dismantling privacy concerns and hoovering up as much data as possible. Lots of scenarios I'm sure. I'm just thinking of FAANG in the general case.
Re: Big data is dead (2023)
#188When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…
Parquet is underdesigned. Some parts of it do not scale well.
I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much.
The flatness of the file metadata require external data structures to handle it more or less well. You cannot just mmap it and be good. This external data structure most probably will take as much memory as file metadata, or even more. So, 4G+ of your RAM will be, well, used slightly inefficiently.
(block-run-mapped log structured merge tree in one file can be as compact as parquet file and allow for very efficient memory mapped operations without additional data structures)
Thus, while parqet is a step, I am not sure it is a step in definitely right direction. Some aspects of it are good, some are not that good.
Re: Big data is dead (2023)
#189Earlier quoted context omitted.
.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…
Third consecutive time in 86 days that you mention .parquet files. I am out of my element here, but it's a bit weird
Sometimes when people discover or extensively use something they are eager to share in contexts they think are relevant. There is an issue when those contexts become too broad.
3 times across 3 months is hardly astroturfing for big parquet territory.
Re: Big data is dead (2023)
#190When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.
It would easy fit in ram: https://yourdatafitsinram.net/