221–230 of 475 posts
Earlier quoted context omitted.
meh. memory address is the ID
Airline booking numbers used to just be the sector number of your booking record on the mainframes HDD.
I wonder how they dealt with common storage issues like backups and disks having bad sectors.
Earlier quoted context omitted.
.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…
Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…
Earlier quoted context omitted.
.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…
Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…
Most tools can run queries across parquet files.
Like everything, it has its strengths and weaknesses, but in most cases, it has better trade-offs over CSV if you have more than a few thousand rows.
When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
One can't go wrong with DuckDB+SQLite+Open/Elasticsearch either with 6 to 8 even 10 TB of data.
[0]. https://duckdb.org/
Earlier quoted context omitted.
I already qualified my statement quite well by stating my background but if it makes you feel better then sure, show me the door. :) I was never a data scientist, just a guy who helped whenever it was necessary.
> I already qualified my statement quite well by stating my background No. You qualified it with "blows my mind" . Why would it 'blow your mind' if you don't have any data background.
Earlier quoted context omitted.
> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with. No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment. You risk comi…
> No need to act smug and superior I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .
> Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.
A lot of industry work really does fall into this category, and it's not controversial to say that going the wrong way on this thing is mind-blowing. More than not being controversial, it's not confrontational, because his comment was essentially re: the industry, whereas your comment is directed at a person.
Drive by sniping where it's obvious you don't even care to debate the tech itself might get you a few "sick burn, bro" back-slaps from certain crowds, or the FUD approach might get traction with some in management, but overall it's not worth it. You don't sound smart or even professional, just nervous and afraid of every approach that you're not already intimately familiar with.
Actually, now that I think about it, we should have two products for the users, one let them to query single records as fast as possible without hitting the production OLTP transactional database, even from really big data (find one record from PB level data in seconds), one to power the dashboards that ONLY show aggregation. Is Lakehouse a solution? I have never used it.
The most interesting applications for "big data" are all (IMO) in the scientific computing space. Yeah, your e-commerce business probably won't ever need "big data" but load up a couple genomics research sets and you sure will.
Earlier quoted context omitted.
> Your entire post reeks of "I'm smarter than you" unlike "blows my mind" ? > As stated the question didn't require any of what you outline here. Right. OP mentioned it was "tricky question" . What makes it tricky is that all those attributes are implicitly assumed. I wouldn't interview at google and tell them my "stack" is "load it on your laptop". I would never say that in an interview even if I think that's the ri…
"blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not. You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.
When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
https://x.com/garybernhardt/status/600783770925420546 (Gary Bernhardt of WAT fame): > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. This is from 2015...