121–130 of 475 posts
Earlier quoted context omitted.
> just go wild with CLI programs, I know dozens that work on CSV and XML ...or put it into SQLite for extra blazing fastness! No kidding.
That's included in CLI tools. Also duckdb and clickhouse-local are amazing.
I had a heck of a time running the server locally before I discovered the CLI.
When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
You tricked candidates with your nonsensical scenario. Hate smartass interviewers like this that are trying some gotcha to feel smug about themselves.
Most candidates don't feel comfortable telling ppl 'just load on your laptops' even if they think thats sensible. They want to present a 'professional solution', esp when you tricked them with the word 'stack'. which is how most of them prbly perceived your trick question.
This comment is so infuriating to me. Why be assholes to each other when world is already full of them.
When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.
Earlier quoted context omitted.
How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.
> How would six terabytes fit into memory? What device do you have in mind? I've seen places use 2TB RAM servers, and that was years ago, and it isn't even that expensive (can get those for about $5K or so). Currently HP allows "up to 48 DIMM slots which support up to 6 TB for 2933 MT/s DDR4 HPE SmartMemory". Close enough to fit the OS, the userland, and 6 TiB of data with some light compression. > It seems like it w…
A better question would be:
Why would anyone stream 6 terabytes of data over the internet?
In 2010 the answer was: because we can’t fit that much data in a single computer, and we can’t get accounting or security to approve a $10k purchase order to build a local cluster, so we need to pay Amazon the same amount every month to give our ever expanding DevOps team something to do with all their billable hours.
That may not be the case anymore, but our devops team is bigger than ever, and they still need something to do with their time.
Earlier quoted context omitted.
True on the premise, yep, though I'm not sure how using CLI programs like LEGO blocks creates a tech debt?
I remember replacing a CLI program built like Lego blocks. It was 90-100 LEGO blocks, written over the course of decades, in: Cobol; Fortran; C; Java; Bash; and Perl, and the Legos "connected" with environmental variables. Nobody wanted to touch it lest they break it. Sometimes it's possible to do things too smartly. Apache Spark runs locally (and via CLI).
Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.
Earlier quoted context omitted.
I need to learn more about the latter for some log processing...
Log files aren’t data. That’s your first problem. But that’s the only thing that most people have that generates more bytes than can fit on screen in a single spreadsheet.
Earlier quoted context omitted.
You have 6 TiB of ram?
You can have 8TB RAM in a 2U box for under 100K. grab a couple and it will save you millions a year compared to over-engineered bigdata setup.
No business is passing 6tb data around on their laptops.
When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
Earlier quoted context omitted.
I agree that keeping data local is great and should be the first option when possible. It works great on 10GB or even 100GB, but after that starts to matter what you optimize for because you start seeing execution bottlenecks. To mitigate these bottlenecks you get fancy hardware (e.g oracle appliance) or you scale out (and get TCO/performance gains from separating storage and compute - which is how Snowflake sold 3x…
> I agree that keeping data local is great and should be the first option when possible. It works great on 10GB or even 100GB, but after that starts to matter what you optimize for because you start seeing execution bottlenecks. The point of the article is 99.99% of businesses never pass even the 10 Gb point though.