Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

421–430 of 475 posts

Re: Big data is dead (2023)

#421

Post by tdeck »

MOARDONGZPLZ wrote: post #30
davedx wrote: post #19

Earlier quoted context omitted.

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

In my experience the drag caused from the thinking to plan for scalability early has been so much greater than the effort to rearchitect things when and if the company becomes a unicorn that one is significantly more likely to become a unicorn if they simply focus on execution and very fast iteration and save the scalability until it’s actually needed (and they can hire a team of whomever to effect this change with t…

Not only that but the work to rearchitect for scalability is often appealing, energizing work that lots of people want to do. While the work of maintaining overly complex software full of affordances for scalability scenarios (or use cases) that never come up is rarely as productive or motivating.

Re: Big data is dead (2023)

#422

Post by llm_trw »

dartos wrote: post #269
coffeebeqn wrote: post #260

Earlier quoted context omitted.

This is the crux of it. Another interviewer would’ve marked “run on a local machine with a big SSD” - as: this fool doesn’t know enough about distributed systems and just runs toy projects on one machine

That is what I think interviewers think when I don’t immediately bring up kubernetes and sqs in an architecture interview

The interview goes both ways.

The few times I've ignored red flags from a company in an interview I've been in a world of pain afterwards.

Re: Big data is dead (2023)

#423

Post by altdataseller »

theamk wrote: post #339
dartos wrote: post #269

Earlier quoted context omitted.

That is what I think interviewers think when I don’t immediately bring up kubernetes and sqs in an architecture interview

depending on the shop? For some kinds of tasks, jumping to kubernets right away would be a minus during interview.

Recommending kubernetes immediately is an insta-rejection for me, if I was an interviewer

Re: Big data is dead (2023)

#424

Post by thsksbd »

palata wrote: post #167
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

I want my boss to be straight with, I need those below me to call me out of Im talking bullshit.

Tell me this isn't big data, and then, if you must, tell me about hadoop (or whatever big data is)

Re: Big data is dead (2023)

#425

Post by thsksbd »

coffeebeqn wrote: post #260
jancsika wrote: post #247

Earlier quoted context omitted.

That's great, but it's really just desiderata about you and your personal situation. E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that da…

This is the crux of it. Another interviewer would’ve marked “run on a local machine with a big SSD” - as: this fool doesn’t know enough about distributed systems and just runs toy projects on one machine

So give both answers. Itd would happen all the time in college, or in a thesis defense, where one disagrees with the examiner and one finds oneself in a position of "is this guy trying to trick me!"

Give both answers, and explain why the obvious "hard" answer is wrong

Re: Big data is dead (2023)

#426

Post by arvinsim »

rqtwteye wrote: post #244
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a system IT had built for us with Hadoop, API gateways, multiple developers and hundreds of thousands of yearly cost. I told him that at our scale (now and any foreseeable future) I could easily run the whole thing on a USB drive attached to his laptop and a few pyt…

Resume Driven Development is real. Yes, you can optimize the solution to just a USB drive. But that wouldn't check the boxes of people looking at your resume.

Re: Big data is dead (2023)

#427

Post by fifilura »

orhmeh09 wrote: post #346
fifilura wrote: post #112

Earlier quoted context omitted.

I agree with this. BigQuery or AWS s3/Athena. You shouldn't have to set up a cluster for data jobs these days. And it kind of points out the reason for going with a data scientist with the toolset he has in mind instead of optimizing for a commandline/embedded programmer. The tools will evolve in the direction of the data scientist, while the embedded approach is a dead end in lots of ways. You may have outsmarted so…

It is actually pretty easy to do the same type of processing you would do on a cluster with AWS Batch.

Possibly, but it seems like overkill for the type of analysis that the OP expected the interviewee to do with awk.

SQL should be fine for that.

Actually, I have a feeling that the awk solution will struggle if there are many unique keys.

For example if they in that dataset have a million customers and want to extract the top 10. Then there is an intermediate map stage that will be storage or memory consuming.

It is like matrix multiplication. Calculating the dot product is trivial, but when the matrix has n:m dimensions and n,m starts to grow, it becomes more and more resource heavy. And then the laptop will not be able to handle it.

(in the example, m is the number of rows, and n is the number of unique customers. The dot product is just a sum over one dimension, while the group by customer id is the tricky part)

Re: Big data is dead (2023)

#428

Post by xnx »

davedx wrote: post #19
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

Are there any tech businesses that failed to succeed because they couldn't scale? Myspace maybe? There are dozens (probably hundreds) of famous examples of companies that self destructed because they bogged themselves down with an elective rewrite or by solving imaginary scaling problems (at scale).

Re: Big data is dead (2023)

#429

Post by [deleted] »

riku_iki wrote: post #389
wenc wrote: post #382

Earlier quoted context omitted.

>But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs. Anecdotally having worked with large CSVs and large on-disk Parquet datasets, my experience is the opposite of yours. My DuckDB queries operate directly on Parquet on disk and never load the entire dataset, and is always much faster than the equivalent operation on CSV files. I think your experience mi…

> I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples original discussion was about CSV vs parquet "reader" part, so this is exactly apple to apple testing, easy to benchmark and I stand my ground. What you are doing downstream, it is another question which is not possible to discuss becaus…

[deleted]

Re: Big data is dead (2023)

#430

Post by orhmeh09 »

fifilura wrote: post #427
orhmeh09 wrote: post #346

Earlier quoted context omitted.

It is actually pretty easy to do the same type of processing you would do on a cluster with AWS Batch.

Possibly, but it seems like overkill for the type of analysis that the OP expected the interviewee to do with awk. SQL should be fine for that. Actually, I have a feeling that the awk solution will struggle if there are many unique keys. For example if they in that dataset have a million customers and want to extract the top 10. Then there is an intermediate map stage that will be storage or memory consuming. It is l…

I agree completely for this scale. I did want to point out that it's fairly easy these days to do the kinds of things one would do on a cluster, which I learned just a few months ago myself :)
Post reply on HN