Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

471–475 of 475 posts

Re: Big data is dead (2023)

#471

Post by coldtea »

the_real_cher wrote: post #270
coldtea wrote: post #196

Earlier quoted context omitted.

> I mean if you're doing data science the data is not always organized and of course you would want multi-processing Not necessarily - I might not want it or need it. It's a few TB, it can be on a fast HD, on an even faster SSD, or even in memory. I can crunch them quite fast even with basic linear scripts/tools. And organized could just mean some massaging or just having them in csv format. This is already the same…

Yeah I agree about over engineering. Im just trying to understand the parent to my original comment. How would running awk for analysis on 6TB of data work quickly and efficiently? They say it would go into memory but its not clear to me how that would work as would still have paging and thrashing issues if the data didnt have often used sections of the data. am I overthinking it and they were they just referring to…

>How would running awk for analysis on 6TB of data work quickly and efficiently?

In that 6TB is not that huge of an amount

That's their total dataset, and there's no "real time" requirement.

They can start a batch process, process the data, and be done with it.

Here's an example of someone using awk (read further down for the relevant section):

https://livefreeordichotomize.com/posts/2019-06-04-using-awk...

"I was now able to process a whole 5 terabyte batch in just a few hours."

>They say it would go into memory but its not clear to me how that would work as would still have paging and thrashing issues if the data didnt have often used sections of the data

There's no need to have paging and thrashing issues if you can fit all (or even most) of your data in memory. And you can always also split, process, and aggregate partial results.

>am I overthinking it and they were they just referring to buying a big ass Ram machine?

Yeah, they said one can buy a machine with several TB of memory.

Re: Big data is dead (2023)

#472

Post by valenterry »

tekknik wrote: post #467
valenterry wrote: post #290

Earlier quoted context omitted.

It doesn't matter. The answer should be "It depends, what are the circumstances - do we expect high growth in the future? Is it gonna stay around 6TB? How and by whom will it be used and what for?" Or, if you can guess what the interviewer is aiming for, state the assumption and go from there "If we assume it's gonna stay at Then the interviewer can interrupt and change the assumptions to his needs.

The interview question made it clear it was a maximum of 6TiB. With this, one can assume it’s not growing yet still think the interviewer either doesn’t know this isn’t big data or that they want to test their knowledge.

From what the comment tells, the interview question said that the maximum IS 6TiB. There was no further information given, so I assume it didn't make any assumptions about how it might change.

Even if it would say "it will stay at 6TiB" I would probably prefer a senior candidate to briefly question it, such as "It is surprising that we know it will stay at 6TiB and if this were a real project I'd try to at least sanitycheck that this requirement is really correct, but for now I'll assume this is a given..."

At least if, as the interviewer, I told them to treat the challenge similar to a real request/project and not to not-question the given numbers etc.

Re: Big data is dead (2023)

#473

Post by pas »

blagie wrote: post #469
pas wrote: post #461

Earlier quoted context omitted.

> if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons most of the business plans are clear. acquire users and/or paying customers, build out the product, raise money and eventually raise prices too. engineering side is even simpler, keep the lights on, work with product designers to find incremental steps, iterate. as…

> they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.) > ... > of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?) Netscape. (Plenty of others too)

Netscape's main revenue came from Navigator, right? A browser. Classic commodity. How did they fail due to engineering deficiencies? (Netscape was better than Mosaic, they invented JavaScript, there was plenty of chops there, and initially they had revenue, IPO went great ... at that point it's arguably not a startup anymore.)

And even when it failed it was worth 4.2B to AOL.

... if we're talking about browsers, Netscape's reincarnation Mozilla is again in the same tough spot. They have zero business sense, absolutely no plan, nada. At that point it's completely irrelevant how long it takes them to fix bugs or implement Web APIs or make it faster on various platforms. And still, they funded Rust and Servo.

Re: Big data is dead (2023)

#474

Post by drumdance »

I was CTO of an analytics startup in the early days of MongoDB hype. We built on MySQL. One of our competitors was MixPanel.

We were having scaling issues with one particular customer, so I looked into the various solutions. They all looked pretty cool, but I didn't like that the developer communities were small compared to MySQL and PostgreSQL.

So I ended up sticking with MySQL and building a simple sharding mechanism that allowed us to move big customers to their own server if necessary. It worked great.

While I was working on that, MixPanel published a long, highly detailed blog post about why they had moved to MongoDB. That led me to have second thoughts. "If these guys are doing it, maybe we should too?" But I was pretty far down the sharding development path and couldn't justify making such a wholesale change based on one blog post.

Fast forward about nine months, and MixPanel published a new blog post about why they had moved off MongoDB. Vindication!

Ever since then I've had conservative disposition when it comes to deploying new technologies.

Re: Big data is dead (2023)

#475

Post by SkipperCat »

kmarc wrote: post #360
SkipperCat wrote: post #347

Earlier quoted context omitted.

That makes total sense if you're archiving the data, but what happens when you want to have 10,000 people have access to read/update the data concurrently. Then you start to need some fairly complex solutions.

This thread blew up a lot, and some unfriendly commenters made many assumptions about this innocent story. You didn't, and indeed you have a point (missing specification of expected queries), so I expand it as a response here. Among the MANY requirements I shared with the candidate, only one was the 6TiB. Another one was that it was going to be serving as part of the backend of an internal banking knowledge base, wit…

Thanks for the follow up. I've always felt any questions is good for an interview if it starts a conversation. Your thread did just that so I'd consider it a success!
Post reply on HN