Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

391–400 of 475 posts

Re: Big data is dead (2023)

#391

Post by int_19h »

palata wrote: post #266
kmarc wrote: post #209

Earlier quoted context omitted.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Sure, but as you said yourself: it's a trick question. How often does the employee have to answer trick questions without having any time to think in the actual job? As an interviewer, why not asking: "how would you do that in a setup that doesn't have much data and doesn't need to scale, and then how would you do it if it had a ton of data and a big need to scale?". There is no trick here, do you feel you lose infor…

Being able to ask qualifying questions like that, or presenting options with different caveats clearly spelled out, is part of the job description IMO, at least for senior roles.

Re: Big data is dead (2023)

#392

Post by int_19h »

xethos wrote: post #300
crowcroft wrote: post #230

Earlier quoted context omitted.

I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.

Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either). [0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... [1] https://www.tomshardware.com/reviews/ram-speed…

I think the point is that if it fits on a single drive, you can still get away with a much simpler solution (like a traditional SQL database) than any kind of "big data" stack.

Re: Big data is dead (2023)

#393

Post by int_19h »

the_real_cher wrote: post #91
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.

Per one of the links below, IBM Power System E980 can be configured for up to 64Tb RAM.

Re: Big data is dead (2023)

#394

Post by int_19h »

qaq wrote: post #75
boppo1 wrote: post #51

Earlier quoted context omitted.

You have 6 TiB of ram?

You can have 8TB RAM in a 2U box for under 100K. grab a couple and it will save you millions a year compared to over-engineered bigdata setup.

I wonder how much this costs: https://www.ibm.com/products/power-e1080

And how that price would compare to the equivalent big data solution in the cloud.

Re: Big data is dead (2023)

#395

Post by int_19h »

internet101010 wrote: post #318
paulryanrogers wrote: post #44

Earlier quoted context omitted.

> Obsessed with this "you must use PostgreSQL for every use case" nonsense. Pg fans are certainly here asking "why not PG?". Yet so are fans of other DBs; like DuckDB, CouchDB, SQLite, etc.

I don't see so much DuckDB and CouchDB proselytizing but the SQLite force always out strong. I tend to divide the Postgres vs. SQLite decision on if the data in question is self-contained. Like am I pulling data from elsewhere (Postgres) or am I creating data within the application that is only used for the functionality of said application (SQLite).

SQLite, in addition to just being plain popular, is a fairly natural stepping stone - you get a lot of fundamental benefits of an SQL RDBMS (abstract high-level queries, ACID etc) without the overhead of maintaining a database server.

Postgres is the next obvious stepping stone after that, and the one where the vast majority of actual real-world cases that are not hypotheticals end up fitting.

Re: Big data is dead (2023)

#396

Post by int_19h »

angarg12 wrote: post #256

Previous years I would have completely agreed with this post. Nowadays with the AI and ML craze I'm not so sure. I've seen plenty of companies using using vast amounts of data to train ML models to incorporate to their products. Definitely more data that can be handle by a traditional DB, and well into Big Data territory. This isn't a value judgement about whether that's a good idea, just an observation from talking…

But do they need to query that data?

Re: Big data is dead (2023)

#397

Post by Terr_ »

the_real_cher wrote: post #172
fijiaarone wrote: post #125

Earlier quoted context omitted.

“How would six terabytes fit into memory?” A better question would be: Why would anyone stream 6 terabytes of data over the internet? In 2010 the answer was: because we can’t fit that much data in a single computer, and we can’t get accounting or security to approve a $10k purchase order to build a local cluster, so we need to pay Amazon the same amount every month to give our ever expanding DevOps team something to…

Well yeah streaming to the cloud to work around budget issues is a while nother convo haha.

I'm having flashbacks to some new outside-hire CEO making flim-flam about capex-vs-opex in order to justify sending business towards a contracting firm they happened to know.

Re: Big data is dead (2023)

#398

Post by int_19h »

deadbabe wrote: post #220
doubloon wrote: post #160

Earlier quoted context omitted.

Thats one of the first things Andrew Ng said in his old ML course

This is why I personally just can’t find motivation to even pay attention to most AI developments. It’s a toy, it does some neat things, but there’s no problem I’ve heard of or encountered where LLM style AI was the only tool for the job, or even the best tool. The main use seems to be content creation and manipulation at scale, which the vast majority of companies simply don’t have to deal with. Similarly, a lot of…

Natural language processing is in obvious area in which LMs are consistently outperforming the best bunches of if-statements by a very large margin, and it has very broad applicability.

E.g. I would argue that its translation capabilities alone make GPT-4 worthwhile, even if it literally couldn't do anything else.

Re: Big data is dead (2023)

#399

Post by 1vuio0pswjnm7 »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

"... or a $199 enterprise HDD"

External or internal? Any examples?

"... it could be loaded (multimple times) to memory"

All 6TiB at once, or loaded in chunks?

Re: Big data is dead (2023)

#400

Post by ninkendo »

marginalia_nu wrote: post #86
ninkendo wrote: post #53

Earlier quoted context omitted.

You don’t need that much ram to use mmap(2)

To be fair, mmap doesn't put your data in RAM, it presents it as though it was in RAM and has the OS deal with whether or not it actually is.

Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram.

It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.

Post reply on HN