Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

241–250 of 475 posts

Re: Big data is dead (2023)

#241

Post by pdimitar »

apwell23 wrote: post #201
pdimitar wrote: post #159

Earlier quoted context omitted.

I already qualified my statement quite well by stating my background but if it makes you feel better then sure, show me the door. :) I was never a data scientist, just a guy who helped whenever it was necessary.

> I already qualified my statement quite well by stating my background No. You qualified it with "blows my mind" . Why would it 'blow your mind' if you don't have any data background.

Are you trolling? Did you miss the part where I said I worked with data but wouldn't say I'm a professional data scientist?

This negative cherry picking does not do your image any favors.

Re: Big data is dead (2023)

#242

Post by crowcroft »

pyrale wrote: post #179
mattbillenstein wrote: post #76

Earlier quoted context omitted.

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

Once you understand that 6tb fits on a hard drive, you can just as well put it in a run-of-the-mill pg instance, which metabase will reference just as easily. Hell, metabase is fine with even a csv file...

I worked in a large company that had a remote desktop instance with 256gb ram running a PG instance that analysts would log in to to do analysis. I used to think it was a joke of setup for such a large company.

I later moved to a company with a fairly sophisticated setup with Databricks. While Databricks offered some QoL improvements, it didn't magically make all my queries run quickly, and it didn't allow me anything that I couldn't have done on the remote desktop setup.

Re: Big data is dead (2023)

#243

Post by lizknope »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I'm on some reddit tech forums and people will say "I need help storing a huge amount of data!" and people start offering replies for servers that store petabytes.

My question is always "How much data do you actually have?" Many times you they reply with 500GB or 2TB. I tell that that isn't much data when you can get 1TB micro SD card the size of a fingernail or a 24TB hard drive.

My feeling is that if you really need to store petabytes of data that you aren't going to ask how to do it on reddit. If you need to store petabytes you will have an IT team and substantial budget and vendors that can figure it out.

Re: Big data is dead (2023)

#244

Post by rqtwteye »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a system IT had built for us with Hadoop, API gateways, multiple developers and hundreds of thousands of yearly cost. I told him that at our scale (now and any foreseeable future) I could easily run the whole thing on a USB drive attached to his laptop and a few python scripts. He looked really annoyed and I was never involved again with this project.

I think it’s part of the BS cycle that’s prevalent in companies. You can’t admit that you are doing something simple.

Re: Big data is dead (2023)

#246

Post by dkz999 »

whamlastxmas wrote: post #234
kmarc wrote: post #209

Earlier quoted context omitted.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

Idk, in this instance I feel pretty strongly that cloud, and solutions with unecessary overhead, are the fast food. The article proposes not eating it all the time.

Re: Big data is dead (2023)

#247

Post by jancsika »

kmarc wrote: post #209
ufo wrote: post #186

Earlier quoted context omitted.

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

That's great, but it's really just desiderata about you and your personal situation.

E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell.

I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that data into account when doing your assessment instead of attempting to optimize based on a single gate.

Edit: clarification

Re: Big data is dead (2023)

#248

Post by unholyguy001 »

One of the problems with the article is BigQuery becomes astronomically expensive at mid to high volumes. So there is a strong incentive to keep the data in BigQuery manageable or to even move off it as data volumes get higher.

Also larger enterprises don’t even use gcp all that much to begin with

Re: Big data is dead (2023)

#249

Post by pdimitar »

apwell23 wrote: post #205
photonthug wrote: post #183

Earlier quoted context omitted.

> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with. No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment. You risk comi…

> No need to act smug and superior I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .

The classic putting words in people's mouths technique it is then. The good old straw man.

If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]".

I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.

Re: Big data is dead (2023)

#250

Post by tored »

Former prime minister of Sweden Stefan Löfven got wind about Big data buzzword back in 2015 and used it in one of his speeches, that this is the future, however he used the Swedish translation of Big data, stordata. That generated some funny quips about where is lilldata.

https://www.aftonbladet.se/senastenytt/ttnyheter/inrikes/a/8...

Post reply on HN