Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

461–470 of 475 posts

Re: Big data is dead (2023)

#461

Post by pas »

blagie wrote: post #16
brtkdotse wrote: post #11

Earlier quoted context omitted.

> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success

It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a met…

> if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons

most of the business plans are clear. acquire users and/or paying customers, build out the product, raise money and eventually raise prices too.

engineering side is even simpler, keep the lights on, work with product designers to find incremental steps, iterate.

as others have mentioned upthread if a startup finds a great niche (and manages to make it a whole new market for example) they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.)

of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?)

Re: Big data is dead (2023)

#462

Post by skeaker »

layer8 wrote: post #327
ufo wrote: post #186

Earlier quoted context omitted.

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

You shouldn’t guess what they expect, you should say what you think is right, and why. Do you want to work at a company where you would fail an interview due to making a correct technical assessment? And even if the guess is right, as an interviewer I would be more impressed by an applicant that will give justified reasons for a different answer than what I expected.

>Do you want to work at a company where you would fail an interview due to making a correct technical assessment?

How much do they pay? How long has it been since my last proper meal? How long until my rent is due?

Re: Big data is dead (2023)

#463

Post by rcxdude »

kbolino wrote: post #142
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Even if a 6 terabyte CSV file does fit in RAM, the only thing you should do with it is convert it to another format (even if that's just the in-memory representation of some program). CSV stops working well at billions of records. There is no way to find an arbitrary record because records are lines and lines are not fixed-size. You can sort it one way and use binary search to find something in it in semi-reasonable…

Yeah, but it very well not be data that needs random access or live insertion. A lot of data is basically just one big chunk of time-series that just needs some number crunching run over the whole lot every half a year.

Re: Big data is dead (2023)

#464

Post by 62951413 »

mrtimo wrote: post #168
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

You kind of can peek into parquet files with a tiny command line utility: https://github.com/manojkarthick/pqrs

Re: Big data is dead (2023)

#465

Post by throwaway2037 »

bee_rider wrote: post #302
whamlastxmas wrote: post #234

Earlier quoted context omitted.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

I’m not sure if you are referencing it intentionally or not, but some chefs (Gordon Ramsey for one) will ask an interviewee to make some scrambled eggs; something not super niche or specialized but enough to see what their technique is. It is a sort of “interview hack” example that’s been used to emphasize the idea of a simple unspecialized skill-test that went around a while ago. I guess upcoming chefs probably prac…

The egg omelet is a classic for testing French chefs. It is much less about the ingredients (quality, etc), and much more about pure skill to cook it perfectly.

What would be the equivalent for a technical interview? Perhaps: Implement a generic linked list or dynamic array.

Re: Big data is dead (2023)

#466

Post by teleforce »

Shrezzing wrote: post #38
teleforce wrote: post #28

Not dead, it's just having it's winter time not unlike AI winter and once it has its similar "chatbot" moment, all will be well. My take on the killer application is the climate change for example earthquakes monitoring. For a case study China has just finished building world's largest earthquake monitoring system with the cost of around USD1 Billion across the country with 15K stations [1]. Somehow at the moment is…

I think these examples still loosely fits the author's argument: > There are some cases where big data is very useful. The number of situations where it is useful is limited Even though there are some great use-cases, the overwhelming majority organisations, institutions, and projects will never have a "let's query ten petabytes" scenario that forces them away from platforms like Postgres. Most datasets, even at very…

Big data is not only about storage size but also about processing e.g. RAM. In the next coming years the trend is that there will be more (read exponential) IoT sensor devices than we can ever imagine and the nature of their data will be mostly big data in the sense of size (storage) and the analysis (RAM). Just check the latest cars or EV cars, many of them have hundreds of sensor devices and these devices are already connected to the Internet.

Another upcoming example is the latest 5G DECT NR+ standards (the first non-cellular 5G), it will only fuels these massive accumulation of datasets and these monitoring sensor devices do not even get connected to the Internet (think of private factory networks) [1].

Apparently there are limited number of human using or having sensors but for non-human based devices the sky is the limit. For human based communication the data is very limited, we rarely communicate with each others and most of our data now is based on our intermittent media consumptions while streaming audio/video [2]. For IoT sensors devices, they mainly have regular and frequent interval sampling that probably in the ranges of every seconds, minutes, hours, etc. Some if not most of this data is not clean data, there are raw data, and raw data is inherently big and huge compared to the data, for example raw image data vs JPEG data, where the former can be several time bigger in size and processing requirements.

[1] DECT NR+: A technical dive into non-cellular 5G:

https://news.ycombinator.com/item?id=39905644

[2] 50 Video Statistics You Can’t Ignore In 2024:

https://www.synthesia.io/post/video-statistics

Re: Big data is dead (2023)

#467

Post by tekknik »

valenterry wrote: post #290
ufo wrote: post #186

Earlier quoted context omitted.

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

It doesn't matter. The answer should be "It depends, what are the circumstances - do we expect high growth in the future? Is it gonna stay around 6TB? How and by whom will it be used and what for?" Or, if you can guess what the interviewer is aiming for, state the assumption and go from there "If we assume it's gonna stay at Then the interviewer can interrupt and change the assumptions to his needs.

The interview question made it clear it was a maximum of 6TiB. With this, one can assume it’s not growing yet still think the interviewer either doesn’t know this isn’t big data or that they want to test their knowledge.

Re: Big data is dead (2023)

#468

Post by snakeyjake »

yapret wrote: post #458
snakeyjake wrote: post #344

Earlier quoted context omitted.

>Big Data today is not: I don't have enough storage or compute. It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities. That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices. We don't usually fill up the RAIDs so we gen…

> We can only fly every other day because it takes a day between flights to offload the data via fiber onto storage servers What’s preventing you from buying a second set of SSD drives and swapping them into planes end of day? The planes would then be able to fly every day while the previous day’s data is being offloaded on the ground, condensing the two weeks into one. Swaps could even be quite easy if you replace f…

>What’s preventing you from buying a second set of SSD drives and swapping them into planes end of day?

The arrays are custom parts made to interface directly with the radar and are VPX modules. VPX edge connectors are not rated for very many matings, believe me we've tried.

Every-other-day flights also keep the pilots comfortably within their sleep limits.

Re: Big data is dead (2023)

#469

Post by blagie »

pas wrote: post #461
blagie wrote: post #16

Earlier quoted context omitted.

It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a met…

> if you haven't outlined a clear path from 1-3 people in a metaphorical garage to reaching $1B, it almost never happens, and sometimes for stupid reasons most of the business plans are clear. acquire users and/or paying customers, build out the product, raise money and eventually raise prices too. engineering side is even simpler, keep the lights on, work with product designers to find incremental steps, iterate. as…

> they simple cannot fail due to inadequate engineering. (because worst case you fire the whole engineering department and just acqui-hire a random startup and you'll be back to growth in no time.)

> ...

> of course this take might be too hot, so I'm happy to read some counterarguments (maybe there are even counterexamples?)

Netscape.

(Plenty of others too)

Re: Big data is dead (2023)

#470

Post by coldtea »

the_real_cher wrote: post #452
Terr_ wrote: post #397

Earlier quoted context omitted.

I'm having flashbacks to some new outside-hire CEO making flim-flam about capex-vs-opex in order to justify sending business towards a contracting firm they happened to know.

Straight to jail

More likely straight to bonus and eventual golden parachute!
Post reply on HN