Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

281–290 of 475 posts

Re: Big data is dead (2023)

#281

Post by dahart »

__alexs wrote: post #85
mattbillenstein wrote: post #76

Earlier quoted context omitted.

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

A moderately powerful desktop processor has memory bandwidth of over 50TB/s so yeah it'll take a couple of minutes sure.

Running awk on an in-memory CSV will come nowhere even close to the memory bandwidth your machine is capable of.

Re: Big data is dead (2023)

#282

Post by noisy_boy »

kjkjadksj wrote: post #143
randomtoast wrote: post #118

Earlier quoted context omitted.

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

For someone who is comfortable with sql we are talking minutes to hours to figure out awk well enough to see how its used or use it.

It is not only about whether people can figure it out awk. It is also about how supportable the solution is. SQL provides many features specifically to support complex querying and is much more accessible to most people - you can't reasonably expect your business analysts to do complex analysis using awk.

Not only that, it provides a useful separation from the storage format so you can use it to query a flat file exposed as table using Apache Drill or a file on s3 exposed by Athena or data in an actual table stored in a database and so on. The flexibility is terrific.

Re: Big data is dead (2023)

#283

Post by elicksaur »

tomrod wrote: post #132
randomtoast wrote: post #118

Earlier quoted context omitted.

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

About $20/month for chatgpt or similar copilot, which really they should reach for independently anyhow.

Many people have noted how when using LLMs for things like this, the person’s ultimate knowledge of the topic is less than it would’ve otherwise been.

This effect then forces the person to be reliant on the LLM for answering all questions, and they’ll be less capable of figuring out more complex issues in the topic.

$20/mth is a siren’s call to introduce such a dependency to critical systems.

Re: Big data is dead (2023)

#284

Post by tored »

whamlastxmas wrote: post #234
kmarc wrote: post #209

Earlier quoted context omitted.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

I think more like, how would you prepare and cook the best five course gala dinner for only $10. That requires true skill.

Re: Big data is dead (2023)

#285

Post by StrLght »

apwell23 wrote: post #252
pdimitar wrote: post #249

Earlier quoted context omitted.

The classic putting words in people's mouths technique it is then. The good old straw man. If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]". I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.

> that people don't try simpler and proven solutions FIRST Well why don't people do that according to you ? Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.

> you can never guess what angle interviewer is coming at you

Why would you guess in that situation though?

It’s an interview, there’s at least 1 person talking to you — you should talk to them, ask them questions, share your thoughts. If you talking to them is a red flag, then high chances that you wouldn’t want to work there anyway.

Re: Big data is dead (2023)

#286

Post by disqard »

noisy_boy wrote: post #274
rqtwteye wrote: post #244

Earlier quoted context omitted.

Plenty of people get offended if you tell them that their data isn’t really “big data”. A few years ago I had a discussion with one of my directors about a system IT had built for us with Hadoop, API gateways, multiple developers and hundreds of thousands of yearly cost. I told him that at our scale (now and any foreseeable future) I could easily run the whole thing on a USB drive attached to his laptop and a few pyt…

In most non-tech companies, it comes down to the motive of the manager and in most cases it is expansion of reporting line and grabbing as much budget as possible. Using "simple" solutions runs counter to this central motivation.

This is also true of tech companies. Witness how the "GenAI" hammer is being used right now at MS, Google, Meta, etc.

Re: Big data is dead (2023)

#287

Post by Stranger43 »

apwell23 wrote: post #253
Stranger43 wrote: post #197

Earlier quoted context omitted.

And here we see this strange thing that data science people does in forgetting that 6TB is small change for any SQL server worth it's salt. Just dump it into Oracle, postgre, mssql, or mysql and be amazed by the kind of things you can do with 30year old data analysis technology on an modern computer.

you wouldn't have been a 'winner' per OP. real answer is loading it on their phones not on sqlserver or whatever.

To be honest OP is kind of making the same mistake in assuming that the only real alternatives is "new data science products" and old school scripting exists as valuable tools.

The extend people goes to to not recognize how much the people creating the SQL language and the relational database engines we now take for granted actually knew what they were doing, are a bit of an mystery to me.

The right answer to any query that can be defined in SQL is pretty much always an SQL engine even if it's just sqlite running on an laptop. But somehow people seems to keep comming up with reasons not to use SQL.

Re: Big data is dead (2023)

#288

Post by jjgreen »

mrtimo wrote: post #168
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Please promote the use of .parquet files!

  apt-cache search parquet
Maybe later

Re: Big data is dead (2023)

#289

Post by torginus »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

It's astonishing how shit the cloud is compared to boring-ass pedestrian technology.

For example, just logging stuff into a large text file is so much easier, performant and searchable that using AWS CloudWatch, presumably written by some of the smartest programmers who ever lived.

On another note I was once asked to create a big data-ish object DB, and me, knowing nothing about the domain, and a bit of benchmarking, decided to just use zstd-compressed json streams with a separate index in an sql table. I'm sure any professional would recoil at it in horror, but it could do literally gigabytes/sec retrieval or deserialization on consumer grade hardware.

Re: Big data is dead (2023)

#290

Post by valenterry »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

It doesn't matter. The answer should be "It depends, what are the circumstances - do we expect high growth in the future? Is it gonna stay around 6TB? How and by whom will it be used and what for?"

Or, if you can guess what the interviewer is aiming for, state the assumption and go from there "If we assume it's gonna stay at Then the interviewer can interrupt and change the assumptions to his needs.

Post reply on HN