Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

141–150 of 475 posts

Re: Big data is dead (2023)

#141

Post by kjkjadksj »

mattbillenstein wrote: post #76
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

Hes hiring data scientists not building a service though. This might realistically be a one off analysis for those 6tb. At which point you are happy your data scientists has returned statistical information instead of spending another week making sure the pipeline works if someone puts a greek character in a field.

Re: Big data is dead (2023)

#142

Post by kbolino »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Even if a 6 terabyte CSV file does fit in RAM, the only thing you should do with it is convert it to another format (even if that's just the in-memory representation of some program). CSV stops working well at billions of records. There is no way to find an arbitrary record because records are lines and lines are not fixed-size. You can sort it one way and use binary search to find something in it in semi-reasonable time but re-sorting it a different way will take hours. You also can't insert into it while preserving the sort without rewriting half the file on average. You don't need Hadoop for 6 TB but, assuming this is live data that changes and needs regular analysis, you do need something that actually works at that size.

Re: Big data is dead (2023)

#143

Post by kjkjadksj »

randomtoast wrote: post #118
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

For someone who is comfortable with sql we are talking minutes to hours to figure out awk well enough to see how its used or use it.

Re: Big data is dead (2023)

#145

Post by WesolyKubeczek »

fijiaarone wrote: post #115
WesolyKubeczek wrote: post #74

Earlier quoted context omitted.

I need to learn more about the latter for some log processing...

Log files aren’t data. That’s your first problem. But that’s the only thing that most people have that generates more bytes than can fit on screen in a single spreadsheet.

Everything is data if you are brave enough.

Re: Big data is dead (2023)

#146

Post by kjkjadksj »

LightFog wrote: post #59

When working in a research lab we used to have people boast that their analysis was so big it ‘brought down the cluster’ - which outed them pretty quickly to the people who knew what they were doing.

Must have been abusing the head node if they did that

Re: Big data is dead (2023)

#147

Post by actionfromafar »

pdimitar wrote: post #126
ImPostingOnHN wrote: post #103

Earlier quoted context omitted.

I remember replacing a CLI program built like Lego blocks. It was 90-100 LEGO blocks, written over the course of decades, in: Cobol; Fortran; C; Java; Bash; and Perl, and the Legos "connected" with environmental variables. Nobody wanted to touch it lest they break it. Sometimes it's possible to do things too smartly. Apache Spark runs locally (and via CLI).

No no, I didn't mean that at all. I meant a script using well-known CLI programs. Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.

Well your little CLI-query is suddenly in production and then... it easily escalates.

Re: Big data is dead (2023)

#148

Post by fock »

throwaway_20357 wrote: post #129
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

It depends on what you want to do with the data. It can be easier to just stick nicely-compressed columnar Parquets in S3 (and run arbitrarily complex SQL on them using Athena or Presto) than to try to achieve the same with shell-scripting on CSVs.

how exactly is this solution easier than putting the very Parquet files on a classic filesystem. Why does the easy solution require an amazon-subscription?

Re: Big data is dead (2023)

#149

Post by 7thaccount »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I am a big fan of these simplistic solutions. In my own area, it was incredibly frustrating as what we needed was a database with a smaller subset of the most recent information from our main long-term storage database for back end users to do important one-off analysis with. This should've been fairly cheap, but of course the IT director architect guy wanted to pad his resume and turn it all into multi-million project with 100 bells and whistles that nobody wanted.

Re: Big data is dead (2023)

#150

Post by surfingdino »

In my experience the only time I worked on a big data project was the public Twitter firehose. The team built an amazing pipeline and it did actually deal with masses of data. Any other team I've been on were delusional and kept building expensive and overcomplicated solutions that could be replaced with a single Postgres instance. The most overcomplicated system I've seen could not process 24hrs-worth of data in under 24 hours... I was happy to move on when an opportunity presented itself.
Post reply on HN