Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

71–80 of 475 posts

Re: Big data is dead (2023)

#71

Post by pdimitar »

WesolyKubeczek wrote: post #69
pdimitar wrote: post #62

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

> just go wild with CLI programs, I know dozens that work on CSV and XML ...or put it into SQLite for extra blazing fastness! No kidding.

That's included in CLI tools. Also duckdb and clickhouse-local are amazing.

Re: Big data is dead (2023)

#72

Post by silvestrov »

Cyberdog wrote: post #36
silvestrov wrote: post #7

Earlier quoted context omitted.

It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.

Assuming you're serious for a moment, I don't think AI is really a practical tool for working with big data. - The "hallucination" factor means every result an AI tells you about big data is suspect. I'm sure some of you who really understand AI more than the average person can "um akshually" me on this and tell me how it's possible to configure ChatGPT to absolutely be honest 100% of the time but given the current s…

A comment from the old post: https://news.ycombinator.com/item?id=34696065

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs.

I think AI is used for the same purpose in companies: signal to the world that the company is using the latest tech and internally for supporting existing political beliefs.

So same job. Hallucination is not a problem here as the AI conclusions are not used when they don't align to existing political beliefs.

Re: Big data is dead (2023)

#73

Post by nevi-me »

pdimitar wrote: post #62
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

To be fair on candidates, CLI programs create technical debt the moment they're written.

A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.

Re: Big data is dead (2023)

#74

Post by WesolyKubeczek »

pdimitar wrote: post #71
WesolyKubeczek wrote: post #69

Earlier quoted context omitted.

> just go wild with CLI programs, I know dozens that work on CSV and XML ...or put it into SQLite for extra blazing fastness! No kidding.

That's included in CLI tools. Also duckdb and clickhouse-local are amazing.

I need to learn more about the latter for some log processing...

Re: Big data is dead (2023)

#75

Post by qaq »

boppo1 wrote: post #51
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

You have 6 TiB of ram?

You can have 8TB RAM in a 2U box for under 100K. grab a couple and it will save you millions a year compared to over-engineered bigdata setup.

Re: Big data is dead (2023)

#76

Post by mattbillenstein »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always.

I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you can connect a plethora of other tooling to it (Metabase, Tableau, etc). And if partitioned correctly, it shouldn't be too expensive - add in rollup tables if that becomes a problem.

Re: Big data is dead (2023)

#77

Post by pdimitar »

nevi-me wrote: post #73
pdimitar wrote: post #62

Earlier quoted context omitted.

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

To be fair on candidates, CLI programs create technical debt the moment they're written. A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.

True on the premise, yep, though I'm not sure how using CLI programs like LEGO blocks creates a tech debt?

Re: Big data is dead (2023)

#78

Post by Joel_Mckay »

geraldwhen wrote: post #46
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I ask a similar question on screens. Almost no one gives a good answer. They describe elaborate architectures for data that fits in memory, handily.

People can always find excuses to boot candidates.

I would just back-track from a shipped product date, and try to guess who we needed to get there... given the scope of requirements.

Generally, process people from a commercially "institutionalized" role are useless for solving unknown challenges. They will leave something like an SAP, C#, or MatLab steaming pile right in the middle of the IT ecosystem.

One could check out Aerospike rather than try to write their own version (the dynamic scaling capabilities are very economical once setup right.)

Best of luck, =3

Re: Big data is dead (2023)

#79

Post by chx »

boppo1 wrote: post #51
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

You have 6 TiB of ram?

If my business depended on it? I can click a few buttons and have a 8TiB Supermicro server on my doorstep in a few days if I wanted to colo that. EC2 High Memory instances offer 3, 6, 9, 12, 18, and 24 TiB of memory in an instance if that's the kind of service you want. Azure Mv2 also does 2850 - 11400GiB.

So yes, if need to be, I have 6 TiB of RAM.

Re: Big data is dead (2023)

#80

Post by _ugfj »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

https://x.com/garybernhardt/status/600783770925420546 (Gary Bernhardt of WAT fame):

> Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.

This is from 2015...

Post reply on HN