Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

431–440 of 444 posts

Re: Big data is dead

#431

Post by krab »

noobermin wrote: post #430
krab wrote: post #391

Earlier quoted context omitted.

That's a different category of big data. I worked for a big pharma and they were building their big data department with Spark and friends. I was quite surprised that their biggest dataset had something like 200 GB. At the same time, though, there was a lot of DNA sequencing data, we were designing CRISPR probes etc. But Spark and Hadoop aren't really that helpful in this area, so the Big Data team wasn't involved in…

I think it does depend on the problem. Genetic stuff always seemed not easily parallelizable like my field (physics simulation) is. That said, the culture here is that MPI works and thus cray still builds computers that work better with it, so we use MPI so it works...etc etc.

> Genetic stuff always seemed not easily parallelizable

Not really. Processing one sample may take a few hours but if you have hundreds or more samples, it's an obvious axis for independent parallelization.

The cool kids use Nextflow or CWL these days. It's something like `make` - it remembers what you've already computed and what the dependencies are - but it uses a batch engine like SGE/Condor/AWS Batch to actually execute the jobs.

Re: Big data is dead

#432

Post by noisy_boy »

Loughla wrote: post #417
noisy_boy wrote: post #382

Earlier quoted context omitted.

Not who you are replying to. I found your example fascinating - would you be able to share one or two concrete examples of this integration you mentioned? I would like a low-level peek or two into how the teaching landscape is changing in light of the rise of LLMs like chatGPT.

Sure - I'll use one of the creative writing classes. In the past, the class would be centered around ideas and themes the class came up with together during the first week of the semester. They would then read and discuss short stories from various authors centered around that theme, preferably from different eras and/or cultures. From there, they would work in pairs/small groups to flush out original ideas they came…

I agree a lot with your analysis - thanks for sharing your insights.

Re: Big data is dead

#433

Post by victorNicollet »

crabbone wrote: post #323
0xB31B1B wrote: post #62

Earlier quoted context omitted.

There are functionally less than 1000 organizations that currently require distributed compute for data analysis. You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. The cost of compute has decreased faster than the amount of data we have to store and process. What we used to do with spark jobs we can do with python on a single box.

> You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. Hold your horses... the beefiest servers that are in production today, unless you count custom-made stuff go to somewhere between 128 and 256 cores per board. These are hugely expensive. Also, I don't know if you can rent those from Amazon. Typical, affordable servers range between 4..16 cores. Doesn't matter if you buy them you…

Azure has the LS series of VMS [1] which can have up to ten 1.92TB disks attached directly to the CPU using NVMe. We use these for spilling big data to disk during high-performance computation, rather than single-machine persistence, so we also don't bother with RAID replication in the first place.

Though it is a bit disappointing that while Microsoft advertises this as "good for distributed persistent stores", there are no obvious SLAs that I could rely on for actually trusting such a cluster with my data persistence.

[1]: https://learn.microsoft.com/en-us/azure/virtual-machines/las...

Re: Big data is dead

#434

Post by crabbone »

victorNicollet wrote: post #433
crabbone wrote: post #323

Earlier quoted context omitted.

> You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. Hold your horses... the beefiest servers that are in production today, unless you count custom-made stuff go to somewhere between 128 and 256 cores per board. These are hugely expensive. Also, I don't know if you can rent those from Amazon. Typical, affordable servers range between 4..16 cores. Doesn't matter if you buy them you…

Azure has the LS series of VMS [1] which can have up to ten 1.92TB disks attached directly to the CPU using NVMe. We use these for spilling big data to disk during high-performance computation, rather than single-machine persistence, so we also don't bother with RAID replication in the first place. Though it is a bit disappointing that while Microsoft advertises this as "good for distributed persistent stores", there…

Well, attaching 10 PCIe devices is going to give a very hard time to your CPU if all of them should be used. The speed of copying from memory or between devices will become a bottleneck. Another problem is that on such a machine you will also need huge amount of memory to allow for copying to work. And, if you want this to work well, you'd need some high-end hardware to be actually able to pull that off. In such a system, your CPU will prevent you from exploiting the possible benefits of parallelization. It seems beefy, but it's entirely possible that a distributed solution you could build with a fraction of the cost would perform just as well.

This situation may not be reflected in Azure pricing (the calculator gives 7.68 $/h for L80as_v3) since if MS has such hardware, it would be a waste for it to stand idle. They'd be incentivized to rent it out event at a discount (their main profit is from traffic anyways). So, you may not be getting an adequate reading of the situation, if you are trying to judge it by the price (rather than cost). But, this is only the price of the VM, I'm scared to think about how much you'd pay if you actually utilize it to its full potential.

Also, since it claims to have 80 vCPUs, well... it's either a very expensive server, or it's, again, a distributed system, where you simply don't see the distributed part. I haven't dealt with such hardware firsthand, but we have in our DC a Gigaio PCIe TOR switch which would allow you to have that much memory (in principle, we don't use it like that) in a single VM. That thing with the rest of the hardware setup costs some six-digit number of dollars. I imagine something similar must exist for CPU sharing / aggregation.

Re: Big data is dead

#435

Post by crabbone »

victorNicollet wrote: post #433
crabbone wrote: post #323

Earlier quoted context omitted.

> You can get off the shelf AWS units with 1000 cores, terabytes of ram and storage, etc. Hold your horses... the beefiest servers that are in production today, unless you count custom-made stuff go to somewhere between 128 and 256 cores per board. These are hugely expensive. Also, I don't know if you can rent those from Amazon. Typical, affordable servers range between 4..16 cores. Doesn't matter if you buy them you…

Azure has the LS series of VMS [1] which can have up to ten 1.92TB disks attached directly to the CPU using NVMe. We use these for spilling big data to disk during high-performance computation, rather than single-machine persistence, so we also don't bother with RAID replication in the first place. Though it is a bit disappointing that while Microsoft advertises this as "good for distributed persistent stores", there…

Ha! On a side note (from the page you linked):

> The high throughput and IOPS of the local disk makes the Lasv3-series VMs ideal for NoSQL stores such as Apache Cassandra and MongoDB.

This is cringe-worthy. Cassandra is an abysmal quality product when it comes to performing I/O. It cannot saturate the system at all... I mean, for example, if you take old-reliable PostgreSQL or MySQL, then with a lot of effort you may get them to dedicate up to 30% CPU time to I/O. Where the reason for relatively low utilization (compared to direct writes to disk) is the need to synchronize that's not well-aligned with how the disk may want to deal with destaging.

Cassandra is in a class of its own when it comes to I/O. You'd be happy to hit 2-3% CPU utilization in the same context where PostgreSQL would hit 30%. I have no idea what it's doing to cause such poor performance, but if I had to guess, some application logic... making some expensive calculations sequentially with I/O, or just waiting in mutexes...

So, yeah... someone who wanted Cassandra to perform well would probably need that kind of a beefy machine :D But whether that's a sound advise -- I don't know.

Re: Big data is dead

#436

Post by mcphage »

rxhernandez wrote: post #238
jacobn wrote: post #117

Earlier quoted context omitted.

This sounds a lot like how my kids will listen to a teacher/coach, but not their parents...

Which is similar to how a lot of parents won't listen to their kids but will listen to the coach, teacher, or priest.

My Mother-in-Law was called by a tech support scammer. Her bank was unwilling to accept their charges, and the scammer wanted her to call the bank to tell them to accept them anyway. My Brother-in-Law was telling her "no, this is a scam, do not do this", but she was unwilling to listen. Eventually, he told her to call me, thinking if she wouldn't listen to her son, maybe she'd listen to her son-in-law. Which she did.

Re: Big data is dead

#437

Post by ayewo »

gingerwizard wrote: post #407
ayewo wrote: post #401

Earlier quoted context omitted.

Really neat that you scour job postings to learn useful intelligence about companies using your product. I do this too :) I'm curious how you have this set up. Is it currently a manual process or you use social monitoring tools to help you find mentions of ClickHouse in the wild?

Just use ClickHouse :) https://sql.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTS...

Thanks for the reply :-) but your link is only for tracking mentions on the HN website.

I was asking about how they are able to track mentions, across the web, of companies using ClickHouse. This type of info is usually listed in the tech stack section of job descriptions (and these links tend to expire once the position is filled).

Re: Big data is dead

#438

Post by swyx »

KaiserPro wrote: post #151

big data isn't big anymore. 1) 10 years ago, having access to 300tb of data that could sustain 10gigabytes/s of throughput would require something like two racks of disks with some SSD cache and junk. 2) people thought hadoop was a good idea 3) People assumed that everything could be solved with map:reduce 3) machine learning was much less of a thing. 4) people realised that postgres does virtually everything that mo…

what is your current explanation for why hadoop turned out NOT to be a good idea and everything couldn't be solved with map:reduce?

Re: Big data is dead

#439

Post by pelatimtt »

Agree. And thing I noticed is that tools like #apache spark have become the de-facto standard for any data engineer work even when data size does not require it. Result is that many jobs are much harder to mantain and often slower (due to all the shuffling) than running on a single node.
Post reply on HN