111–120 of 126 posts
Earlier quoted context omitted.
CERN is a correct example. The LHC reportedly generates 1PB per second: https://home.cern/news/news/computing/cern-data-centre-passe...
If you define “generates” to mean “discards” then yes.
Otherwise the tape alone to store it on would exceed their total operating budget in a day, so they have to be a bit clever about it.
Earlier quoted context omitted.
I've only done a few 30+TB moves, largest being 50TB which had similar table structure - one table per machine holding calibration readings that were constantly streamed in from a MUCH beefier RabbitMQ instance. Old CPU, not a ton of cores, spinning disks, etc. Same base problem. I think the best thing to do here would be change the order of some things around. 1. Buy the hardware first with blazing drives and roarin…
Having step 1 of your solution be "eliminate the hardware constraints that made this job painful" is kind of a cheat. We can all do a better job on any problem if we lift those constraints.
Earlier quoted context omitted.
Sorry what was that? https://www.spie.org/news/photonics-focus/mayjun-2020/square...
Ignoring the fantasy b.s. in the second half of the article, the stuff at the top is exactly what I mean. A mighty 400 GB/s: i.e. much less than the > 50 PB/day of logs the other person mentioned; 1600 hours of SD video per second: i.e. about 1-2 million concurrent HD streams, or much less than the amount actually served by YouTube. IBM Summit "world's most powerful supercomputer": < 5000 nodes, i.e. much below the m…
Earlier quoted context omitted.
Sorry what was that? https://www.spie.org/news/photonics-focus/mayjun-2020/square...
Ignoring the fantasy b.s. in the second half of the article, the stuff at the top is exactly what I mean. A mighty 400 GB/s: i.e. much less than the > 50 PB/day of logs the other person mentioned; 1600 hours of SD video per second: i.e. about 1-2 million concurrent HD streams, or much less than the amount actually served by YouTube. IBM Summit "world's most powerful supercomputer": < 5000 nodes, i.e. much below the m…
Earlier quoted context omitted.
Ignoring the fantasy b.s. in the second half of the article, the stuff at the top is exactly what I mean. A mighty 400 GB/s: i.e. much less than the > 50 PB/day of logs the other person mentioned; 1600 hours of SD video per second: i.e. about 1-2 million concurrent HD streams, or much less than the amount actually served by YouTube. IBM Summit "world's most powerful supercomputer": < 5000 nodes, i.e. much below the m…
it’s adorable how proud you are to have worked at FAANG and how angry you get at the idea some other organisation handles equivalent scale
Earlier quoted context omitted.
Having step 1 of your solution be "eliminate the hardware constraints that made this job painful" is kind of a cheat. We can all do a better job on any problem if we lift those constraints.
Hardware is way cheaper than 11 months salary + lost opportunity cost.