>Big Data today is not: I don't have enough storage or compute.
It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities.
That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices.
We don't usually fill up the RAIDs so we generate about 176TiB of data per day and over the two weeks we do 7 flight, or 1.2PiB per deployment or 7.2PiB per year.
We can only fly every other day because it takes a day between flights to offload the data via fiber onto storage servers that are usually haphazardly crammed into the corner of a hangar next to the apron. It is then duplicated to a second server for safekeeping and at the end of the mission everything is shipped back to our HQ for storage and processing.
The data is valuable, but not "billions" valuable. It is used for resource extraction, mapping, environmental and geodetic research, and other applications (but that's not my department) so we have kept every single byte for since 2008. This is especially useful because as new algorithms are created (not my department) the old data can be reprocessed to the new standard.
Entire nations finally know how many islands they have, how large they are, how their elevations are changing, and how their coasts are being eradicated by sea level change because of our data and if you've ever used a mapping application and flown around a city with 3d buildings that don't look like shit because they were stitched together using AI and photogrammetry, you've used our data too.
We have to use hard drives because SSDs would be space and most certainly cost prohibitive.
We stream 800GiB-2TiB files each representing a complete stripe or circular orbit to GPU-equipped processing servers. Files are incompressible (the cosmic microwave background, the bulk of what we capture, tends to be a little random) and when I started I held on to the delusion that I could halve the infrastructure by writing to tape until I found out that tape capacities were calculated for the storage of gigabyte-sized text files of all zeros (or so it seems) that can be compressed down to nothing.
GPUs are too slow. CPUs are too slow. PCIe busses are too slow. RAM is too slow. My typing speed is too slow. Everything needs to be faster all of the time.
Everything is too slow, too hard, and too small. Hard drives are too small. Tuning the linux kernel and setting up fast and reliable networking to the processing clusters is too hard. Kernel and package updates that aren't even bug fixes but just changes in the way that something works internally that are transparent to all users except for us break things. Networks are too slow. Things exist in this fantasy world where RAM is scarce so out-of-the-box settings are configured to not hog memory for network operations. No. I've got a half a terabyte of RAM in this file server use ALL OF IT to make the network and filesystem go faster, please. Time to spend six hours reading the documentation for every portion of the network stack to increase the I/O to 2024-levels of sanity.
I probably know more about sysctl.conf than almost every other human being on earth.
Distributed persistent object storage systems for people who think they are doing big data but really aren't either completely fall apart under our workload or cost hundreds of millions of dollars-- which we don't have. When I tell all of the distributed filesystem salespeople that our objects are roughly a terabyte in size they stop replying to my emails. More than one vendor has referred me to their intelligence community customer service representative upon reading my requirements. I am not the NSA, buddy, and we don't have NSA money.
Every once in a while we get a new MBA or PMP who read a Bloomberg article about the cloud and asks about moving to AWS or Azure after they see the costs of our on-premises datacenter. When I show them the numbers, in terms of both money and time, they throw up in their mouths and change the subject.
To top it all off all of our vendors are jumping on the AI/cloud bandwagon and discontinuing product lines applicable to us.
And now I've got to compete for GPUs with hedge funds and AI startups trying to figure out how to use a LLM to harvest customer data and use it to show them ads.
I do not have enough storage or compute, and the storage and compute I do have is too slow.
DPUs/IPUs look interesting but fall on their face when an object is larger than a SQL database query or compressed streaming video chunk.