I'm trying to calibrate a working mindset to get into the industry from a bit of a distance, so thanks for the feedback - and for engaging my somewhat hastily written comment :)
I recently realized (light bulb moment) that cloud platforms like AWS are awesome for prototyping and experimentation: you can spin up seriously wide+deep instances that bill $1/hr, crunch through a complex workload in a couple of hours, and use hundreds of GB of RAM connected to hundreds of CPU cores for the cost of a stereotypical cup of coffee. But that bottoms out after just a very short period of time; a literal couple of hours. I wouldn't be surprised to learn the price scales was intentionally tuned to feel accessible at the prototyping stage - this would take advantage of communicative blind spots on both sides of the management/engineering boundary during evaluation and testing, where management gives engineering a green light budget to play around and learn about the platform, the engineers get insightful field experience after not very much expenditure, it feels possible to do a lot of work without spending too much, and... woops, a short time later they're staring down the barrel of a deadline, the fastest solution is to use their newfound experience and familiarity... and the cycle repeats.
The theoretical blind spot here is of course in the learning/tinkering stage that is so hard to communicate to management - and so when engineering makes positive noises about having some level of understanding and confidence about how to do a particular thing on AWS, the nuance of the hair-thin line between "thing that is generally achievable" and "I've practically played with AWS' implementation of the thing and I understand how to do it in that sandbox" is lost. (And the cycle repeats.) What's the right word for something incredibly well-engineered for altogether depressing reasons? :(
(Of course manglement and disorganization can also be at fault here, although I wonder where the cultural/structural root cause was when NASA forgot the egress bandwidth costs of migrating to AWS :D https://news.ycombinator.com/item?id=22626097)
Maybe the moral of the above story/thought experiment might be to theorize that AWS's market dominance has had a significant impact on broad tinkering and experimentation, and adjust for that by explicitly authorizing a broad-spectrum tinkering budget - "go and play" meets "engineering feasibility study" or something. Take the brakes off just enough that the engineering opinions that eventually come back are the product of having had the chance to stare into the horizon blankly for a bit, that sort of thing. Hypothetically.
Regarding the data intensity bit you mentioned, I'm personally up to the "just buy the whole computer" point at the moment mindset-wise, since this presents the most cost savings where it's feasible to do so. One somewhat cute but relatively transferable/normalizable way of broadly articulating the hard differences is to find vendor workstation/server configurators, adjust all the settings until the big number with the $ in front won't go any higher :), then (try and) configure something comparable on EC2.
An HP Z8 workstation with 2x28 core Xeon 8280s, 3TB of RAM, 172TB of storage, and 3 NVIDIA GPUs will set you back $111,575: https://zworkstations.com/products/hp-z8-workstation/?config...
An EC2 x2iedn.24xlarge (wat) with 96 CPUs, 3TB of RAM, 2 1.4TB SSDs, and 50TB outbound bandwidth, will set you back
- $192,625 for 1 year at standard monthly rates: https://calculator.aws/#/estimate?id=e9d103d78312322c281bc0c...
- $185,436 if you buy 1 year in advance: https://calculator.aws/#/estimate?id=c57064223f0aa745207efcf...
- $231,575 if you buy 3 years in advance: https://calculator.aws/#/estimate?id=2795a93261ed9353668b1bd...
An EC2 p4d.24xlarge (again, wat) with 96 CPUs, 1.1TB RAM, 8TB NVMe, 8 GPUs and no outbound transit (I couldn't figure out how to add any) is $196,398 for 1 year at standard monthly rates: https://calculator.aws/#/estimate?id=ef23ee773d298c703015453...
So this specific class of hardware would pay for itself in just over half a year ((111/192)*12=6.9375 months), give you total monopoly over resource availability, and redefine the bandwidth usage situation. The question then is whether the same sort of wet-fish disparity also exists within the target performance window you'd be aiming for.
In practice, datacenter colocation only charges for rack U height, power consumption and network bandwidth, regardless of the cost of the hardware (very cool) - and hmm, now I'm wondering if insurance options offer different premiums for different types of colocation facility standards - and then the only ongoing costs are having solid sysadmins (oh hey, remember those? lol) and inevitable hardware replacement (something something ZFS hot/warm spares).
The infuriating twist (sigh) is the ML part, and whether you want to do that on GPUs or otherwise.
I read a highly impressionable comment chain from someone who'd had some great engineering experiences playing with Google TPUs a few months back (https://news.ycombinator.com/item?id=27728225), and I wonder if the status quo described there has shifted at all - perhaps the scope of access has been reined in somewhat, or billing have come in and poured a pricing bucket over the engineering parade. I'm not sure. But what's stuck with me, sadly, about accelerator technology like this is that the cloud-computing theme of normalization-of-centralization unfortunately means this sort of hardware is only network-accessible from a large vendor like Google; you can't buy them. Infuriatingly. So even if the amazing-sounding prototyping opportunity described in the above link is still a thing (oh hey... the prototyping stage... this seems familiar ......), at production scale the only question that's on the table is "what are the pricing tiers?" since there's no leverage to be had, you either want the service at the price it's offered at or you don't.
Hence my question about using GPUs, which are, arguably, only barely at the break-even point in terms of buying vs renting for a lot of workloads. (Probably in large part only because enough large-scale scenarios demand on-premises provenance.) Because your alternatives are HaaS (hardware-as-a-service) for whatever the cloud providers offer. Yay.
I'm not entirely sure what you were referring to in terms of ML centralization - whether from a hardware or software standpoint - but I definitely go for the the "build a moat around it" mentality myself. I don't have enough experience to have the first clue what that would look like in practice though - just that it's a giant rabbithole of risk. (And a nuanced one, since Google et al want to generate demand for their product/service! Just... within the comfort zone of rent-seeking. Adlskdfglkjsdflgd *headdesk*)
In terms of what sounds like front/mid-end caching, Cloudflare currently seems to want to seriously differentiate on egress bandwidth, making it legitimately hard to dismiss them as interesting; but I do wonder what they'll do once they have both engineering familiarity and account entrenchment. I recall how Amazon Cloud Drive initially launched as both free and unlimited, then abruptly announced $60/TB pricing after people had uploaded TBs of data (and that one guy who uploaded 1 PB of test images). Cloudflare are certainly smart enough to see that similar actions would rapidly reorganize their reputation in a hurry, so I do wonder what their long-term strategy is there. Distilling the value-add down to basically "200+ PoPs and the world's lowest latency", it's totally possible to sustainably reproduce a heavy subset of that capability using commodity replication in a few key locations; and that may well suffice, supporting the argument that the value-add is not a critical requirement.