The article ends with the question about why the bandwidth is capped.
I think the answer could be mclock scheduler.
https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
SSDs have become fast, except in the cloud
341–350 of 427 posts
Re: SSDs have become fast, except in the cloud
#342Earlier quoted context omitted.
Do you remember the companies that did not scale? friendster did well until it failed to scale, and Facebook took over. So the converse argument might be: don't bungle it up because you failed to plan. Provision for at least 10x growth with every (re-)implementation. https://highscalability.com/friendster-lost-lead-because-of-...
Hold on... You think Facebook took over from Friendster because of scaling problems?! MySpace was the one that took the lead over Friendster and it withered after it got acquired for $500 million by news corp because that was the liquidity event. That's when Facebook gained ground. Your timeline is wrong. The MySpace switch was because of themes and other features the users found more appealing. Twitter had similar c…
How sure are you that they switched because of themes? Did you see user research? I left because of its poor performance, and MySpace was no substitute for friendster; it targeted an artsy demographic. But Facebook was.
Re: SSDs have become fast, except in the cloud
#343Earlier quoted context omitted.
This is untrue of Local SSD ( https://cloud.google.com/local-ssd ) in Google Cloud. Local SSDs are PCIe peripherals, not network attached. There are also multiple Persistent Disk ( https://cloud.google.com/persistent-disk ) offerings that are backed by SSDs over the network. (I'm an engineer on GCE. I work directly on the physical hardware that backs our virtualization platform.)
It's notable that your second link has a screenshot for 24(!) NVMe SSDs totalling 9 terabytes, but the aggregate performance is 2.4M IOPS and 9.3 GB/s for reads. In other words, just 100K/400MB per individual SSD, which is very low these days. For comparison, a single 1 TB consumer SSD can deliver comparable numbers (lower IOPS but higher throughput). If I plugged 24 consumer SSDs into a box, I would expect over 30M…
I should've aimed for more clarity in my original comment -- the first link is to locally attached storage. The second is network attached storage (what the GP was likely referring to, but not what is described in the article).
Persistent Disk is not backed by single devices (even for a single NVMe attachment), but by multiple redundant copies spread across power and network failure domains. Those volumes will survive the failure of the VM to which they are attached as well as the failure of any individual volume or host.
Re: SSDs have become fast, except in the cloud
#344Earlier quoted context omitted.
That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking. As a hint for you, I said " a network", not " the network." You can also look at public presentations about how Nitro works.
I've linked to public documentation that is pretty clearly in conflict with what you said. There's no wiggle room in how AWS describes their service without it being false advertising. There's no "ah, but what if we define the entire building to be the host computer, then the networked SSDs really are inside the host computer" sleight of hand to pull off here. You've provided cryptic hints and a suggestion to watch s…
I see wiggle room in the statement you posted in that the SSD storage that is physically inside the machine hosting the instance might be mounted into the hypervised instance itself via some kind of network protocol still, adding overhead.
Re: SSDs have become fast, except in the cloud
#345Well, as hardware becomes more and more powerful, what's possible in a small footprint becomes bigger and bigger. And distributed software for disaggregated storage is becoming more accessible. You put these two together, running on-prem footprints at the scale of 50-100M$ capex makes a lot of sense. In my personal experience, at this scale, (if your cloud bill for compute+storage+local-network is $50M+/year), you can get 2-4x more on-prem private-cloud capacity for the same money. Of course, this only makes sense if you have an in-house software engineering team already and the marginal cost of adding another 50-100 engineers to build and operate this is strategically valuable to your business.
In big data AI space, this is exactly what's happening with the top 20th to 100th companies in the world right now.
Re: SSDs have become fast, except in the cloud
#346Earlier quoted context omitted.
Hold on... You think Facebook took over from Friendster because of scaling problems?! MySpace was the one that took the lead over Friendster and it withered after it got acquired for $500 million by news corp because that was the liquidity event. That's when Facebook gained ground. Your timeline is wrong. The MySpace switch was because of themes and other features the users found more appealing. Twitter had similar c…
How sure are you that they switched because of themes? Did you see user research? I left because of its poor performance, and MySpace was no substitute for friendster; it targeted an artsy demographic. But Facebook was.
Yes. I worked in social networks 15 years ago. It was a heavy research topic for me.
You're a highly technical user. Non-technical people are weird - part of the MySpace exodus was the belief that it spread "computer viruses", really
There was more to the switches but I'd have to dredge it up probably through archive sites these days. The reasons the surveys supported I considered ridiculous but it doesn't matter it's better to understand consumer behavior - we can't easily change it.
Especially these days. It was not possible for me to be a teenager with high speed wi-fi when I was one 30 years ago. I've got near zero understanding of the modern consumer youth market or what they think. Against all my expectations I've become an old person.
Anyways, the freeform HTML was a major driver - it was geocities with less effort, which had also exited through a liquidity event and currently has a clone these days https://neocities.org/browse
Re: SSDs have become fast, except in the cloud
#347This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…
It does not fundamentally have to be. That's an architectural choice driven by cloud providers backing off from the "instances can die on you" choice that AWS started with and then realized customers struggled with, towards attempting to keep things running in the face of many types of hardware failures.
Which is ironic given that when building on-prem/colo'ed setups you'll replicate things to be prepared for unknown lengths of downtime while equipment is repaired or replaced, so this was largely cloud marketing coming bak to bite cloud providers' collective asses. Not wanting instances to die "randomly" for no good reason does not always mean wanting performance sacrifices for the sake of more resilient instances.
But AWS at least still offers plenty of instances with instance storage.
If I'm setting up my own database cluster, while I don't want it running on cheap consumer-grade hardware without dual power supplies and RAID, I also don't want to sacrifice SSD speed for something network-attached to survive a catastrophic failure when I'm going to have both backups, archived shipped logs and at least one replica anyway.
I'd happily pick network-attached storage for many things if it gets me increased resilience, but selling me a network-attached SSD, unless it replicates local SSD performance characteristics, is not competitive for applications where performance matters and I'm set up to easily handle system-wide failures anyway.
Re: SSDs have become fast, except in the cloud
#348Earlier quoted context omitted.
How much faster would the network need to get, in order to meet (or at least approach) the speed of a local SSD? are we talking about needing to 2x or 3x the speed, or by factors of hundreds or thousands?
The Samsung 990 in my desktop provides ~3.5 GB/s streaming reads, ~2 GB/s 4k random-access reads, all at a latency measured at around 20-30 microseconds. My exact numbers might be a little off, but that's the ballpark you're looking at, and a 990 is a relatively cheap device. 10GbE is about the best you can hope for from a local network these days, but that's 1/5th the bandwidth and many times the latency. 100GbE wou…
Networking doesn't have to have high latency. You can buy network hardware that is able to provide sub-microsecond latency. Physical distance still matters, but 10% of typical NVMe latency gets you through a kilometer of fiber.
Re: SSDs have become fast, except in the cloud
#349Earlier quoted context omitted.
Thanks for the details! Does this mean you're colocating your own server in a data center somewhere? Or do you have your own data center/running it off a bare metal server with a business connection? Just wondering if the TCO included the same levels of redundancy and bandwidth, etc.
We were colocated in large data centers right on the major IX with redundancy. All of this was accounted for in their TCO model. We had a better switch fabric than is typical for the cloud but that didn’t materially contribute to cost. We were using AWS for overflow capacity when we exceeded the capacity of our infrastructure at the time; they wanted us to move our primary workload there. The difference in cost could…
> All of this can be modeled out in Excel. No one does it anymore but I am from a time when it was common, so I have that skill in my back pocket. It isn’t nearly as much work as it sounds like, much of the details are formulaic. You do need to have good data on how your workload uses hardware resources to know what to build.
And this is one of the big "screts" AWS success: Shifting a lot of resource allocation and power from people with budgeting responsibility to developers who have usually never seen the budget or accounts, don't keep track, and at most retrospectively gets pulled in to explain line items in expenses, and obscuring it (to the point where I know people who've spent 6 figure amounts worth of dev time building analytics to figure out where their cloud spend goes... tooling has gotten better but is still awful)
I believe a whole lot of tech stacks would look very different if developers and architects were more directly involved in budgeting, and bonuses etc. were linked at least in part to financial outcomes affected by their technical choices.
A whole lot of claims to low cloud costs come from people who have never done actual comparisons and who seem to have a pathological fear of hardware, even when for most people you don't need to ever touch a physical box yourself - you can get maybe 2/3's of the savings with managed hosting as well.
You don't get the super-customized server builds, but you do get far more choice than with cloud providers, and you can often make up for the lack of fine-grained control by being able to rent/lease them somewhere where the physical hosting is cheaper (e.g. at a previous employer what finally made us switch to Hetzner for most new capacity was that while we didn't get exactly the hardware we wanted, we got "close enough" coupled with data centre space in their locations in Germany being far below data centre space in London - it didn't make them much cheaper, but it did make them sufficiently cheaper to outweigh the hardware differences with a margin sufficient for us to deploy new stuff there but still keep some of our colo footprint)
Re: SSDs have become fast, except in the cloud
#350Earlier quoted context omitted.
My experience lately is that consumer drives will also lie and use a cache, but then drop your data on the floor if the power is lost or there’s a kernel panic / BSOD. (Samsung and others.)
Rumors of that. I've never actually seen it myself.
I can get it to happen easily. 970 Evo Plus. Write a text file and kill the power within 20 seconds or so, assuming not much other write activity. File will be zeroes or garbage, or not present on the filesystem, after reboot.