What users notice most is latency, not bandwidth. For example, if every page asset has an extra 1 second latency, then you are adding several seconds to your base page load time as all those 1 second delays stack up. First the main frame has to load, then if you have a subframe, the subframe has to load, then finally the content in it. Nest frames or other content, and suddenly your site seems sluggish for a single u…
the latency difference between fios and co-lo is going to be tiny. 10ms vs 50ms at most. (yeah, 5x difference, but it's still fifty friggen milliseconds)
L3 latency is usually a small fraction of the initial website load time. Inline the js + css, strip whitespace, enable gzip, use memcached, get a server that can fit the whole DB in RAM, etc.
Preventing packet loss is very important, though (a TCP retrans adds 2-3 seconds). I do think datacenters have a real edge there.