I don't buy the speed argument. In a modern PC, many things are pluggable. PCI express can transmit up to 30 GB/s, IIRC. Even CPUs and RAM are modular. If there was such a benefit from soldering them together, I'm sure we would be seeing more integrated systems.
Furthermore, the modem part of many smartphones is already a black box that is heavily isolated from every other component. It's basically just hooked into a bus, and connected to microphone, speakers, and antennas, and doesn't necessarily interface that much with the CPU.
I'm not sure the PhoneBloks concept would work. (Crazy ideas follow.) But what I can definitely see working is separating the computer from the shell (case, screen, speakers, camera, ...). We are reaching a point where the speed difference between smartphone generations becomes meaningless. In the future, you might be able to buy standardized "computing bricks". For one phone, you might need a 2017 model A brick, for a tablet at least a model C, and if you want to build a server, you just pick up a few (kilos of?) CPU bricks. These bricks would be so cheap and featureless, that they would allow really ubiquitous computing. Building a house? Build in a few, like power sockets. You'll get the equivalent of today's top PCs thrown after you as advertizement gifts.
This is not necessarily what must happen, but the reason I think it could happen is the following: Once the raw computing sector stops growing, and there is no longer much profit to make, the industry will want to move on. As we are already seeing today, specs will not be the main selling point. Instead, people will pay for features (a great camera, physical buttons :-), wireless charging, ...), a better integrated experience, a certain design, or a brand name (e.g. all Apple). Development will focus even more on that, and there will be a drive to reduce the cost of raw computing to it's minimum.