Pools delegate small pieces of work to individual miners, who join at will, and assemble the results of their work in order to find blocks. When a block is found, the pool divides the reward amongst the miners that contributed hash power toward finding it.
That's all handled internally. The Bitcoin network sees the pool as any other node; there is a hard stop there. It doesn't know that the answer was found by compounding the hash power of many servers.
As another commenter said, that alone is a risk, because you have (presumably) thousands of servers masked behind 5 central brokers. They are, in effect, the centralized banking cartel, and the pool participants are like their branches, ancillary components that are used to help perform the work, but which ultimately are at the mercy of those in the central office.
Consider that in late 2015, at the Scaling Bitcoin Hong Kong conference, the managers of all of these pools met together to attempt to devise a strategy for the blocksize problem. The controllers of what at the time was over 90% of btc's hash power were in the same room. [1]
Since miners stand to benefit from a limited blocksize and normal consumers stand to suffer, you can guess where their interests aligned. How is this different from the executive teams of the large banks getting together and agreeing to collaborate on other efforts?
Less likely, but much more worrying, is the question of what would've happened if the Chinese government incidentally decided to declare running a bitcoin pool illegal that day (I know that HK has an independent-for-now government, but Beijing has made incursions before, and is getting increasingly aggressive as the 50-year integration timeline narrows). These people could've been arrested and their resources could've been confiscated.
Beyond the concerns of a pool operator getting compromised, we also have to worry about the general matter of pool transparency. There's no way to see whose hash power is being contributed into the pool (afaik). We don't know how much of that hash power is from independently-operating nodes. Guaranteed very little of this is occurring on commodity hardware; most of it is in data centers in China, running custom mining hardware.
Initially, people attempted to produce and distribute consumer-level miners, but that has more-or-less burned out because the ROI is immediately decimated. There is still a small amount of interest in it for the novelty, but no one expects to make money from it.
As soon as the network gets an appreciable increase in hash power, like that which would be precipitated by the wide distribution of fast miners, the miner becomes worthless at the next difficulty evaluation. That's how the protocol works, and it means that consumer-level bitcoin hardware is a pipe dream.
The incentive is clearly to develop the fastest hardware possible for oneself, and then to prevent as many others as possible from getting something similar. This means proprietary hardware. Custom ASICs are extremely expensive to develop, especially at small scale, and only the biggest players are going to be able to do this. btc long ago blew away CPUs, FPGAs, and GPUs.
I don't think that was the intention, since it obviously leads to centralization and secrecy, exactly the problem set btc was trying to solve.
Since we can't see into the pools, we have no way to know where that hash power actually lies. It'd be very dumb to fire up uber-secret hardware with unprecedented hashing performance and not pretend that there's a pool in front of it. It's possible, I would even say likely, that a huge portion of the hash power in these pools comes from a small number of big contributors, potentially even the pool's operators themselves.
btc was originally envisioned as something everyone would run on their home computer, and it would be as decentralized as the internet was. Due to multiple design flaws, it's failed horrendously in that, and is now very centralized (and not just in mining).
I know there are a lot of nuances to btc and I don't claim to be a btc expert, so after you downvote, please correct me. ;)
[1] https://news.bitcoin.com/scaling-bitcoin-workshop-hong-kong-...