I believe the GrapheneOS team demands, in particular, a security feature called Memory Tagging Extensions (MTE). This feature has a phone's processor attach a "tag" to each memory location saying what's stored there and requires accesses to that memory to bear a matching tag. Buffer overflow type attacks are effectively prevented because a read into the adjoining region having a different tag gets rejected.
This feature is usually NOT present in desktop computers, and has only really been implemented by the Google Pixel hardware so far. It's unclear why it's viewed as being so critically necessary for Graphene: MTE is certainly useful but I personally wouldn't say its absence indicates an "insecure" device, and it imposes both a clear performance (and battery life!) penalty and a significant hardware burden on the SoC manufacturer. It's a pretty significant trade-off.
Anyway, that's the biggest single reason why GrapheneOS devs say "this hardware is not adequately secure": 99% of phone chips are excluded because they do not support MTE.