I can't speak for fly.io, but as someone who architects a lot of systems, you learn by doing, failing, and doing again. If you're smart your failures will all be proofs-of-concept rather than real-world, but some amount of real-world failure is inevitable.
You must learn how to learn from mistakes, because the more typical human reaction to failure is to get emotional and try to rationalize the failure away (including blaming others). You need to become ruthlessly analytical. Good analytical intuition can only come from experience/failure so you can't rush it.
Start with simple/straight forward problems to solve. For example, design an architecture for a headless, stateless API server. Start with a single instance exposed directly to the internet, and then put a load balancer in front of it and scale it up to at least 3 instances (horizontal scaling).
Now add a relational database (like postgres or mysql). Now make the server stateful (only for practice, avoid stateful services this like the plague in real life).
Now add a websocket to the server (which will need to go through the load balancer).
Now add a UI. Start with a few server-rendered pages, and then add a SPA. Try serving the SPA from the server app, and also try separating it into two distinct, independently deployable apps.
Now add some UDP to the app (adding WebRTC is an easy way to do this without having to write a lot of code).
Now add asynchronous jobs (sometimes called delayed jobs) just using the existing database.
Now add a queue system (like redis) for the jobs to communicate through and scale up the number of job instances.
Now add logging and metrics collection. I recommend EFK stack for logging and Prometheus/Grafana for metrics. At this point if you haven't used containers or kubernetes yet, it's probably a good time to repeat the whole process in k8s.
Now start adding microservices! Start with simple ones and get increasingly complex. Add a service that is never accessed directly by users, and put mTLS in front of it. Make sure that your services are aggregating logs to your logging solution and metrics are being collected.
Now add some services that use gRPC, protobufs, etc. You already have some non-HTTP in the form of UDP, but try adding some non-HTTP TCP based services. An SMTP server is a good challenge. You'll have to start getting creative!
To be well rounded, try using various products/approaches and see how they solve problems you have. It's good to do things "the hard way" (such as setup manually on VMs) to learn how they work, but IRL you won't want to do everything that way. Kubernetes is exploding for a reason. Make sure you learn some industry tools/solutions as well. I prefer open source to vendors as the vendors often obscure all the learning (which can be good for a real company, but isn't good for your goal of learning).
As you get better and more experienced, many of these scenarios can be done entirely as thought experiments, although if you are doing it for real, you should try to build a PoC (proof of concept) for anything that hasn't been proven before.
Lastly, bounce ideas off of people with experience! Be prepared to have your ideas challenged, and be prepared to critically think about others ideas and challenge them! In the real world you will rarely design it all yourself.
Pretty soon, you will be able to design and build complex systems. Also the Google SRE book(s) can be really helpful.
Holy cow this went a lot longer than I intended. I think I'll turn this into a blog post.