There is one instance where I saw you take a shared_ptr and then proceed to move from it. That could literally have been a unique_ptr and if you are making an assumption that that shared_ptr is still valid in another part of the codebase you at best now have UB but it is almost certainly a bug.
Just replace shared_ptr with unique_ptr, the advantage of managed languages is that you are in control of the memory, to then proceed to just use ref counting means you may as well have written the code in Java / C# since you incur all the same overhead of atomic references and don't get the advantages of a sophisticated GC.
unique_ptr can also trivially be upgraded to shared if you actually do need ref counting.