There were two notable challenges for me, though not everyone in my class experienced these. Some of it will also be limited if you only have them there through 6th grade.
The first was figuring out how to socially transition from a small private school that effectively functioned like a big family, to a large public school. I didn’t really learn how to meet people being at the same small school for so long, and the culture shock of leaving of leaving Montessori was pretty bad. (Both with the way people my age approached learning and the way the public school system operated, i.e. everything is for a grade nothing else matters).
The second is because of some combination of the curiosity driven learning and the particular sequence of teachers I had, I managed to avoid getting a good foundation in algebra because I didn’t really feel like it. This turned out to be a big problem for me for many years, partially because no one realized so I kept being put in higher math classes, doing well through brute force, and being very frustrated and confused about all of it. I’m now getting an advanced degree in math, so everything worked itself out, but I do think math specifically wants a little more structure at those early steps than what I got.
Overall it was still absolutely worth it for me. Sometimes I wonder who I would be now if my curiosity hadn’t been so strongly reinforced when I was young. Happy to answer any specific questions! Hope this helps.