As a student who went through gifted programs in public elementary and in private middle and high school, but otherwise not plugged into the literature on it, what does the evidence look like that gifted/magnet programs actually provide significant tangible benefit to the students? One way to break it down would be to look at the relative number of students in these categories:
1. Students identified as gifted who do well because of their gifted program/who would have done well if they were in a gifted program.
2. Students identified as gifted who would do well regardless of whether they had a gifted program, even if they might do marginally better in a gifted program.
3. Students not identified as gifted who would benefit from additional resources/gifted peers in their non-gifted program
4. Students not identified as gifted who would not particularly benefit from gifted peers/already have adequately resourced classrooms.
I would have a hard time believing that the number of students in any of those categories is negligibly small. So then the question becomes how do we, as a society, best balance the tradeoffs between them. For any given student, we don't generally have the ability to try it both ways and see what would have been better, and of course any given parent will tend to prefer to take any tiny marginal improvement in opportunity for their children, regardless of whether it's net beneficial for society. But some of the role of society is to not allow people to take actions with mismatched externalities.
If there are more #3 students than #1 students (and/or the benefits to society are larger), and if we cannot distinguish between #2 students and #1 students, then yeah, let's get rid of gifted programs.
For myself, I think I was probably a #2—I had affluent, educated parents, benefited from significant extra-curricular enrichment, and could probably have had more enrichment with several tens of thousands of extra dollars per year of saved tuition. At what point could my parents have known that I was a #2?