My own life situation is pretty messed up, so take this with a grain of salt. But:
> I have always dreamed of becoming a scientist. I always loved maths, sciences, puzzles, theory but also programming and engineering.
For hope, keep in mind that there are other people like you. And that some of them do achieve the kind of goals you state.
> I find it the most soul-crushing experience to continue working while pretending that the whole thing is getting anywhere.
If things have gotten to this point, you definitely need a change. Take it from someone who did not make the change: Continuing in such a situation leads to a downward spiral. I guess there can eventually be some learning value in the experience, but it is a brutal journey and the return ticket is not guaranteed.
Evaluate what you have, and what is missing from achieving your goals -- or (they say it's the journey, not the goal, that matters) from making progress towards your goals.
It sounds like you have a pretty stable financial and professional position. One that also provides you access to a lot of useful tools. How much time, effort, and "spirit" do you have to give to the organization in return for this? If you strike off on your own at this point, do you have a plan or some idea of how you will get these things: Self-sufficiency and the tools you need?
Not that lack of a detailed plan should necessarily stop you. But do you know your next steps and have some comfort with them?
I should add, too, that although some aspects of your current professional community suck, it can be easy to undervalue one's participation in such a community until it is gone. Sounds like there are people there who you respect. And there is likely professional and social contact you would miss. Being in such an environment can almost inherently help one to remain "plugged in" to one's field, as well as providing exposure to things one might not otherwise encounter or consider.
So, you have yourself, and you have the situation you are in. What is stopping the progress?
Is it the situation? For example, a heavy weight of demands for your assigned work may leave you with insufficient time and/or energy left over to pursue your interests.
Is it you yourself? Do you need more external guidance to follow your desired path? (This isn't necessarily a bad thing, although it can have a bearing on the kind of work you may want to focus on. It may be painful, but it is a question that eventually needs an honest answer.) Are you a person who fundamentally cannot accept situations that you find "wrong"? (This, also, is not necessarily a bad thing. People who have refused to accept the status quo and/or incomplete or faulty reasoning, have eventually forced very valuable re-evaluations. This, too, can be a painful process and require fortitude.)
While you are in your current position, you might focus on -- I kind of hate to use this word, not being the most social person myself -- networking. Try to get to know the people you do respect, and their work, better. Learn what other people are doing both at your institution and elsewhere. I imagine it's true in academia and government organizations as well as in business, that people hire people. If people you like and respect come to like and respect you, this may lead to new opportunities, and those will be with people you know and trust.
Finally, I was recently in touch with an old friend who's part of the a very large research project. And, for all the intellectual brilliance you might imagine is there, some of the individual qualifications, or lack there of, and a lot of the personal behavior, as described to me was a bit stunning. There are some people there -- researchers -- for largely political or even bedroom reasons. There apparently is A TON of self-interest, and competition.
Some of this appears to be inherent in research these days. Resources are limited, and the type A personalities will push hard to get what they can. People become very territorial. Aggression plays a significant role in at least some people's success. I say this looking in from the outside, based on this friend's comments; it may not be an accurate and/or an entire picture, but it's the one I've gained.
My friend is one of those "brilliant" people who's always had to find his own path -- hardly mainstream and without the pretty, mainstream CV/resume some might be looking for. But, he can figure anything out, if it interests him, and is amazingly effective at "getting shit done". People who get to know him learn this and trust him. When someone else wasn't working out, one of those people brought him in to this project.
My point is not to ramble on, but to say that for all the assholes, there are also good people out there, who "get shit done". And some of them get to know each other. It doesn't erase the bullshit, but it compensates. And apparently it lets them achieve their goals, if with some aggravation.
Some of those people who "get shit done" will do so regardless of the environment. If there is an obstruction, they will find a way around it, or a way to obviate it. Or they may simply ignore it, come up with a minimum effort to supply required reporting, and get on with the things that matter (to whatever project they are officially on and/or to them). My description is, I guess, a bit simplified and even romantic. But in essence, it's how some people work.
I'll try to tie this back to your situation. What do you feel are the real stumbling blocks? Is it the environment and what it demands? Or do you find yourself needing more direction?
Regardless of whether you remain or not, you'll need some direction to pursue. Preferably one you are passionate about. You may not remain passionate about it forever, or even past the next year. But once you own it, environment becomes a means to an end. Hopefully not one you simply exploit entirely to your own ends -- sounds like you've already seen some of the effects of such an attitude in what you currently face. But one you turn, as best you can, to your purpose. And one where you contribute, because a better environment helps you meet your purpose.
And if you do stay, perhaps a bit of that attitude might encourage your colleagues and improve the atmosphere a bit.
Then again, I myself am currently in the pits. Take this with a grain of salt. But I'll submit my response nonetheless, in case there's any value either in or between the lines.
P.S. I would avoid actively opposing people or engaging in hostility, even where justified. Senior researchers can burn you with a lasting mark. Rather, if you stay, I'd be tempted to think of it as more of a guerrilla campaign. One where you are out to achieve your goals while leaving any ignorant powers that be as blissfully unaware as possible.