Right. This has been known for a long time. It's why rockets aren't much better than they were 40 years ago. Chemical fuels are as good as they can get. Space travel with chemical fuels is just barely feasible.
In the 1960s, it was assumed that nuclear power would be necessary for space flight. Everybody involved knew the rocket equation. The original plan for Apollo included a nuclear upper stage. The engine (NERVA) was built and tested. A Nuclear Assembly Building at Canaveral was planned. But, because the goal was so narrow ("man, moon, decade"), the solution chosen was a disposable rocket the size of a 50-story building to send an RV-sized payload to the moon.
The crash of a nuclear rocket would produce a radioactive mess. Not Chernobyl or Fukishima sized, but at least small-town sized. Launching from an isolated island would help.
Various schemes have been tried or proposed to beat the rocket equation. Launching from a balloon was tried early. Launching from an aircraft is still used by Virgin Aerospace. It helps a little.
A space vehicle that's an air-breather while it's in the atmosphere and transitions to rocket mode once out has been proposed many times, but making something that's both a rocket and an airplane is hard and adds a lot of weight. As an airplane, it has to go hypersonic to get up enough speed that it's worth doing this. Building a hypersonic aircraft is very hard; so far, only a few small demo craft have done it. The National Space Plane (hypersonic single-stage-to-orbit) was proposed in the 1980s. Ben Rich, head of the Lockheed Skunk Works and the designer of the SR-71's propulsion system, declined to let Lockheed bid on it. (His comment: "We used titanium (on the SR-71). You know anything stronger?") Remember, it has to be strong at a few thousand degrees.
The same problems apply to launch track systems. Going hypersonic near the ground is possible; the Holloman AFB test track, 50,000 feet of very straight railroad track, has been used to reach Mach 8.6. The required acceleration is about 14g. Far too much for humans.
The "space elevator" requires not only unreasonable strong materials but the ability to put so much mass in space that you wouldn't need a space elevator if you had that kind of launch capacity.