Nuclear rockets could have easily made space relatively cheap. The tech was actively tested by NASA, and it worked pretty well. Nixon canceled that program and saddled NASA with a mandate for a Shuttle without the proper funding.

The USSR's manned program, OTOH, was built mostly to hit a number of firsts (first dog in space, first man in space, first woman in space, first space walk, etc.), but do it as quickly as possible. This resulted in a series of "get it done right the fuck now" decisions. NASA did it the slow way, with each technical advancement building on the last, which is better in the long run (if you fund it, mind you). Russia did enough to build Soyuz and then ran that for decades.

The tech did not hit physical limits. The two major approaches to space flight hit different bureaucratic limits first.