• Having to buy out private land owners who will take the government to court which can take years to decades to figure out
• Having to do environmental assessment of all the land that will be impacted by the high speed rail then having to re plan the route to avoid sensitive areas and wildlife which would take years if not decades to complete
• Having to raise taxes or shift infrastructure funding to build the rail line
• Contracting out the work to private companies that will inevitably raise costs dramatically because it's a government project
• Having to negotiate with state and local governments that will want (or not want) the rail line to go through their town, city and state.
• If the route is going through a mountainous region having to build massive tunnels (not unheard of just more money and longer build time)
• Contracting out companies to build the rail cars specifically and having them work closely and accurately with the companies building the rail lines.
• If the rail line is going through Texas and is going to use electricity then the government will have to negotiate with the Texas electrical grid, probably demand that they do a better job of keeping it online.
• Promoting the rail line enough so that it gets used more
• Dealing with oil, gas, automotive, and airline lobbies along with all the secondary and tertiary companies and industries that rely on those industries that will lobby to keep it from going through.

This is all just stuff that came off the top of my head. I don't know if it is all valid or not.

Don't get me wrong I would fucking love to have a extensive, reliable high speed rail in the US I just don't think it will happen without huge push from the voters and I'm skeptical that will happen.