Financial incentive does exist, but the problem is that it's a tragedy of the commons. Me upgrading only makes sense if everything else is also upgraded. Until then, it makes sense for me not to spend anything. However, everyone else is making exactly that same calculation.

ISPs have a lot of trouble managing IPv4. How much so depends on when you got your allocations. The first ISPs in the US got tons. The ones that grew out in other countries had to pick over the scraps. Even later US ISPs, particularly mobile carriers, got hit just as hard.

Those later arrivals have to implement Carrier Grade NAT, where all traffic goes through a small set of IPv4 addresses. Sometimes, it's multiple layers of NAT. It takes extra equipment and network design to support all this, which in turn affects speed, reliability, and cost.