The phone numbers being hashed doesn't matter because of how small the input space is. A standard phone number is a country code plus 9 digits. If we assume that anybody looking at this information already knows what country the people they're targeting is from, that means there is 1000 000 000 possible phone numbers to check for any hash. Even if the hash is extremely slow, and takes 1 second to compute on a strong CPU, that still only takes 1000 000 000 / (60 * 60 * 24) = 11574 days, or 31 years to compute on a single core. For any large organization (like, say, any government or any large tech company), getting 1000 cores to run the hashes in parallel would be quite simple, reducing the time it takes to have a complete hash list down to 11 days to get a complete database of all possible hashes. Hashing phone numbers is literally just a mild inconvenience.

Edit: Actually looking it up phone number formats vary quite a lot by country, but the point still stands.