There's maybe two problems with this:

• public housing is a part of the picture, and so are public libraries. The solution is certainly not to cut library spending just because there are homeless people using it
• Thinking there being homeless people around is an issue that needs solving is itself pretty bigoted. Like, maybe you have a problem with people who haven't showered for a while? or people who use the library for personal activities because there are no better places for them to do them? But 'these people are a problem' itself becomes problematic because you've consolidated those qualities you find objectionable into a class of person, and that makes it really easy to forget/misplace/dismiss the humanity those people deserve.

It's a common attitude, so don't feel like i'm picking you out personally to scold. More people should be aware of how that attitude dehumanizes people experiencing shelter insecurity.