I know your question isn't necessarily US centric, but I'm going to write what I know.

The legal system in the USA is rarely fair or just to the average person. A completely innocent person can be detained and severely punished in a variety of ways from detention, life altering legal fees, coerced into pleading guilty, trial by media / public humiliation, and the police can literally, legally steal your property and money.

Worse, there are so many laws on the books and so many gotchas that essentially everyone in the USA (above a certain age) is breaking / has broken a law. Some things are obvious (driving even 1 MPH over the speed limit is breaking a law), but there are a myriad random things that are technically illegal but the average person doesn't know of, the laws are inconsistent, and they vary from town to town, county to county, state to state, etc.

Privacy helps shield people from some of the negative impacts of these regrettable features of our legal system. For me it really does boil down to the fact that while I myself may not have anything to hide, a right to privacy still protects me from unwarranted persecution / prosecution.

I'm old enough to recall a time when people like me (oh no the gays) were routinely outed and shamed by police even when they'd broken no laws. They'd publish "arrest reports" in the local papers making sure to state that the innocent person who had no charges filed against them was detained leaving the "Brass Bull Bar" (aka the well-known gay bar) or that the suspect was noted to be unclothed in bed with another man when arrested in his home. Basically people who had not actually broken a law, but still punished because at the time these revelations were enough to have people ostracized by family and community, but also it put their lives and health in danger.