The stigma, the questions, all that nasty old blame. It's rotten really and it's not like they can help it.
Which is why a judge - with a keen sense for the mood of the public, 51 per cent of which in this country is female and almost 100 per cent of which isn't a judge - has told a pervert it would be "utterly cruel" to send him to jail.
Andrew Jackson, the pervert in question, attacks young women in parks. He approaches schoolgirls and talks to them about sex. He's flashed a girl of 13, and these are just a few examples of a larger pattern of behaviour. Police have spoken to him twice before about stalking young girls in playgrounds and woods, but Jackson has a low IQ and a form of autism so he's not really taken it on board.
Yesterday Jackson admitted a sexual assault on a 21-year-old woman on a park bench, who was saved only when a friend heard her cries for help and pulled him off her.
He's not a rapist, but he is also not what you might call safe to be out and about.
He is out and about though, because Judge Jeremy Richardson QC seems to think that sex offences happen because men, whatever their IQ, just can't help themselves.
The judge said Jackson was "eccentric". He added that because "you have never had a full sexual relationship, it must be very frustrating" and "you are very much to be pitied". He told him to sign the sex offenders' register and have two years of community supervision.
I'd love to think the judge took pity because of Jackson's mental condition, but I doubt it. His words imply a belief that the crime could be explained by the fact he was a virgin, that there was a pent-up impulse any man is naturally a slave to.
It is not far from talking about frustration and pity to mentioning other words like natural, healthy, and normal. It is not much of a leap at all to say that willies are beyond anyone's control and that they can be blamed for bad behaviour, rather than their operator.
The fact is that most men can help it and manage to get through life without assaulting anyone. Those that don't tend to say things like 'I couldn't stop myself', waving their bits like a Get Out Of Jail Free card which makes judges and juries feel sympathy for men who cross the line even when they know it is there.
The true reason for Jackson's crime is not that he is a male, but because his brain is wired differently. His condition makes him clumsy and obsessive, and that's why he follows women and girls around. A woman with a similar set of mental instructions may well do exactly the same but no judge would say it was because she owned a mimsy.
Most of us would agree with that the last place Jackson needs to be is a prison surrounded by hardnuts and psychopaths, drug addicts and thieves. But he does need to be somewhere he's not going to be molesting women and frightening kiddies.
There used to be hospitals and special units for people with Jackson's problems - some of them awful, others amazing - but he can't be sent there for proper treatment because we've shut most of them down.
Instead he's being cared for 'in the community', which means he lives at home with his parents. I'll bet you any money they're good people who try their best and are at the end of their tether, with their allowances cut and no help on hand when they really need it. They are expected to provide the care of a hospital unit, for free, and forever.
The local police already keep a careful eye on Jackson - as I'm sure the neighbours do - but the judge has now decreed he will spend the next two years being formally 'supervised'. This will involve a weekly or monthly meeting with a social worker or counsellor who is already too busy, and who have no chance of changing things permanently because you can't rewire a brain.
As a result, Jackson will be back in a playground at some point. If we're lucky the police will spot him before he does anything and move him on, but one day they'll be busy doing something else. When that happens it won't be the police's fault, or his parents', or even that of his winky.
It will be because someone who needed more care than he got didn't have it. It will be because of a ridiculous Victorian notion that men are naturally rampant, and sexual offences are a normal consequence of masculine frustration.
It will be because a silly judge said a stupid thing, and didn't have the guts or the wit to see that the crime laid out before him had a cause far bigger and wider than anything that could fit in somebody's trousers.
A judge has the power to shame society into doing better.
This one just lets you do what you like.
Judge Jeremy Richardson QC: not bright.