People talk. People talk behind people's backs. Do we all not know it already?
I'm really stating the obvious here - rarely do people talk positively of others behind their backs. Of the negative things said, though some may be true, much, I believe, are likely to be false, or immensely exaggerated to appear worse. The reason is simple - it is more enjoyable putting people down. The reason it is more enjoyable putting others down is also simple - it raises one's self-worth, albeit unjustifiably. So, really, people talk.
Sometimes, I come to know of the stuff said behind my back. There are good things, there are bad ones. There are false accusations, there are baseless assumptions. Mostly, they feel like arguments made against one, to which one simply does not have the means nor the opportunity, ever, to offer a rebuttal, or a defense, or anything at all.
This is life. And people will keep talking. Regardless.
So, let them. Let them have this one small recreation to liven up their otherwise mundane life.