Which part of the road should a cycler take?


When riding a bicycle I regularly heard "friendly" words from both pedestrians and drivers. The first ones shout at me for riding the bicycle on the pavement: You should be on the street, you demented moron! - so they say.
And when I`m on the street I get honked at and shouted at by drivers (it doesn`t matter, how close to the right side of the street I ride my bicycle, I`m still offending them): Move away from the street, you retarded faggot! - so they say.
And I always have the dilemma - which way to choose, as in both cases I`m the target of cursing. And I wonder - why? According to traffic rules I can ride my bike in the street, if I`m close enough to the right side; and I can ride it on the pavement, if I don`t disturb pedestrians. The problem is that for some reason both drivers and pedestrians usually considers cyclists as some kind of untermensch - that is worse than themselves, so they think that pushing around a cyclist and letting their anger out.
And how can I prove them that in all the years that I have been cycling not once have I crashed into a pedestrian and not once have I done damage to a car (although I was hit by a car once but it was not my fault then).