There's a certain threshold zone in the middle where people's own self-esteem has a high impact on the esteem that others hold them in, a range in which it becomes a self-fulfilling prophecy for the general public. (Obviously, I Am Not A Professional Psychologist, so this is my natterings; if you think you fall into the bad zone I postulate, it might be something to talk to a Real Counselor Person about. Don't forget to tell them that I live on the internet and am therefore on crack!)
Obviously, when you have someone whose self-esteem is pathologically high (think "You're So Vain" plus "Everybody Loves Me, Baby", and then some) and they think that they are all that, in strict defiance of reality, people are not going to share their inflated opinion of themselves. Similarly, when someone feels the veriest wretch, but they're actually really really nice, people aren't going to think horribly of them.
But in the middle, when there's a range of doubt about what kind of person someone is, the natural human instinct is to trust the other person's self-assessment.
In some range, a poor self-esteem can become a self-fulfilling prophecy that links to a cycle of utter social fail. So when someone gives off every sign that they feel they are not up to standards (within the range of the reasonable), unless they manage to demonstrate not only performance that's up to standards, but in fact performance over the standards, someone who's not paying very close attention will be more likely to consider their performance substandard. It's not going to work on people who really have gotten to know you, and it's not going to work so much on people who aren't intellectually lazy, but there's a little psychological mechanism that goes, "Well, people don't think that poorly of themselves for no reason, so there must be something besides what I'm seeing that makes them think that way." And then they'll start looking for points of fail. And they'll find them.
Similarly, if someone is only actually average by any objective measure, but they act with the confidence and assurance of someone above-average, others are more likely to think of them as above-average. (Note that this one falls down somewhat when things like job performance that can be measured by objective standards comes into play, but people with good stats in social can pull off some crappy performance and still do well. This is why.) Again, there's that little thing that goes, "Well, X and Y aren't so hot, but people don't have this kind of genuine confidence in themselves for no reason. What do they have going for them?" And they look for points of win, and, not surprisingly, find them.
