If a person prefers to be Krishna concious and to not talk prajalpa, should he/she care if others perceive that behavior as being unsocialable? My answer to that would be "no". Let others make such judgements. That is their own problem. But if they are serious sadhakas and they are sincere -- the type of people that you should want to associate with -- the day will come when they will begin to appreciate your prior unwillingness to take part in their prajalpa. If they are not sadhakas and you need to associate with them for reasons other than the direct service of Krishna, then you should simply avoid opening your heart to them. Maintain some small talk with them, but don't forget that your association with them is not for spiritual purposes, but for some other business which needs to be done while you are still in a material body. Affection should be reserved for those who are devotees of Krishna. Affection towards non-devotees is shown by helping them to turn their attention toward Krishna.
Collected Letters to Friends and disciples Vol 5, Pg 119, last paragraph."