Srila Bhaktisiddhanta Swami says that (Cc. Adi 7.92, Purport): a devotee gives the non-devotees his association but is not affected by their misbehavior.” He is not affected because he is deriving his spiritual satisfaction in the association of devotees. This higher taste actually protects him, and he therefore goes out and gives association to non-devotees and is unaffected by their association. Srila Bhaktivinoda Thakura also makes this same point that even a householder, in executing household duties, has to work and deal with non-devotees. Nevertheless, he says, “Working and dealing with non-devotees does not mean associating with them. ‘Association’ means that there is affection for the non-devotional qualities. When we start developing affection for the non-devotional qualities, then we are associating with them, and there is danger of becoming contaminated.” We can protect ourselves by affection for the association of devotees. Affection for associating with devotees is centered on hearing and chanting the glories of the Lord, and we derive satisfaction from those activities.
Caitanya Caritamrita Lectures Book, Adi Lila 7.62-92 Donetsk—June 17, 2000, Page 61