You ask how does one sever a friendship with someone when you know it's not good association. My best recommendation is that you should make every effort to be good association for that “friend”. If they're really your friend then they'll accept that good association and then they'll become good themselves. If they don't like that good association, then they'll become exposed, and a natural disinclination to continue associating with you will automatically follow.
Collected Letters: Volume 2