Although we may be chanting, why do we still feel distress? Bhaktivinoda Thakura writes in his song atma-nivedana:
“All the troubles encountered in Your service shall be the cause of great happiness, for in Your devotional service joy and sorrow are equally great riches. Both destroy the misery of ignorance.” (Song 8, verse 4)
Having fully taken shelter of the Supreme Lord, even if one has to undergo some great difficulty that might appear a cause of distress, devotees will see that difficulty as a source of happiness, because they have transcended the temporary happiness and distress of this world. They have realized that real happiness in spiritual life is found in pleasing Krishna. However, the ignorant suffer from dvesa: they hate unhappiness and anything that appears to cause it. Abhinivesa, the last form of ignorance, is described by our acharyas as still instinctively clinging to the idea that the body can provide enjoyment and that death means the end of our enjoyment. Therefore all types of ignorance are meant to be destroyed at the root by chanting the holy name.
(LFD, Vol2, Chanting Purely, Pg 109, 2nd paragraph)