Nothing is wrong with having a desire to return to the spiritual world. If we make attaining Krsna’s lotus feet our goal, and then do everything motivated by the desire to achieve it, then even if there are still material desires within the heart, they will be gradually spiritualized.
Lectures from a disciple Vol 1, pg 323, Bhaktyaloka, Moscow, 1.12.97