Bhagavad Gita, Chapter 6, Verse 38

Does he not, fallen from both, perish, without support, like a rent cloud, O mighty-armed, deluded in the path of Brahman?

The real fear underneath the question is being left with nothing, not fully spiritual, not fully worldly either.

Arjuna names the worst-case outcome directly, someone caught in between, having given up worldly life without ever fully reaching the spiritual goal. It's a genuine, understandable fear, not an idle hypothetical.