Bhagavad Gita, Chapter 2, Verse 67

For the mind that follows the wandering senses carries away one's understanding, as the wind carries a boat off its course on the waters.

It only takes one unchecked sense to blow the whole course off track.

A boat doesn't need every direction of wind working against it to get pushed off course, one strong gust is enough. The mind works the same way: it doesn't take total indulgence to derail clear thinking, just one sense left unwatched.