Turns out the biggest backlash Kanye faced after saying 400 years of slavery "sound like a choice" ... was at home. He says, "There was a moment where I felt like after TMZ, maybe a week after that, I felt like the energy levels were low."

Ye told The NY Times, he actually called several family members and asked, "Was Kim thinking about leaving me after TMZ?"

Obviously, they've worked things out.

BTW, the Times asked Kanye if he wanted to rephrase the statement about "slavery." He refused, saying ... "I never said slavery itself -- like being shackled in chains -- was a choice. That's why I went from slave to 400 years to mental prison to this and that. If you look at the clip you see the way my mind works."