We spoke to the rapper on TMZ Live Wednesday ... and he tells us JB was actually considering taking to the stage for his former mentor Usher's big performance ... but ultimately decided to sit it out 'cause it was just a lot for him to take on.

LJ adds they all wanted Biebs in the show ... but all the rehearsals, responsibility, time, and eyeballs on him apparently wasn't something he wanted to tackle right now ... and as we all saw Sunday, he opted to chill in the audience with his wife Hailey.

Of course, fans were bummed he didn't say yeah to Usher.

Alicia Keys, H.E.R., Ludacris, and will.i.am ended up hopping onstage alongside Lil Jon, so it's not like fans were deprived of an A-list line-up during Usher's 13-minute-long performance. And while Justin wasn't up there ... Lil Jon says he was still supporting him.

Still, there's no denying that the hyped-up JB/Usher collab would've added an extra layer of magic to the spectacle ... especially since we were first to break the story they were in direct talks of executing the moment that would've been remembered for years to come.

In case you didn't know, Usher helped catapult fresh-faced 13-year-old Bieber to massive fame back in the day ... and he's been working overtime ever since, so from his end, it looks like he still needs a break from performing.