Witnesses at the scuffle tell TMZ ... the rapper was sitting in his car in Times Square in the wee hours of New Year's Day when a man and his wife walked by. We're told the wife said "hi" to Busta, but the husband didn't like Busta's response -- he allegedly invited her to hang with him in his ride.

We're told the man had words for Busta, who got out of his car and began arguing with him ... which is where the video picks up. Things escalated pretty quickly, and the guys got face-to-face and grabbed each other before Busta's security broke it up.

Everyone eventually walked away, but we're told cops showed up and talked to the man who got into it with Busta. There are no signs any arrests were made during the incident.

We've reached out to Busta's rep, but no word back yet.

Confrontations like this are nothing new for Busta. We broke the story ... he snapped on a dude in Miami in October 2017 and looked ready to brawl before his crew stopped him.

There's also his infamous gym freakout in 2015 ... where he threw a protein shake at an employee.