The Dipset rapper was taking pics with some fans, mostly female, early Sunday morning after he hosted at Club Zone in Springfield, MA. A crowd of dudes gathered and got into a standoff with Jim's security.

After a few tense moments of yelling back and forth ... one guy threw a haymaker, and then all hell broke loose! Police showed up, but everyone quickly dispersed. We're told no one was arrested, and no ambulances were dispatched -- but watch the video. Definitely looks like a few people could've used assistance.

Security did its job though -- Jim came through without a scratch. A source close to Jim says he was not involved in the fight, although someone in his crew had to hold him back at one point.

The source described the chaos as ... "Just an old-school brawl outside of a club."