Here's the deal ... Justin was at Dogpound Gym in West Hollywood with a bunch of people, but then Taylor and her crew rolled up.

We're told everyone -- including JB was asked to get the heck out of the gym -- because Taylor had an appointment. Justin's security team was approached and asked to leave, but JB's team held firm and said he was going to finish his workout before taking off.

We're told Justin, who did not have an appointment, finished -- although he was mostly goofing around and dancing -- and left ... but he had no idea it was Taylor who had requested the gym be cleared. Once he left, everyone who remained was asked to leave.

One source connected to the gym tells us Taylor did indeed have an appointment with a trainer in the gym ... and the appointment was to train alone. Justin just happened to be in the wrong place at the wrong time.

JB gave fans a preview of his routine that day, spending quite a bit of time dancing in front of mirrors to his new single, "Yummy."