The actor/rapper took the stage at The Observatory North Park in San Diego Thursday ... when during a break in the show, Smith addressed the sold-out crowd about all the memes trolls have made of him and Diddy.

Watch the clip ... Smith says it's hard to tell what's true or not in today's age -- and the memes online are clearly causing a ton of confusion.

WS makes it simple ... saying "I don't have s*** to do with Puffy, so y'all can stop all the memes."

Smith adds he's never been to a "Freak-Off" ... and, he's got enough of his own drama without being dragged into someone else's.

Will's got jokes too ... saying he doesn't even like baby oil -- a reference to feds allegedly finding hundreds of bottles of the stuff at Diddy's residences.

Obviously, Diddy and Will know each other ... being two of the biggest stars on the planet, there are pics of the two chatting and laughing together. But, Will says that's the extent -- and, he wasn't down to get down at any of Diddy's alleged parties.

As you know ... Diddy is currently locked up in MDC Brooklyn -- a notorious NYC prison -- where he's likely to remain at least until his trial in May after his most recent bail motion was denied.

He's facing racketeering conspiracy, sex trafficking and transportation to engage in prostitution charges.