If you think it's time for Tom Brady to retire ... PUT DOWN THE CRACK PIPE, so says Brady's old teammate ... who believes anyone saying he's washed up has gotta be on drugs.

TMZ Sports spoke with Jermaine Wiggins -- who won Super Bowl XXXVI with the Pats -- about New England's week 1 loss to the Chiefs ... and he says TB12 naysayers don't know football.

"The people that say he's finished, those generally are the people that have no clue what they're talking about," Wiggins says. "They don't know their ass from their elbow."

Wiggins says the loss to the Chiefs will end up being water under the bridge once the Pats win their next 10 games.