GLAAD has spoken out on all the chatter this week about how and when Damon ever used it, and whether he's just recently retired it. Damon is now clarifying, saying he did not retire it because he's never hurled it at anyone -- but to the folks at GLAAD the debate is kind of irrelevant, because the end game is the same.

A rep for the non-profit tells TMZ ... "The conversations that have arisen after Matt Damon’s original interview and subsequent remarks today are an important reminder that this word, or any word that aims to disparage and disrespect LGBTQ people, has no place in mainstream media, social media, classrooms, workplaces, and beyond."

They add, "There needs to be accountability at a time when anti-LGBTQ slurs remain rampant today and can fuel discrimination and stereotypes, especially when used by those outside of the community to defame or describe LGBTQ people."

That last part is interesting ... they seem to be suggesting that Matt even uttering the word in a joke is harmful -- even if it wasn't hurled as an epithet at someone. GLAAD's basic point is ... STOP saying it.

Like we mentioned, Matt addressed what most people perceived he was saying in his interview with the Sunday Times ... that he'd nixed the phrase and any variation of it from his lexicon only after one of his daughters called him out. Frankly, that is exactly how it came off from the quote attributed to him.