Undefeated UFC star Justin Gaethje is saying sayonara to MMA after another dozen fights, tops ... telling TMZ Sports the sport's "too brutal" to gun for 50-0 like Floyd Mayweather.

"I'll get to 30-0 and I'm outta here," Justin told us ... before saying there's NO WAY a UFC fighter could hit 50 like Floyd.

Gaethje would know -- dude's got the second longest undefeated streak in the biz at 18-0 ... and is looking to make it 19 when he fights rival "TUF 26" coach Eddie Alvarez.

Justin says he'll wreck Eddie to stamp his ticket to a Conor McGregor fight  ... and told us Conor's legitimately "terrified" of that matchup.

With Nate Diaz demanding CRAZY cash ... Gaethje might just get it.