The former New England Patriots star made the revelation during a chat with Fred Taylor, Ryan Clark and Channing Crowder on "The Pivot" ... telling the guys he loved when he was the butt of the barbs on Netflix's 'Greatest Roast of All Time' -- but not so much when his family was brought up.

"I didn't like the way they affected my kids," he said of some of the quips.

Brady didn't elaborate on which ones specifically impacted the youngins -- although it seems clear ... it was the ones about his divorce with Gisele Bündchen.

You'll recall, everyone from Kevin Hart to Jeff Ross to Nikki Glaser to even Drew Bledsoe threw shade Brady's way over the split ... which, at the time, brought laughs from former signal-caller, who now says he'll never do anything like that again.

"It's the hardest part about -- like the bittersweet aspect of when you do something that you think is one way and then all the sudden you realize I wouldn't do that again because of the way that it affected actually the people that I care about the most in the world," he said.

Brady, of course, isn't the only one regretting parts of the special ... his ex, Gisele, was reportedly pissed over the remarks too.

Brady, however, did encourage people to still get on mics and perform comedy routines even if it's not necessarily for him anymore ... adding, "If we're not laughing about things, we're crying."