We spoke with Dave outside Saddle Ranch in West Hollywood ... and he says he truly believes the 7-time Super Bowl champ is really done -- but his final stop will be Foxborough.

Portnoy tells us, "I think he’s going to sign probably a 1-day contract with the Pats and retire as a Patriot. That's where his heart is."

Dave says all is well between Brady and the Pats after the QB initially snubbed them out of his retirement message ... calling it a "mistake" on his end.

"He apologized to me personally. He said I'm a soldier for life. The next day he created a Patriot video, and he did it two years ago ... he thanked the Patriots fans."

BTW, Portnoy is as loyal as it gets when it comes to the Pats. He was arrested back in 2015 at NFL headquarters after Brady was suspended by the league for Deflategate.

Dave seems to think that TB12's mental blunder wasn't intentional at all ... explaining he was probably forgetful in that moment -- but now he's back in his good graces.

As for Tampa, Dave says Brady's past 2 seasons pretty much meant nothing ... and it doesn't even come close to the relationships he built with the team that drafted him back in 2000.

"He has no love for Tampa Bay," Dave says ... "And he wants to retire where it all began, and where he built his legacy. And it's definitely not Tampa."

Dave says he's confident Tom could've played until he was 50 if he wanted to ... but the only way he wants to see him return is under one condition.