Joe's divorce attorney Autumn Blackledge tells TMZ ... Joe believed he'd fallen in love with fellow inmate John Graham, but ever since John got released from their federal prison in Texas ... he's moved on and ended things with Joe.

We're told Joe will continue to have a close relationship with John's son and some members of his family, and despite their split ... Joe has only nice things to say about his ex.

Sounds like Joe's broken heart is healing quickly, though -- we're told a "very special man" came back into his life recently, someone picking up where they left off before he "put his head up his ass." No official word on who it is yet, but a name comes to mind.

TMZ broke the story ... Joe filed for divorce from Dillon Passage in March, with his relationship with John being the reason why. Joe and Dillon were estranged for more than a year before Exotic filed divorce docs from jail.

We're told JE's divorce from Dillon is still making slow but steady progress, so there's a chance this mystery man might be someone totally new ... to the public, anyway.