The actor swung by SiriusXM's "The Spotlight with Jessica Shaw," where he reflected on his guest role stint on Ellen's self-titled sitcom from the '90s. As he put it ... Ellen and her team were eager to have Patrick on for more episodes, but he turned down the gig ... as he was ready to "move on."

To Patrick's credit, he merely was done with guest starring roles ... so, he told Ellen and co. he had a conflict.

While that should've been the end of the saga, Patrick claimed Ellen did not handle the rejection well -- and confronted him when he was out to dinner with a friend in Beverly Hills.

He recalled ... "Ellen walks up to the table and she goes, 'Too big to do my show now, huh?' and I go, ‘No, Ellen.’ I go, ‘I just had a conflict,’ which she probably sensed was just bulls*** and she walked away."

According to Patrick, this drama was why he was never invited on Ellen's wildly successful talk show -- which ran for 19 seasons.

He added ... "She was spurned."

This isn't the first time that Ellen has been called out for being less than friendly. Her eponymous chat show was put to task a few years back after it was dubbed a toxic workplace by current and former staffers.

Don't forget the time Dakota Johnson put ED in her place for lying about not being invited to the actress' birthday party.