We caught up with the former child star outside LAX, where he told us what it was like working with the film and stage legend ... and he had nothing but nice things to share.

As Patrick put it ... everyone knew who James was while filming the 1993 coming-of-age movie ... but, they unfortunately didn't get a ton of interaction with the A-lister.

While James' character, Mr. Mertle, was the film's antagonist-turned-hero, Patrick said the late star only filmed for a few days ... leaving not a lot of time for the cast to form special memories with the actor.

He added ... "We didn't get to hang out with him too much. Plus, it was about nine 13-year-old kids, I'm sure we were probably overwhelming."

Yet, Patrick did recall one funny exchange with James ... sharing how the actor quoted his famous "Star Wars" line to his costar, Marty York, during one moment between scenes.

Though the child stars of "The Sandlot" were never super close with James, they as a group did remain close ... with Patrick telling us the cast gets together pretty regularly.

As for whether there's an official reunion in the works ... watch the video, Patrick gives his honest opinion of what's to come.

Patrick's insight comes 3 weeks after James passed away at his home in upstate New York while surrounded by his family. He was 93.