Kanye sounded pissed off about the amount of face time he's getting with their 4 kids when he sat down with Hollywood Unlocked's Jason Lee. Ye said, "Don't play with me, don't play with my children. Ain't no security gonna get in between me and my children and you ain't gonna gaslight me."

While he never mentions his estranged wife by name, the reference to security standing between him and the kids would seem to indicate Kim's involvement. He added, "Nothing with my career, with this rap, with this media, with none of that, that's gonna keep me from my children. And that's what I want everybody to know."

Ye's rant about the kids is puzzling to Kim, because sources connected to her tell TMZ ... Kim's never denied him time with the North, Saint, Chicago or Psalm. We're told she goes "above and beyond" to make sure he gets as much time with them as his schedule will allow.

Kanye was last photographed with all the kids last weekend.

Our sources say the one thing Kim has asked Kanye to do is call ahead of time when he wants to come to her house to see the children. We're told she simply wants some structure in order to protect her right to privacy in her home.

Ye also said he was annoyed by media coverage of his recent purchase of a home next to Kim's ... saying the only reason he did that was to remain close to his kids.

TMZ broke the story ... before he allegedly punched an autograph seeker early Thursday morning, Kanye told fans outside his hotel he was going through a lot related to his divorce and seeing his kids.