Sources with direct knowledge tell TMZ, Kanye's lawyer, Samantha Spector, is making custody job #1. As we reported, Kanye groused over the weekend again that Kim is not letting him see their kids when he wants.

That's certainly not the way Kim sees it ... she has welcomed her ex-husband's involvement in their kids' lives and wants him to spend time with their 4 children, and claimed he picked them up for school Monday.

What's more ... our sources say Kanye's spent 3 days with the kids in the last week -- the first time he's been in L.A. since the Super Bowl. Kanye also had 3 of the kids with him at the Donda 2 event in Miami last month -- we're told Kim kept North in L.A. because she had a test at school she'd been studying for that she didn't want to miss -- so Kim made school the priority.

Here's the issue. Up until now, there has been no formal arrangement for custody. They have tried working it out amongst themselves. When Kanye's in town and wants to see the kids, she's game, unless the kids are in the middle of something else.

But, there's clearly a conflict, and it's getting worse. We're told Spector will first try to establish something more formal between Kim and Kanye, but if that doesn't happen she'll go to court and ask a judge to impose ground rules for custody and visitation.

Here's the rub. Kanye travels frequently outside of L.A., and it's hard to formalize a custody arrangement when schedules keep changing.

As we reported, Kim has been doing everything she can to keep things peaceful, but Kanye's so angry over her relationship with Pete Davidson as well as custody issues, the notion of reaching an out-of-court agreement seems unlikely.