Jesse Williams' hopes for an amicable divorce just went down the toilet ... he's now in a nasty custody fight with his estranged wife.

The "Grey's Anatomy" star filed docs saying he went above and beyond to informally handle custody issues with Aryn Drake-Lee. Jesse filed for divorce back in April and we were told things were civil. But Jesse says it's now turned into a legal war.

In docs obtained by TMZ, Jesse's asking to get more time with 3-year-old Sadie and 2-year-old Maceo, claiming Aryn only lets him see them shy of 3 hours a day and she refuses to allow sleepovers.

Jesse says he rented a home less than 3 miles from Aryn's home so he could be close to the kids. To prove he's a hands-on dad, Jesse listed their nicknames and their favorite food -- mac & cheese for "Sadie Munchkin" and "anything we put in front of 'Mace.'"

Still ... Jesse says Aryn's made things impossible so he's asking the judge to step in and issue a formal custody agreement.