Some of Ye's Calabasas neighbors tell TMZ ... there's been a crazy amount of construction work going on at the site. They say construction crews were working for more than 12 hours Saturday -- the city allows construction from 8 AM - 5 PM -- and Sunday as well. The city does not allow Sunday construction.

We're told at least one neighbor called the cops Sunday complaining of noise, and L.A. County Sheriff's deputies responded at 7:30 PM and found a construction crew hard at work on one of the domes.

We're told the deputies told the crew to stop ... and they did.

Kanye's neighbors have been complaining for a while about the prototypes, which he constructed as models for his vision of an egalitarian community.

Turns out Kanye's crew didn't get permits and he was cited for violating the building code. He was given the option to either get the proper permits or tear the 4 structures down by Sept. 15. He's chosen the latter ... thus the continued noise.

It's unclear if Kanye's going back to the drawing board or jettisoning the project.