We've spoken to several neighbors of the $35 million mansion he just closed on and asked them about Ye's recent Instagram Stories post ... referencing "DROAM," a self-sustaining city Ye once planned to build in the Middle East, but now seems to imagine he'll build on the Westside of L.A.

Kanye doesn't come right out and say that's the goal in his post ... but, his neighbors in the North Beverly Park gated community are clearly telling us it won't be happening in their backyard -- with one person we spoke to calling the star "delusional."

We're told the neighborhood's HOA is complicated, so it's unlikely anyone's voting to let Kanye build Droam in the area ... and, there's not a lot of space left to build, so Ye would have to dole out many millions to take over enough space to build the hypothetical city.

Some neighbors claim Ye is already a chaotic force, pointing to the multimillion-dollar Malibu mansion he gutted and then eventually had to sell at a discount ... and some worry about lunatic fans trying to break in, or other potential celebrity-related disturbances of the peace and quiet of the neighborhood.

The best-case scenario, one neighbor says, is for Kanye to always be on vacation ... instead of trying to uproot the whole neighborhood.

We broke the story ... Kanye purchased a 20,000-square-foot, 11-bedroom home that sits on a 7-acre plot of land in the community.

Kanye's still living an "upper-class homeless" existence in Tokyo -- recently hitting the street with his wife, Bianca Censori -- so, he's not moving in just yet.

It is unclear how literal Kanye is about Droam -- or if he's referring to his new property as his own personal oasis.