Diplo's going after an ex-fling in a new lawsuit ... claiming she's dangerously unstable because she won't stop harassing him or his family with explicit and racist messages.

The famous DJ filed docs claiming Shelly Auguste is a "relentless and unstable fan who manipulated herself into" his life years ago, but won't take no for an answer after he rebuffed her.

In docs, obtained by TMZ, Diplo claims Auguste's gone to extreme and dangerous lengths to harass him ... like creating fake social media accounts and flooding the DJ and his family with sexually suggestive messages.

Diplo, who says he had a brief consensual relationship with Auguste in 2019 before things went south in 2020, claims she bombarded him with 44 messages that included videos and voice recordings involving Diplo.

What's more ... Diplo claims she also sent sexually explicit photographs and videos of him to his friends and family. He also claims she'd show up unannounced to his house ... and he became so fearful about her erratic behavior he says he was forced to sell his house last month.

As if things couldn't get any worse ... Diplo claims Auguste sent his baby mama racist and disturbing messages earlier this month, saying things like, "ugly dark skin bitch" and "Go kill yourself I know I see right through you you're not gorgeous."

As we first reported ... Diplo got a restraining order against Auguste back in December. At one point, she also got a temporary restraining order against him after making her own allegations of revenge porn, which Diplo denied ... that restraining order was eventually tossed.

He's now suing for stalking, trespassing and distribution of private materials ... or revenge porn. His attorney, Bryan Freedman, tells TMZ ... "As this Complaint makes clear, my client and his family have been harassed, stalked and abused by Ms. Auguste for far too long -- and it’s time for the courts to step in and put an end to her appalling and dangerous behavior."