Some of Anne Heche's family members are duking it out in court over the late actress' estate ... and it's getting nasty.

Here's the deal ... Anne, who had 2 sons with 2 different men, died without a will and her oldest son, 20-year-old Homer Laffoon, wants to run the estate. He also wants his younger half-brother, 13-year-old Atlas Heche Tupper, and himself listed as Anne's sole heirs.

That would've been clean and simple, but enter James Tupper -- Anne's former "Men In Trees" co-star and Atlas' father says Anne wanted him to be in charge of her estate.

According to new legal docs, obtained by TMZ, James claims Anne emailed him and 2 other people a copy of her will back in 2011, when they were still dating. James says in the email Anne stated she nominated him as the executor of her will so he could carry out her financial wishes.

Meanwhile, Homer's recently doubled down on allegations James is trying to manipulate him ... with Homer claiming he can't get in touch with his half-brother because James is controlling Atlas' phone.

James is firing back in this latest legal filing ... claiming Homer "is not suitable" to run the estate, claiming he's only 20, is unemployed, dropped out of school and was estranged from Anne at the time of her death.

What's more, James claims he's trying to meet up with Homer, but says he's being ghosted ... and James also accuses Homer of locking him and Atlas out of Anne's house as soon as she died.

As we reported ... Anne lost consciousness shortly after last month's car crash and never regained it, dying from her injuries days later in a hospital from thermal injuries and smoke inhalation.