According to new legal docs ... the 'Avengers' star and Sonni Pacheco exchanged nude photos while they were married, but he says over the course of their custody battle she submitted uncensored pics of his genitals to their lawyers and the evaluator "for no other purpose than to cause me extreme embarrassment."

Renner claims Pacheco failed to mention to the evaluator she sent nudes to him first and requested that he return the favor. He says she conveniently failed to send her nudes to the evaluator. He also says she threatened to sell the pics of him to the media.

The docs allege Sonni is obsessed with Jeremy's sex life and the sex lives of those in his orbit, including one of his nannies. The actor claims she tracked down racy internet photos of the nanny and sent those to the evaluator and his lawyers.

There's more ... Renner says his ex's obsession with sex is expressed in her sculptures, which he calls sexually explicit and "disturbing." The legal docs include several examples, including one where their daughter, Ava, is staring at Pacheco's creation.

Jeremy says he's deeply concerned Sonni is "exposing Ava to such dark, graphic, sexual material on a daily basis."

We broke the story ... Renner fired back at Pacheco Monday in legal docs, calling her a liar with drug and mental health issues ... after she accused him of talking about killing both of them.

The 2 have had a long, nasty history since she filed for divorce in December 2014 after being married for only 10 months. There have been multiple child support issues, and last month Sonni filed for sole custody.