The St. Louis Cardinals legend is reportedly splitting from 'Real Housewives of Orange County' star Meghan King Edmonds after 5 years of marriage ... and there are rumblings it's all over Jim's alleged infidelity.

And, when a fan referenced the alleged nanny-banging on social media over the weekend, Meghan commented with a thumbs-up emoji, adding fuel to the fire.

Now, 49-year-old Jim is breaking his silence ... coming out swinging hard against the allegations and seemingly taking veiled shots at Meghan as well.

"It’s one thing to be accused of something. It’s another thing to be so negligent and so careless, and ruin the lives of innocent young people," Jim said on Instagram.

"I did not sleep with our nanny."

"She’s not just a nanny, she is someone that we brought into our house and promised to protect and look after while she was in our home. We have been treating and raising this young girl like she is our own child! For someone to accuse me of sleeping with her is not only wrong, it’s disgusting and irresponsible."

Edmonds continued ... "Just because I am an athlete and I was portrayed as a bad person on bravo TV. Doesn’t mean I dont have a clue what life is all about."

"We made a promise to that young girls mom, that We would take care of her and make sure that she was safe while she was in our hands. I will not sit here and take this bulls**t without fighting back."

Here's where it seems Jim goes after Meghan -- who reportedly filed for divorce last week -- though he never mentions her by name.

"Just because you’re extremely whitty and intelligent, and know how to play the Instagram game. Doesn’t give you the right to use that platform to wrongfully accuse."

"Not only is it irresponsible. It is dangerous and puts my family in harms way."