Khloe Kardashian will no longer allow Kanye West to continue to tear Kim Kardashian down, she's speaking up -- telling him to leave Kim and their family alone as he tries to defend himself for mistakes he's made.

Kanye took to Instagram Wednesday, where he seemingly attempted to deflect blame for his choice to wear a "White Lives Matter" shirt. Ye pointed to a narrative he's tried before, claiming Kim held a "secret" birthday party for their daughter, Chicago, almost a full year after she filed for divorce.

At the time, sources close to Kim told us Kanye had planned on throwing his own birthday party for Chicago, and decided last minute he wanted to attend Kim's party ... which he did.

Khloe says, "Ye, I love you. I don’t want to do this on social media but YOU keep bringing it here. You are the father of my nieces and nephews and I’m trying to be respectful but please STOP tearing Kimberly down and using our family when you want to deflect."

She continues, "Again with the birthday narrative. Enough already. We all know the truth and in my opinion, everyone’s tired of it. You know exactly where your children are at all times and YOU wanted separate birthdays. I have seen all of the texts to prove it. And when you changed your mind and wanted to attend, you came."

In a recent interview, Kanye admitted it was Kim raising their kids, and said he needed to work harder to make things calm for her. Khloe, drawing from that finishes, "Like you have pointed out yourself, she is the one taking care of your kids 80% of the time. Please Leave her and the family out of it so that the kids can be raised peacefully."

Kanye's taken a ton of heat since Monday when he wore the controversial shirt to his YZY Fashion Show. It's just the latest in social media attacks and other controversies he's brought over the years ... including when he declared slavery was a choice in the TMZ offices.