The rift stems from Biden's decision to remove a portrait of our 7th president from the Oval, replacing Jackson with a painting of Benjamin Franklin. It's a change from the layout during Donald Trump's time in the Oval.

Jackson's great, great, great-granddaughter, Barbara Jackson, tells TMZ ... she approves of Biden's big change, because her relative did some really bad things.

She says Jackson did some good, but she can't get over the fact he was a slave owner and was a fierce opponent of ending slavery.

Jackson's great, great, great-grandson, Andy Jackson, tells TMZ ... he's unhappy and thinks his great-great-grandpa has a rightful place near the resolute desk.

Andy, Barbara's cousin, says President Jackson should be remembered for his accomplishments -- defeating the British in the War of 1812 and increasing suffrage -- even if history doesn't look kindly on his cruel policies on Native Americans.

There's some common ground among Jackson's relatives, who descend from his adopted son Andrew Jackson II ... they both feel the decision was ultimately Biden's to make, and they respect it.

Here's a potential tiebreaker -- Andrew Jackson's Hermitage.

Howard Kittel, the President and CEO of Jackson's former plantation turned museum, tells TMZ ... he was disappointed to see the portrait taken down, but not surprised, given Jackson's ties to slavery.