Bernie Madoff's family is not done paying back the money he scammed from thousands of investors.

The estates of Bernie's late sons -- Mark and Andrew -- have agreed to forfeit $23 million in money they acquired through years of their dad's infamous Ponzi scheme. It's part of a deal struck with Irving Picard, who's responsible for recovering cash for Madoff's victims.

Both sons died years ago -- Mark of a suicide in 2010, and Andrew of cancer in 2014. Neither was ever charged in connection to their father's crimes. After the payback, their estates will still reportedly be worth $1.75 mil for Mark, and $2 mil for Andrew.

Bernie is currently serving a 150-year federal prison sentence in North Carolina for pleading guilty to 11 federal felonies in March 2009.