Rohan Silva -- a UK native of Sri Lankan heritage and former aide to Prime Minister David Cameron -- says he was invited to a meeting at the Palace to discuss policy with the Queen's son, and asked him if the trade department could be doing a better job.

To that question, Silva claims Prince Andrew said ... "Well, if you'll pardon the expression, that really is the n***** in the woodpile."

Silva says the prince's racial epithet left him reeling as he left Buckingham Palace, but he's ashamed to say he didn't do anything about it. He says, "For a long time after, I kicked myself for not confronting the prince on his choice of words -- and it's something I still regret today."

Silva claims this wasn't the first time Prince Andrew used bigoted language either. At another Buckingham Palace meeting a year prior, Silva claims the prince told him ... "What you have got to remember is that you’ll never get anywhere by playing the white man."

The former aide says he was unfamiliar with the phrase at the time, but when he looked it up he realized it implies that dark-skinned people are less trustworthy and don't play by the rules like white people.

Silva's claims come on the heels of another Prince Andrew scandal ... related to his friendship with Jeffrey Epstein. Andrew's accused of having sex with an underage woman named Virginia Roberts (now Virginia Giuffre), who he met through Epstein at least 3 times from 1999 through 2002.

The Royal family has denied Virginia's allegations, but the prince did himself no favors by making an appearance on "BBC Newsnight" and claiming he couldn't even remember Giuffre ... despite posing in a photo with her in 2001.