The 39-year-old made the announcement Tuesday ... saying he's suffered a setback on his right knee -- the same knee he had procedures on back in 2020.

"During the grass court season, I unfortunately experienced a setback with my knee, and have accepted that I must withdraw from the Tokyo Olympic Games," Federer said.

"I am greatly disappointed, as it has been an honor and highlight of my career each time I have represented Switzerland."

Federer -- who just lost in the quarterfinals at Wimbledon earlier this month -- said he has already begun rehabbing and is clearly hoping to still participate in the U.S. Open in August.

"I have already begun rehabilitation in the hopes of returning to the tour later this summer," Federer said. "I wish the entire Swiss team the best of luck and I will be rooting hard from afar."

Federer has had big success in the Olympic in his career ... he won a gold medal in the doubles event at the 2008 Beijing Games and then a silver medal in the singles event in the 2012 London Games.

Federer did not compete in the 2016 Rio Olympics due to a knee injury.

The list of tennis stars who will not be in Tokyo this summer, meanwhile, is now growing rapidly ... in addition to Federer, Serena Williams, Rafael Nadal and Nick Kyrgios have all pulled out of the event.