The 83-year-old -- who's been trying to help keep the summer Games on schedule despite the pandemic -- was in a meeting earlier this week when he reportedly said he believes women talk too much.

Mori, according to the Japanese national newspaper, The Asahi Shimbun, said during the virtual meeting, "Women are very competitive. When one of them raises her hand, they probably think they have to say something, too. And then everyone says something."

He reportedly continued, "If we are going to have more women directors, someone has remarked, then meetings go on for a long time unless we restrict the comments. I'm not saying who that is."

The sexist comments were met with immediate backlash ... and the criticism grew so loud, Mori was later forced to apologize.

"The statement made at the Japan Olympic Committee was an inappropriate expression, contrary to the spirit of the Olympics and Paralympics," Mori said on Thursday.

"I am deeply remorseful. I would like to withdraw the statement. I would like to apologize for any unpleasant feelings."

Still, even with the mea culpa, there's been growing pressure on Mori to step down from his high-ranking role on the Olympics committee.

Mori, though, explained he will do no such thing ... saying, "I have been working hard and helped devotedly for seven years. I will not be stepping down."

The controversy is just the latest dark cloud surrounding the Tokyo Olympics ... the planned event has been marred by issues ever since the pandemic forced its postponement last year.

In fact, some believe the Games -- which are currently scheduled to kick off on July 23 -- will have to be canceled altogether due to COVID-19.