3:00 AM PT Carter just told TMZ, he does indeed owe money, but the amount was being negotiated when the lawsuit was filed.

7:30 PM PT -- Troy Carter tells TMZ, he never borrowed money from Scooter. He says he agreed to "re-purchase" Atom Factory -- a management company which Scooter bought from Troy -- and Scooter offered him a discount if he paid up front. Carter does not specifically say if he still owes Scooter money.

Troy says he feels he's the victim of extortion because Scooter threatened to sue him for fraud if he didn't pay him the money, adding, "It's ironic how I can make it out of the streets, only to be extorted by paper gangsters and their attorneys."

He then goes after Scooter, "The only fraud is Scooter Braun professing to be a high integrity individual. The spurious speeches about equal rights don't square when you make excuses for clients that make jokes about 'n******s.' Or, attempting to spin the press when a client diminishes the horrific crimes of slavery. He goes on ... "Had these comments been anti-semitic, he would've made sure those clients never worked again. It's financial gain over personal principals."

And, he says, "There are only a handful of African-American executives left in our business and yet he's okay with attempting to falsely ruin my reputation."

Scooter Braun's company is going after Lady Gaga's former manager for millions it loaned him, but never got back ... according to a new lawsuit.

Ithaca Management Holdings, Scooter's Co., is suing Troy Carter -- in addition to managing Gaga, he used to be a muckety-muck at Spotify. In the docs, Ithaca Management says it loaned Carter in excess of $10 mil in June 2016.

IMH says the deal was if Carter's company got paid from a case it was arbitrating, it would fork over the $10 mil-plus it was loaned. Sources connected to the case tell us Carter was arbitrating a dispute with his ex-client, Gaga.

According to the suit, obtained by TMZ, Carter's firm did settle the case but tried to hide the money, instead of paying off the loan. IMH sent a letter to Carter in August demanding its funds ... to no avail.