Law enforcement sources tell TMZ ... Matheson told cops someone’s been harassing him online since he took a jab at Melania Trump last month, by saying it was wonderful to have a First Lady who can speak English.

Ironically, it's the kinda thing you'd expect to hear from Melania's husband.

Tim later apologized, but we're told the threats keep escalating -- from the person saying they're going to kick Tim's ass to ... "pay me big bucks or I will kill you."

Our sources say the actor contacted the LAPD after the death threat, filed an extortion report because of the money demands ... and now cops are investigating.

As for Tim's Twitter snafu -- he replied to Melania's farewell message on Twitter to say sorry. He wrote, "Thank you for your work as the First Lady.I apologize to you for a stupid, ignorant,& foul remark that I made."