The retired Chicago Bears star went public with his troubles on Instagram earlier this week ... saying a "chicken serial killer" has been ripping the heads off his birds at his home in Tennessee.

So ... is it a coyote? A raccoon? A monster?!?

Cutler initially thought his own cat, Thelma, had blood on her paws ... but he later determined the pet was not responsible.

After eliminating Thelma as a suspect, Cutler came up with a plan to catch the REAL murderer -- an all-night stakeout!!!

So #6 grabbed his night vision goggles, camo and a nice stogie and assumed his position in his kids' treehouse to look over his rooster, Old Blue after sundown ... updating his 500k followers throughout the night.

"Obviously things have escalated," Jay said on IG. "We're gonna sit here and make sure Old Blue has a good night tonight and whatever happens, happens."

But, the whole stakeout had an anticlimactic ending Friday morning -- Cutler says he was unable to find the killer ... but he's vowed to continue the hunt.

"No culprit last night, which is okay. Stayed up way too late. Super tired this morning. But, Old Blue's gonna be on the case today."