7:09 AM PT -- Allen spoke with ESPN's Stephen A. Smith -- fessed up to the tweets and explained he was "young and dumb."

Smith says Allen insisted the "If it ain't white" line was a reference to an episode of "Modern Family" -- and was not meant to be racist.

Allen also claims at least one of the N-word tweets was a reference to a Rick Ross song.

Allen told Smith there's no excuse for the tweets and he understands posting that stuff was wrong -- but says his teammates know he's not a racist person.

Josh Allen -- one of the top QBs in the NFL Draft -- is apologizing for dropping the n-word on Twitter years ago ... saying he was "young and dumb."

The tweets surfaced Wednesday night ... one day before the draft -- and Allen had posted them to his real, authentic account back when he was in high school.

Some of the tweets ...

"i dont think you n**gas want a troubled son!" -- June 14, 2012

“N**gas Trying To Get At Me" -- February 12, 2013

"Bout to show up these N**gas at pong" -- February 26, 2012