Barry Myers, 41, a prosthetic technician from Eastbourne, was refereeing a Hastings and East Sussex Sunday Football League match between Ore United and The Fountain at Harrow Lane on Sunday, when he was allegedly punched to the ground by The Fountain's Timmy-Lee Saunters.
Mr Myers said: "It was all going well until the 60th minute when a player I had already cautioned showed some dissent.
"I was in the process of sending him off when he charged at me, punched me and knocked me to the ground. I was left with a broken nose and the match had to be abandoned there and then."
Saunters, who was playing his first game for the Fountain, left the ground before the police arrived.
Inspector Gary Keeting, of Hastings Police, said: "The incident is subject to an ongoing investigation."
Mr Myers has been refereeing matches for 21 years and said this was the first time anything like this has happened to him.
But Mr Myers was back in action on Wednesday night officiating an FA Youth Cup game between Dover and Hastings at the Pilot Field.
He said: "I had a lot of encouragement from local referees asking me to carry on.
"I will carry on and my first Sunday game will be refereeing the Fountain.
"I am not going to let one individual stop me from doing something I have enjoyed for the last 21 years."
Mr Myers, a father of one, added that this sort of behaviour was not prevalent in the leagues in his experience.
Saunters, who also plays for Rye reserves, has been charged under Football Association rules and is suspended until the hearing. If found guilty, he could receive a life ban.
Mr Myers is also considering taking the matter to the civil courts.
League vice-chairman and honorary fixtures secretary Robin Powell said: "If it was a Premiership referee, there would be a huge outcry."
The incident comes in the same week that former Premier League Referee Keith Hackett launched a national campaign calling for respect for referees. A DXTL Weekend of Action concentrates on teaching children, parents and supporters the importance of respect in football.