Wellingborough Town’s Martin Potton is alleged to have made the comments in a game refereed by Mary Harmer.
They were losing 7-0 to Rushden and Diamonds when he was overheard saying Harmer was not fit enough to referee a women’s match, let alone a men’s game.
It is alleged he made further comments about other women in sporting roles.
He told BBC Sport: “I deny all the charges but was found guilty in my absence as I was denied the opportunity of a personal hearing.”
Lindsay England, founder of equality group Just A Ball Game, told the Victoria Derbyshire programme that she was at the United Counties League Premier Division match last April and overheard the comments.
She reported him to the Football Association, and others complained as well.
She did not know he was chairman of the non-league club, adding: “Nobody challenged him and nobody laughed and joined in with what he was saying. I think he actually knew what he was saying and he believed that he still had that right to do that.”
Potton was banned for the maximum five games by an FA disciplinary commission and fined £75 after being found guilty of using derogatory and insulting language towards an official. He was also ordered to attend a mandatory education lesson.
Sue Ravenlaw, head of the FA’s equality and safeguarding team, told BBC Sport: “Any time that discriminatory abuse occurs in football, it’s just frankly unacceptable.
“We’ve been encouraging an environment of collective responsibility from club level, with stewards, safety officers all the way through to ourselves. Whenever discriminatory abuse arises, of any form, people should be confident to report it.”