Could HMRC incorrectly withdraw you from annual accounting?

Your business can only use the annual accounting scheme (AAS) if its annual sales are below certain limits. Is there a risk that HMRC could wrongly instruct you to leave and, if so, what can you do about it?