Gregg Wallace says accusations against him come from 'handful of middle-class women of certain age'