ARMAX models are widely used in control engineering for both system description and control design. They can accurately describe a large class of real processes with relatively low complexity, but do not take into account observation errors, which can be particularly important in applications like filtering and fault diagnosis. Due to the intrinsic dependence in ARMAX process output, a single outlier may contaminate multiple data entries and completely spoil the conventional least-squares estimate. In this study, the authors develop a novel Moving Horizon Estimator that is not only robust to outliers but also able to reveal outliers. By utilising the fact that outliers are extreme errors that occur infrequently, the estimation problem is formulated as a least-squares optimisation problem with outliers explicitly modelled and regularised with -norm to induce sparsity. A coordinate descent-based solver is developed to obtain iterative algorithms with guaranteed convergence and closed-form solution available to each coordinate sub-problem per iteration. Due to the explicit modelling of outlier vectors, the impact of an outlier on multiple time instants can be estimated and mitigated. Simulation tests demonstrate that the proposed algorithm can effectively cope with outliers, even for the case when the commonly used Huber's M-estimation approach breaks down.