Extreme Learning Machine (ELM) has recently increased popularity and has been successfully applied to a wide range of applications. Variants using regularization are now a common practice in the state of the art in ELM field. The most commonly used regularization is the \(\ell_2\) norm which improves generalization but result in a dense network. Regularization based on the elastic net has also been proposed but mainly applied to regression and binary classification problems. In this paper, we propose a generalization of regularized ELM (R-ELM) for multiclass classification problems, termed GR-ELM. We achieve such generalization using the \(\ell_2,_1\) and Frobenius norm. Traditional R-ELM is a particular case of our method when binary classification tasks are considered. We also propose an alternative algorithm for GR-ELM when training data is distributed, namely GR-ELM. We use alternating direction method of multipliers (ADMM) for solving the resulting optimization problems. Message passing interface (MPI) in a single program, multiple data (SPMD) programming style is chosen for implementing DGR-ELM. Extensive experiments are conducted to evaluate the proposed method. Our experiments show that GR-ELM and DGR-ELM have similar training and testing accuracy when compared to R-ELM, although usually faster testing time is obtained with our method due to the compactness of the resulting network.