Abstract: Transformers have demonstrated considerable success across various domains but are constrained by their significant computational and memory requirements. This poses challenges for deployment on resource-constrained devices. Quantization, as an effective model compression method, can significantly reduce the operational time of transformers on edge devices. Notably, transformers display more substantial outliers than CNNs, leading to uneven feature distribution among different channels and tokens. To address this issue, we propose an Adaptive Outlier Correction Quantization (AOCQ) method for transformers, which significantly alleviates the adverse effects of these outliers. AOCQ adjusts the notable discrepancies in channels and tokens across three levels: operator level, framework level, and loss level. We introduce a new operator that equivalently balances the activations across different channels and insert an extra stage to optimize the activation quantization step on the framework level. Additionally, we transfer the imbalanced activations across tokens and channels to the optimization of model weights on the loss level. Based on the theoretical study, our method can reduce the quantization error. The effectiveness of the proposed method is verified on various benchmark models and tasks. Surprisingly, DeiT-B with 8-bit post-training quantization (PTQ) can achieve 81.57% with only 0.3% drop while enjoying 4× faster runtime. Furthermore, the weights of Swin and DeiT on several tasks including classification and object detection can be post-quantized to an ultra-low 4 bits, with a minimal accuracy loss of 2%, while requiring nearly 8× less memory.