Early-exit Mechanism for Sequence Labeling
Accelerating BERT Inference for Sequence Labeling via Early-Exit
In this paper, we first propose a simple extension of sentence-level early-exit for sequence labeling tasks.To further reduce the computational cost, we also propose a token-level early-exit mechanism that allows partial tokens to exit early at different layers.Considering the local dependency inherent in sequence labeling, we employ a window-based criterion to decide for a tokenwhether or not to exit.The token-level early-exit brings the gap between training and inference, so we introduce an extra self-sampling fine-tuning stage to alleviate it.Extensive experiments on three popular sequence labeling tasks show that our approach can save up to 66%-75% inference cost with minimal performance degradation.Compared with competitive compressed models such as distilbert, our approach can achieve better performance under the same speed-up ratios of 2x, 3x, and 4x.
