Hierarchical Pruning for Transformer Model Deployment
Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization
State-of-the-art transformer-based models, with gigantic parameters, are difficult to be accommodated on resource constrained embedded devices.Moreover, with the development of technology, more and more embedded devices are available to run a transformer model.To address the deployment challenge of transformer and the problem to select the best device, we propose an algorithm& hardware closed-loop acceleration framework.We optimize the sparse matrix storage format for hpmatrix to further reduce memory usage for the hardware implementation.Our framework can achieve 37x, 1.9x, 1.7x speedup compared to cpu, gpu and fpga, respectively.Experiments on both state-of-the-art transformer and tinybert model show that our framework can find different devices for various constraints, covering from low-end devices to high-end devices.
