A Gentle Introduction to Fourier Transformers for NLP

Miguel González-Fierro
May 23, 2021
 
 

The attention mechanism is responsible for much of the recent success in NLP tasks such as text classification, named-entity recognition, Q&A, translation to name a few. However, computing attention is expensive. In this post, we summarize a new approach that replaces self-attention with a Fast Fourier Transform, which achieves 92% of the accuracy of BERT while being 7 times faster.

 
 

nlp; transformer; attention

 
 
 
 
 
blog comments powered by Disqus