The Rise of RNN? Review of “Retentive Network: A Successor to Transformer for Large Language Models”
A review of RetNet’s parallel, recurrent, and chunkwise retention, and its promise of parallel training with constant-time inference.
Writing archive
2023
Posts from this year.
GZIP + KNN > BERT for Text Classification??
A look at compressor-based text classification with GZIP, normalized compression distance, and k-nearest neighbors.
Compositional Generalization
What is compositional generalization problem and why is this interesting?
Big NLP Trends from 2016 to 2022
Recent trends in NLP from Attention to ChatGPT