Sequence Modeling

Sequence modeling refers to the task of learning statistical patterns in sequential data (text, audio, time series, etc.). Neural approaches to sequence modeling have evolved from recurrent architectures (RNNs, LSTMs) to attention-based models (Transformers).

Foundational architectures

• Attention Is All You Need — introduces the Transformer, replacing recurrence with pure self-attention for sequence transduction.

Related topics

• Transformers — attention-based sequence models
• Attention mechanisms in NLP — the mechanism underlying modern sequence models
• Neural NLP — application of sequence modeling to language tasks