Long-Context Language Models¶
Language models extended with the ability to process much longer input sequences than traditional transformers. Recent advances have extended context windows from ~2K tokens to 100K+ tokens through techniques like sparse attention, rotary position embeddings, and efficient attention mechanisms. However, simply increasing context window length does not guarantee effective use of that context.