Sequence & State-Space Models

Emerging architecture alternatives to transformers for processing long sequences efficiently, including state-space models and mixture-of-experts.

Where to Start

This cluster covers 1 topic. Here's a suggested reading order from fundamentals to advanced.

1

Mixture of Experts

Start here

Mixture of Experts is a neural network architecture that splits computation across multiple specialized sub-networks called experts. A gating mechanism selects only a small subset of experts for each input, so the model can store far more knowledge in its parameters without proportionally increasing the compute needed for every prediction. This sparse activation pattern enables trillion-parameter models that remain practical to run. Also known as: MoE

6 articles

Explore by Perspective

Related Themes

Embeddings & Vector Search 1 topics Inference Optimization 1 topics LLM Training & Pre-Training 1 topics Model Evaluation & Benchmarks 1 topics Neural Network Architectures 1 topics Safety & Red Teaming 1 topics Transformer & Attention Internals 1 topics