Hidden State

A hidden state is an internal memory vector in a recurrent neural network that accumulates information from previous inputs, enabling the network to process and understand sequential data.

What It Is

Every time you read a sentence, you carry the meaning of earlier words forward. You don’t restart from scratch at each new word — your brain maintains a running summary of what you’ve read so far. A hidden state works the same way inside a recurrent neural network (RNN). It is the mechanism that gives the network memory across time steps, and it is the reason RNNs can handle sequential data where order matters.

Standard feedforward neural networks process each input independently. Hand them a single word from a sentence and they have no idea what came before it. For sequential data — text, audio, stock prices, sensor readings — that’s a problem. The meaning of “bank” changes depending on whether “river” or “account” appeared three words earlier. Hidden states solve this by creating a chain of information that flows from one processing step to the next.

Here is how it works. At every time step, the RNN receives two inputs: the current data point (say, a word embedding) and the previous hidden state. It combines them using a set of learned weight matrices and an activation function — typically tanh — to produce a new hidden state. This updated vector is a compressed representation of everything the network has processed so far. Think of it as a fixed-size notebook where the network rewrites its notes at every step, keeping what seems relevant and letting less useful details fade.

The recurrence formula captures this: h_t = f(W_h * h_{t-1} + W_x * x_t + b), where h_t is the current hidden state, h_{t-1} is the previous one, x_t is the current input, W_h and W_x are weight matrices, and b is a bias term. The network’s output at each step is then computed from h_t.

One critical detail: the hidden state has a fixed size regardless of how long the sequence is. Whether the input is 5 tokens or 500, the hidden state vector stays the same dimensionality. This compression is both a strength (constant memory usage) and a limitation (information from early steps gradually gets overwritten). Architectures like LSTM and GRU add gating mechanisms to control what the hidden state retains and what it discards, directly addressing this bottleneck.

How It’s Used in Practice

The most common place you encounter hidden states is in sequence-to-sequence tasks. Language translation models, for example, read an entire source sentence and encode its meaning into a final hidden state, which a decoder network then unpacks into the target language. Older text prediction systems relied on hidden states to track what you had typed so far and suggest the next word.

Beyond text, hidden states power time series forecasting — predicting tomorrow’s energy demand based on patterns from previous days — and speech recognition, where audio frames arrive one at a time and the network must accumulate acoustic context before deciding which phoneme it heard. In anomaly detection for manufacturing or cybersecurity, the hidden state captures what “normal” looks like over a window of events, flagging deviations as they appear.

Pro Tip: If your RNN performs well on short sequences but falls apart on longer ones, the hidden state is likely losing early information. Switch to an LSTM or GRU architecture before increasing model size — gating mechanisms almost always help more than adding extra parameters.

When to Use / When Not

Common Misconception

Myth: Hidden states store a complete, lossless record of every input the network has processed. Reality: Hidden states are a fixed-size compressed summary. Older information gets progressively diluted as new inputs arrive. Standard RNNs struggle to recall details from more than roughly 10-20 steps back, which is exactly why gated architectures like LSTM were invented — they add explicit mechanisms to protect important information from being overwritten.

One Sentence to Remember

A hidden state is the running summary that lets a recurrent neural network read sequences one step at a time while keeping track of what happened before — but like short-term memory, it fades without gating mechanisms to protect important details.

FAQ

Q: How is a hidden state different from model weights? A: Weights are learned during training and stay fixed during inference. Hidden states change dynamically at every time step, updating as each new input arrives in the sequence.

Q: Why do hidden states cause the vanishing gradient problem? A: During backpropagation through time, gradients multiply across many steps. Small values compound toward zero, making it nearly impossible for the network to learn dependencies from early inputs in long sequences.

Q: Do transformers use hidden states? A: Not in the traditional sense. Transformers use self-attention to access all positions directly rather than passing information through a sequential chain, which is why they handle long-range dependencies more effectively.