Exploring Artificial Intelligence

Network Diagram Vanilla RNN
RNN Cell Active Cell Input Hidden State Output
Hidden State Heatmap dim = 8

Controls

5

Recurrent Neural Networks

A Recurrent Neural Network (RNN) processes sequences by maintaining a hidden state that is updated at each time step. This gives the network a form of memory over previous inputs.

LSTM (Long Short-Term Memory) adds gating mechanisms to control what information to forget, store, and output, solving the vanishing gradient problem of vanilla RNNs.

How to Use

  • Select an architecture (Vanilla RNN or LSTM)
  • Choose an input sequence from the dropdown
  • Press Process to animate the full sequence
  • Press Step to advance one time step at a time
  • Watch the hidden state heatmap evolve over time

Vanilla RNN Update

  1. Receive input xt at time step t
  2. Combine with previous hidden state ht-1
  3. Apply weight matrices Wh and Wx
  4. Pass through tanh activation
  5. Output new hidden state ht

LSTM Gates

  • Forget Gate (ft) — decides what to discard from the cell state
  • Input Gate (it) — decides what new information to store
  • Output Gate (ot) — decides what to output from the cell state
  • Cell state Ct provides a highway for gradient flow

Vanilla RNN

ht = tanh(Wh ht-1 + Wx xt + b)

Wh maps hidden-to-hidden, Wx maps input-to-hidden, b is bias.

LSTM Equations

ft = σ(Wf [ht-1, xt] + bf)
it = σ(Wi [ht-1, xt] + bi)
ot = σ(Wo [ht-1, xt] + bo)
Ct = ft ⊙ Ct-1 + it ⊙ tanh(Wc [ht-1, xt] + bc)
ht = ot ⊙ tanh(Ct)

σ is sigmoid, ⊙ is element-wise multiplication.

RNN Metrics

Time Step 0 / 6
Hidden Dim 8
Cell Type Vanilla RNN
Current Input -
Hidden Norm -
Status Ready