← Exploring Artificial Intelligence
CSE 10124 · Building ChatGPT · Solutions

irishQwen

The Qwen3 (0.6B) diagram from class, wired to the repo. Click any block to zoom into the code that builds it.

← → walk the forward passEsc zoom out
irishQwen.py Qwen3 (0.6B) qwen_transformer.py Linear output layerlinear_layer.py Final RMSNormrms_norm.py + Feed forwardswiglu.py RMSNorm 2rms_norm.py + Masked grouped-queryattentiongrouped_query_attention.py RMSNorm 1rms_norm.py ×28 Token embedding layerembedding.py Tokenized textqwen_tokenizer.py Sample input text QK-Normrms_norm.py RoPErotary_embedding.py Supported context length of 32k tokens Vocabulary size of 151k SwiGLU in · X (B, T, 1,024) out · (B, T, 1,024) Linear (down)linear_layer.py ⊙ SiLUsilu.py Linear (gate)linear_layer.py Linear (up)linear_layer.py Hidden layer dimension of 3,072 16 query heads, 8 KV groups Embedding dimension of 1,024