crumb-ai
CRUMB (Compact Recurrent-hybrid Underlying Mamba Blocks) — a research project studying efficient architectural hybrids of Mamba-3 selective state-space layers and grouped-query attention (GQA) layers for small-scale Python program synthesis.
What we publish
Pre-trained decoder-only language models that mix Mamba-3 and GQA layers in a single stack, plus the training code, configurations, per-step logs, and per-configuration model cards needed to reproduce the published numbers.
Phase-1 ablation (11 main + 2 pilots)
All configurations are pre-trained from random initialization on the Python subset of bigcode/the-stack-dedup-v2 for 5.37B tokens at sequence length 4096, and evaluated on the same 328M-token held-out carve-out.
| Configuration | Mamba : Attn | Arrangement | True params | Eval PPL |
|---|---|---|---|---|
crumb/abl_2_1_interleaved |
2 : 1 (8M + 4A) | interleaved | 146,075,776 | 3.4182 |
crumb/abl_3_1_interleaved |
3 : 1 (9M + 3A) | interleaved | 148,210,992 | 3.4359 |
crumb/abl_3_1_backloaded |
3 : 1 (9M + 3A) | backloaded | 148,210,992 | 3.4493 |
crumb/abl_2_1_backloaded |
2 : 1 (8M + 4A) | backloaded | 146,075,776 | 3.4683 |
crumb/abl_1_1_backloaded |
1 : 1 (6M + 6A) | backloaded | 141,805,344 | 3.4763 |
crumb/abl_pure_mamba |
12 : 0 | — | 154,616,640 | 3.5237 |
crumb/abl_1_1_interleaved |
1 : 1 (6M + 6A) | interleaved | 141,805,344 | 3.5407 |
crumb/abl_pure_attn |
0 : 12 | — | 128,994,048 | 3.5939 |
crumb/abl_3_1_frontloaded |
3 : 1 (9M + 3A) | frontloaded | 148,210,992 | 3.6798 |
crumb/abl_2_1_frontloaded |
2 : 1 (8M + 4A) | frontloaded | 146,075,776 | 3.7078 |
crumb/abl_1_1_frontloaded |
1 : 1 (6M + 6A) | frontloaded | 141,805,344 | 3.7315 |
crumb/abl_pure_attn_pilot |
0 : 12 | — | 150,062,112 | 3.34 |
crumb/abl_pure_mamba_pilot |
12 : 0 | — | 149,952,224 | 3.39 |
Architecture (fixed across the standard ablation)
| Property | Value |
|---|---|
d_model |
768 |
n_layers |
12 |
n_heads / n_kv_heads |
12 / 4 (GQA) |
d_head |
64 |
d_ff |
3,072 (3,832 / 2,904 for the two pilots) |
vocab_size |
32,768 |
seq_len |
4,096 |
| Position encoding | RoPE (base = 10,000) |
| Normalization | RMSNorm (eps = 1e-5) |
| Embedding tying | on |
| Mixed precision | BF16 autocast |
| Attention | FlashAttention-2 |
Tokenizer
A custom Python byte-level BPE (crumb/crumb-python-bpe) trained on the Python subset of The Stack v2. Vocabulary 32,768. Fertility 2.47 on a Python source corpus — comparable to GPT-4 (2.22) at three times the vocabulary size, and substantially better than GPT-2 (4.60).
Repository
Full training code, configs, evaluation scripts, and logs: github.com/ahmed-farhanur-rashid/crumb-ai
License
MIT. All checkpoints, configs, and the tokenizer are released under the MIT license.
Author
Ahmed Farhanur Rashid GitHub