OlmPool

OlmPool is a controlled suite of 26 fully open 7B language models from Ai2, each released with 38 pretraining checkpoints. It reveals how four architectural choices — QK normalization, grouped-query attention, sliding window attention, and pretraining context length — compound to degrade long-context extension performance by up to 47%.

OlmPool is a controlled suite of 26 fully open 7B language models from Ai2, each released with 38 pretraining checkpoints. It reveals how four architectural choices — QK normalization, grouped-query attention, sliding window attention, and pretraining context length — compound to degrade long-context extension performance by up to 47%.

• 0126 fully open 7B models with 38 checkpoints each for long-context architecture research
• 02Identifies 4 architectural factors that compound to hurt long-context by up to 47%
• 03Full weights, code, and training data open-sourced under Apache 2.0