Back to News

### TL;DR

DeepSeek-V4-Pro is a Mixture-of-Experts (MoE) language model developed by DeepSeek AI, featuring 1.6 trillion total parameters with 49 billion activated parameters. It utilizes a hybrid attention architecture combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), achieving 27% of single-token inference FLOPs compared to its predecessor, DeepSeek-V3.2, at a 1 million-token context length. The model supports a context window of 1 million tokens, enabling efficient processing of extensive text inputs. DeepSeek-V4-Pro is available under the MIT license, allowing for both commercial and non-commercial use. It is accessible through platforms like Hugging Face and NVIDIA NGC, with integration options for various libraries and inference providers. The model has been evaluated on multiple benchmarks, demonstrating strong performance across tasks such as reasoning, coding, and general language understanding. For instance, it achieved an 87.5% score on the MMLU-Pro benchmark and a 90.1% score on the GPQA Diamond benchmark. These evaluations highlight DeepSeek-V4-Pro's capabilities in complex reasoning and problem-solving tasks. The model is optimized for deployment on various hardware platforms, including NVIDIA GPUs and Huawei's Ascend AI processors, reflecting DeepSeek AI's commitment to supporting diverse hardware ecosystems. This versatility ensures that DeepSeek-V4-Pro can be effectively utilized across different computational environments, catering to a wide range of applications in natural language processing and AI-driven tasks.

Key Insights & Metrics

Key Features

• 1.6 trillion total parameters with 49 billion activated parameters
• Hybrid attention architecture combining CSA and HCA
• 27% of single-token inference FLOPs compared to DeepSeek-V3.2 at 1 million-token context
• Supports a context window of 1 million tokens
• Available under the MIT license for commercial and non-commercial use
• Evaluated on multiple benchmarks with strong performance

Discussion