Perplexity Releases Lily for Local Qwen 35B Inference on Apple Silicon

AIDeveloper44 Team

Overview of the Lily Inference Server

Perplexity has shared new developments in its open-source inference technology through the release of Lily, a specialized inference server hosted in the pplx-garden repository. Lily is a minimal Metal-based server built in Rust, designed to facilitate local execution of a specific large language model (LLM) checkpoint: the Qwen3.6-35B-A3B. The tool is tailored for Apple Silicon and focuses on efficient, on-device inference by utilizing the MLX framework's quantization standards.

Unlike general-purpose inference engines that support a wide variety of models and quantization formats, Lily is intentionally restricted. It is designed to work exclusively with MLX affine 4-bit weights with a group size of 64. The server validates the architecture and quantization layout at the time of loading, rejecting unsupported formats such as GGUF, AWQ, GPTQ, or standard floating-point weights (BF16, fp8). This strict adherence to a specific architecture allows for deep optimization within the Metal environment.

Hardware and Software Prerequisites

According to the technical documentation provided in the Lily repository, the server has specific hardware and software requirements that point toward future-facing Apple technologies. The project specifies the need for Apple GPU family 10 or later, which corresponds to M5 series chips and newer. Additionally, the software requires macOS 26 or later to support necessary Metal tensor operations.

The build process is managed via Rust 1.92, as pinned by the project's toolchain configuration. For model compatibility, users must provide a local Qwen3.6-35B-A3B MLX checkpoint. Perplexity specifically recommends the mlx-community revision `38740b847e4cb78f352aba30aa41c76e08e6eb46` for benchmarking and testing, ensuring that the quantization remains immutable for performance tracking.

Technical Architecture and Kernel Design

The core of Lily lies in its Metal kernels, which are responsible for the computational heavy lifting on the GPU. These kernels are not pre-built; they compile from source at runtime, eliminating the need for an offline shader build step. The source layout of the project indicates a highly modular approach to these components:

• src/kernels/: Contains the Rust dispatch logic and the raw Metal shader source files.
• src/model.rs: Defines the prefill and single-token greedy model graph.
• src/weights.rs: Handles the loading of MLX affine Q4 weights into memory.
• src/config.rs: Performs strict validation of the 35B-A3B checkpoint parameters.

By focusing on greedy decoding—where the model always selects the most likely next token—Lily simplifies the generation loop. This design choice is reflected in src/generate.rs, which manages the tokenizer-backed decode sequence without the overhead of complex sampling parameters like temperature or top-p.

The Session Prefix Cache

One of the distinctive features of Lily is its implementation of a fixed two-entry Least Recently Used (LRU) session prefix cache, located in src/serve/session.rs. This cache is designed to improve efficiency during interactive conversations. When a new prompt is received, the server checks if the token sequence matches a sequence already stored in the cache. A state is only reused if the new prompt's token sequence is a strict prefix of the existing entry.

Users can manage this behavior through an optional prompt_cache_key. While providing a key allows the server to prefer a specific cache entry, the system still mandates token equality to ensure accuracy. The server reports the number of reused tokens in the response metadata under usage.prompt_tokens_details.cached_tokens, providing transparency regarding the efficiency gains of the cache during multi-turn interactions.

API Surface and Implementation Limits

Lily exposes a minimal subset of the OpenAI chat completions API. It supports standard endpoints such as POST /v1/chat/completions, GET /v1/models, and GET /health. However, the request surface is deliberately constrained to maintain stability and performance. The server only accepts text-only system, user, and assistant messages. Furthermore, it enforces that the final message in a request must have the "user" role.

Many standard API features are explicitly rejected by Lily to maintain its minimal footprint. These include streaming responses, tool calling (function calling), multimodal content, and speculative decoding. The server also disables "thinking" templates for the checkpoint, focusing purely on direct text generation. The --max-seq flag controls the total token capacity for the prompt and completion, which is capped at 262,144 tokens by the kernel, though it is often further restricted by the checkpoint's own max_position_embeddings.

Open Source Contributions and Licensing

As part of the pplx-garden initiative, Lily is released under the Apache-2.0 license. This allows developers to inspect, modify, and integrate the inference server into their own workflows on Apple Silicon. The repository includes comprehensive testing suites, covering CPU-reference kernel tests, API surface validation, and tokenizer accuracy. For developers with the 35B checkpoint available, the repository provides golden tests to verify output consistency against known benchmarks.

Enjoyed this?

Get more posts like this delivered to your inbox.

Subscribe