PlanetScale Introduces Neki, a Sharded PostgreSQL System
PlanetScale has announced Neki in platform preview, bringing horizontal sharding and automated online operations to native PostgreSQL clusters.
PlanetScale's Neki distributes native PostgreSQL tables across multi-node clusters using dynamic routing and declarative topologies.
- PlanetScale launched Neki in platform preview, offering horizontally sharded PostgreSQL built from scratch rather than as a Vitess fork.
- Each shard runs native, unmodified PostgreSQL with one primary and at least two replicas across three availability zones.
- The system uses specialized routers, sidecar connection pooling, and a declarative JSON data topology to execute online schema changes, upgrades, and resharding.
Horizontal Scaling for PostgreSQL
PlanetScale announced the platform preview release of Neki, a distributed database management system designed to horizontally shard PostgreSQL. Developed by the engineering team behind the MySQL-based Vitess scaling technology, Neki addresses single-node hardware limits encountered by large database workloads.
According to PlanetScale's official announcement, the product was created in response to enterprise users outgrowing standard vertical scaling options. As single-node databases swell, teams encounter persistent operational hurdles, including extended vacuum durations, slow full-backup generation, transaction ID wraparound risks, and connection exhaustion. Rather than relying on custom database engines that simulate Postgres interfaces, Neki maintains standard, unmodified PostgreSQL engines on every shard.
Core Architectural Components
Neki’s operational model is split into four primary components that manage query lifecycle, connection resources, topology definitions, and node health:
- Neki Routers: Client applications interact directly with Neki routers using standard PostgreSQL wire protocol drivers and ORMs. Each router incorporates a PostgreSQL query parser, distributed planner, and buffering system. Routers inspect incoming queries, route planned tasks across relevant shards, and aggregate returned data into a single unified stream. Routers support horizontal and vertical scaling to prevent networking bottlenecks.
- Shards and Shard Groups: Every physical shard consists of an unmodified PostgreSQL deployment featuring a primary instance and at least two read replicas distributed across three availability zones. Tables and workloads are allocated to specific shard groups, each governed by configuration profiles defining instance capacities, storage quotas, replica topologies, and PostgreSQL parameter tunings.
- Sidecar Connection Pooling: Instead of employing external connection proxies like PgBouncer, Neki deploys sidecar processes directly alongside each database instance. Because the control plane coordinates both the routing layer and instance-level sidecars, connection pools are dynamically adapted according to actual compute node capacities.
- Control Plane: The control plane automates cluster management, supervising health checks, automated failovers, online zero-downtime resharding procedures, and version upgrades.
Topology Management and Online Operations
Database administrators define their sharding architecture through a declarative JSON data topology. This specification dictates shard keys, hashing strategies, and table partition boundaries. Routers cache this document locally to construct query execution paths without querying external state stores during execution.
Beyond horizontal partitioning, Neki adapts operational workflows so that maintenance tasks do not necessitate scheduled service windows. Schema migrations, software updates, and resharding routines operate as background tasks. Workflows deploy target nodes, synchronize data through replication, cut over active connections via internal functions such as __neki, and retire depreciated instances. PlanetScale platform features, such as database branching, schema recommendations, and Insights telemetry, remain fully supported across sharded clusters.
Availability and Platform Preview Scope
Neki is available directly within the PlanetScale platform under a preview designation. PlanetScale advised against deploying production-critical workloads to Neki during this evaluation window, noting that features and APIs may experience breaking changes before general availability.
Clusters can also be initiated without immediate sharding, running as a single primary instance with replicas. Under this configuration, operators can adopt Neki's connection pooling and online schema migration mechanisms before partitioning tables across multiple nodes later via background resharding workflows.
Enjoyed this?
Get more posts like this delivered to your inbox.
