PyTorch Introduces Cross-Repository CI Relay for Ecosystem Testing

AIDeveloper44 Team

As the primary infrastructure layer for a vast machine learning ecosystem, PyTorch supports numerous hardware backends and downstream software libraries. Components like Intel XPU, AMD ROCm, Apple MPS, and ecosystem projects such as vLLM and Hugging Face Transformers maintain their own separate repositories. Previously, testing compatibility between the main pytorch/pytorch repository and these out-of-tree dependencies created a coordination gap. Upstream maintainers lacked visibility into whether a new pull request (PR) broke external components, and downstream maintainers relied on manual checks to discover upstream changes.

To resolve this blind spot, PyTorch developers implemented the Cross-Repository CI Relay (CRCR). The CRCR operates as a fully automated pipeline that triggers continuous integration (CI) workflows in downstream repositories whenever a PR is opened or a commit is pushed to the core PyTorch codebase. The pipeline routes the status and results back to the primary PyTorch CI HUD, offering maintainers a singular dashboard to monitor overall ecosystem health without requiring downstream repositories to engineer custom dashboard integrations.

End-to-End Execution Architecture

The CRCR is built utilizing AWS Lambdas, GitHub Actions, and an ecosystem of data storage solutions. The end-to-end data flow operates through five distinct stages:

• Webhook Reception: When a pull or push event occurs on pytorch/pytorch, GitHub transmits a signed webhook to a dedicated Webhook Lambda running on AWS. The Lambda verifies the HMAC-SHA256 signature and loads a tiered allowlist to determine which projects to notify.
• Fan-Out Dispatch: The Lambda broadcasts a repository_dispatch event concurrently to all allowlisted external repositories. The event payload carries necessary PR context, including the commit SHA and PR number. At this point, a DISPATCHED state is registered in a Redis cache to track timing.
• Downstream CI Execution: Receiving repositories initiate their own CI workflows using the specified PR's commit SHA. A composite GitHub Action named cross-repo-ci-relay-callback is employed to report standard status updates: in_progress when the job starts, and completed when it finalizes.
• Authenticated Callback: The GitHub Action generates a GitHub OpenID Connect (OIDC) token. A separate Callback Lambda receives this token, verifying it cryptographically to prove the origin repository. The Lambda applies rate limits, checks the state machine, and forwards valid results.
• HUD Persistence: A backend API writes the testing records into Amazon DynamoDB. A real-time replicator streams this data into ClickHouse, enabling the frontend PyTorch CI HUD to display the external test results within seconds.

Tiered Participation Levels

To support incremental onboarding across various project maturities, CRCR employs a tiered allowlist spanning four participation levels:

• Level 1 (Notify): Entry point for projects. They receive dispatches to trigger CI but do not report results back to PyTorch.
• Level 2 (Report): Full pipeline execution where the external project runs compatibility checks and reports results directly to the CI HUD dashboard for upstream visibility.
• Level 3 (Signal): Generates a non-blocking check run directly on the upstream GitHub PR tab.
• Level 4 (Gate): Reserved for critical downstream dependencies. This level creates a blocking check run that prevents an upstream PR from merging if it causes external breakage. Requires dedicated on-call contacts.

Security and Data Integrity Modeling

Accepting and displaying continuous integration outputs from distinct external environments necessitates rigid security controls. The CRCR implements a five-layer security model to authenticate records.

Central to this process is GitHub OIDC Identity Verification. The system never relies on a self-reported identity from a callback payload. Instead, it extracts the repository claim cryptographically bound to the RS256 JWT token. Following identity verification, an Allowlist Authorization check confirms the calling repository is explicitly listed at Level 2 or higher in the upstream YAML configuration.

Additionally, the architecture utilizes State Machine Validation backed by Redis. This restricts workflows to valid transitions: a callback is rejected if a job reports a COMPLETED status without previously registering an IN_PROGRESS state, or if duplicate updates are received. This prevents external repositories from injecting arbitrary results for testing events that were never dispatched.

Finally, data forwarded to the PyTorch HUD is separated into namespaces. Values verified by the relay, such as the repository origin and calculated execution times, are placed in a trusted block, while self-reported test totals reside in an untrusted block presented as informational context.

Future Enhancements

The infrastructure provides granular data to track two primary metrics: queue time (delays before execution starts) and execution time (duration of the test). PyTorch plans further extensions to the relay system, including a process for stale job cleanup to manage jobs that fail to report completions, broader implementations of upstream merge gating using L3 and L4 tiers, and extending dispatch support beyond PRs to include continuous integration events on the post-merge main branch.

Enjoyed this?

Get more posts like this delivered to your inbox.

Subscribe