OpenAI’s Pando Rebuilds Git for Codex-Scale Development

OpenAI’s Pando backend cut median Git push latency by nearly 80%. Here is how causal consistency, reusable packs and regional caches make it work.
Laptop screen showing code at a developer workstation
Photo: Mahmudul Hasan / Unsplash

OpenAI has replaced the source-control backend behind its main code repository with Pando, a Git-compatible system built to absorb the surge in branches, pushes and CI fetches created by parallel Codex work.

The company disclosed the system on November 2 after running it in production since April. In OpenAI’s engineering account, Pando reduced median push latency by nearly 80%, improved merge reliability and delivered close to a petabyte of Git data per day. Its busiest reported hour moved more than 57 terabytes of Git pack data, with over 99% served from local disk caches.

Pando does not replace the Git commands engineers use. It changes what happens behind the server endpoint: pushes become immutable objects in blob storage, branch pointers are coordinated independently, and clients fetch previously stored packs through regional caches. That design sacrifices a repository-wide, point-in-time view of every branch in exchange for much higher concurrency.

Codex turned source control into a parallel-systems problem

OpenAI traced the bottleneck to the compound effect of agent-assisted development. An engineer can have several changes ready at once; each push can trigger multiple CI jobs; and development environments, research jobs and automation may all fetch the same new history. By late 2025, the existing monorepo service was producing slow pushes and fetches, failed merges and reliability that sometimes fell below 90%.

The pressure was not simply a large repository. It was a large number of writers acting on different branches while many readers repeatedly requested the same objects. Conventional distributed Git services often coordinate reference updates across replicas or order repository changes through a shared log. Both approaches can make unrelated pushes wait at a common coordination point.

Pando narrows that coordination boundary. Competing updates to the same branch still conflict, but pushes to unrelated branches can proceed independently. The system preserves serialization for each individual reference while dropping the requirement that every client see all repository references at one universal moment.

What happens during a Pando push

A Git push normally sends a packfile containing the objects the server needs to complete the requested reference update. Pando validates that incoming pack in a streaming pass and writes it to blob storage without first unpacking it into a local repository.

It then records a version of the branch reference in a metadata store. That record includes the new commit tip, the uploaded pack and any earlier reference versions on which the pack depends. An atomic update moves the branch pointer, providing read-your-own-write behavior. OpenAI describes the synchronous path as a constant number of metadata reads and writes, regardless of the push size.

The server also limits what it advertises during push negotiation. Showing every branch lets a client omit more objects from its upload, but at very large scale the advertisement itself becomes expensive and the resulting pack may depend on objects that another reader does not have. OpenAI noted that an advertisement containing millions of references can exceed 100 MiB.

By default, Pando advertises the designated trunk branch as the shared base. A feature-branch push may therefore resend some objects that exist on another side branch, but the pack is easier to reuse. In OpenAI’s monorepo, the median pack produced under this policy was under 20 KiB and the 99th percentile was below 3 MiB.

Fetches reuse packs instead of rebuilding them per client

The read path is the other half of Pando’s design. A conventional Git server compares the objects a client has with the objects it wants, then constructs and compresses a response pack. Those client-specific packs consume CPU and disk I/O and are difficult to cache.

Pando maps the client’s Git have and want commit IDs to stored reference versions, then selects existing packs that cover the missing history. It sends the locations through Git’s packfile-uri protocol feature, which was designed to let clients retrieve pack data from separate URLs such as a content-delivery layer. Pando sends an empty pack in the main response and directs the client to the real packs.

A regional service called Express caches those immutable packfiles. OpenAI reports cache hit rates above 99% and says fetches and clones can saturate 10 Gbps links on ordinary cloud virtual machines. Negotiation and data delivery can scale separately because the Git-serving process is no longer generating and transmitting every byte itself.

Thousands of tiny packs would eventually make catch-up operations slow, so Pando compacts them in the background. New pushes start as L0 packs, which are grouped into progressively larger covers up to L4 in the current monorepo. The hierarchy works like a skip list over branch history, allowing fetch planning in logarithmic time relative to the gap between the client’s current and requested versions.

The consistency tradeoff is deliberate

Pando provides causal consistency rather than a globally consistent snapshot of all references. A client fetching branches foo and bar might observe a newer version of one and an older version of the other if both move during negotiation. A second client could see the opposite combination.

What remains ordered is the history of each branch and the dependencies needed to reconstruct the state that produced it. That fits workflows in which engineers and agents collaborate through individual branches and eventually merge to a designated trunk. Tools that require a repository-wide atomic view of every reference would need closer scrutiny.

The tradeoff also explains why Pando is not simply a faster storage layer. Its write throughput depends on relaxing a consistency guarantee that many Git services assume, while its read throughput depends on shaping pushes so their packs can be reused later.

OpenAI built it with agents, but the evidence is still internal

OpenAI says Pando went from kickoff to production in 10 weeks. One engineer designed the core architecture, teams of coding agents implemented it and a small group of engineers deployed it. The Rust codebase was tested against standard Git behavior with a conformance suite and deterministic simulations intended to expose race conditions in negotiation and pack planning.

GPT-6 Astra agents also wrote Lean and TLA+ specifications to formalize parts of the design, while Daybreak-capable models were used to red-team Git and Pando. Deployment began as a read- and write-through proxy in front of the old system, allowing the two backends to stay synchronized while traffic and hundreds of integrations moved in phases.

The production numbers are substantial, but they are OpenAI’s own measurements. The company did not publish Pando’s source, independent benchmarks, infrastructure cost, repository object count or a breakdown of failure rates after migration. Its comparison with three-phase commit and write-ahead-log designs is explicitly a teaching simulation rather than a production benchmark.

What is available outside OpenAI

OpenAI has published its Git client fork and contributed fixes for parallel pack downloads, interrupted-transfer cleanup, tracing and pack validation. Some changes have reached upstream Git, while others remain in the public fork. Pando itself is still internal, although the company says it is exploring ways to share the technology with users.

The broader Git ecosystem is moving in a similar direction. Git’s packfile-uri capability has existed for years, and GitLab’s work around Git 2.56 describes pluggable object databases and server-supplied packfiles as foundations for storage backends that no longer assume the traditional on-disk object store.

Most engineering organizations do not need a Pando-scale replacement today. The useful signal is where pressure appears first: repeated CI clones, expensive pack generation, slow reference advertisement, repository-wide write coordination and multi-region fetch traffic. As coding agents multiply concurrent work rather than merely accelerate one developer’s session, source control becomes part of the agent infrastructure itself.

Previous Post
John Ternus, Apple's incoming CEO, seated for an official Apple portrait

Apple’s $114.5B Quarter Comes With a Siri AI Cost Warning

Related Posts
AWS Trainium3 AI chip on a circuit board

Amazon’s Trainium Talks Push AWS Chips Beyond the Cloud

AWS is in early talks to sell Trainium AI chips for use in other companies’ data centers, a shift that could move Amazon from cloud-only accelerator provider toward a more direct role in the AI chip market. The opportunity is real, but so are the constraints: Trainium capacity is already tight, Nvidia still owns the broadest software ecosystem, and selling racks outside AWS could weaken the cloud bundle that makes custom silicon so valuable to Amazon.
Read More