Meta is running production AI inference on Google’s Tensor Processing Units through a new PyTorch-native serving stack, giving Google its clearest public proof yet that TPUs can support a major external customer without forcing that customer to rebuild its software around JAX.
The deployment uses SGLang-TorchTPU, one of two open-source inference engines Google unveiled on October 20 alongside the release of TorchTPU as open source. Meta’s PyTorch-TPU lead, Zongwei Zhou, described the system as running “in production at Meta today” after joint work by Meta, Google and the SGLang team.
Neither company has disclosed the size of the deployment, which Meta products it serves, where the hardware is located or the economics of the arrangement. The confirmed development is narrower than claims that Meta has broadly replaced Nvidia GPUs. It is still consequential: a company that created PyTorch and operates some of the world’s largest AI systems has put Google’s PyTorch path into live service.
TorchTPU changes the software path, not just the chip choice
Google TPUs have long supported PyTorch through PyTorch/XLA, but developers frequently encountered a different execution model and a porting burden compared with the CUDA-centered tools used on Nvidia GPUs. TorchTPU is designed to make a TPU look like a native PyTorch device instead.
Tensors remain standard PyTorch tensors on a tpu device. Code can run eagerly for development and debugging, then compile through torch.compile and XLA for production. Distributed jobs use torch.distributed, while the serving engines retain their familiar schedulers, API servers and key-value cache managers.
The two new serving backends take slightly different routes:
- SGLang-TorchTPU, co-developed with RadixArk, is the engine confirmed in production at Meta.
- vLLM-TorchTPU, built with Inferact, is an out-of-tree vLLM platform plugin that keeps vLLM’s command line, OpenAI-compatible API server, scheduler and model definitions.
That separation matters. “Native PyTorch” does not mean every low-level implementation is hardware-neutral. The engines reuse upstream model code where possible, then call TPU-specific Pallas kernels for performance-sensitive work such as attention, Mixture-of-Experts routing and quantized matrix multiplication. Portability now reaches much farther up the stack, but optimized kernels and layout choices still determine whether a model runs economically.
The serving features are built for agent workloads
Google’s engine announcement lists automatic prefix caching, host-memory cache offload, speculative decoding, structured output, tool calling, long-context support and multimodal inputs. Those are not peripheral features for modern inference systems. Multi-turn agents repeatedly send growing conversation histories, making cached-prefix reuse and memory placement central to latency and cost.
The stack can also separate prefill and decode across different TPU hosts. Prefill processes the input context; decode generates the output tokens. Google’s open-source TPU Sync library moves the key-value cache between those hosts without an extra staging copy in host memory, while transfers run outside Python and overlap with model execution.
For vLLM, speculative decoding supports MTP, EAGLE-3, DFlash, DSpark, n-gram and suffix-based approaches. Its automatic prefix cache also handles hybrid architectures such as Qwen 3.5 and Kimi K3, where the system must preserve linear-attention state alongside the conventional KV cache.
Model support is broad, but production readiness is uneven
The TorchTPU architecture paper reports that 4,821 of 4,917 tested Hugging Face models ran without crashing, while 638 of 657 representative architectures matched GPU numerical results under the project’s test criteria. Those figures show wide framework coverage, but they should not be read as proof that 98% of models are tuned, cost-effective and ready for an enterprise service-level objective.
The SGLang project’s own support matrix makes the distinction explicit. Optimized configurations currently include GPT-OSS 120B, Qwen3-Coder 480B-A35B, Qwen 3.5 397B-A17B, GLM-5.3-Flash and Kimi K3. Other models sit in “supported,” “quickstart” or “experimental” tiers, with specific quantization formats and host topologies attached to each result.
vLLM likewise recommends Ironwood, also called TPU v7x, and tests that generation on every pull request. Earlier v6e, v5e and v5p hardware is currently described as experimental in the vLLM-TorchTPU release notes. A team evaluating existing TPU capacity should not assume that an older accelerator inherits Ironwood’s validated feature set.
Google’s benchmark numbers need workload-matched testing
Google reports a cost of 2.01 cents per million tokens for a Qwen 3.5 configuration in the SemiAnalysis InferenceX benchmark and 0.83 cents per million tokens in its AgentX submission. The company also says a Kimi K3 setup built around a Pallas megakernel served more than 700 tokens per second to one user on Ironwood.
Those results demonstrate that the stack can be competitive after substantial tuning. They do not establish a universal TPU advantage. Model architecture, prompt and output lengths, request concurrency, latency targets, quantization, cache-hit rate, region capacity and committed-use pricing can all reverse a cost comparison.
The paper provides another useful caution. Its strong Qwen 3.5 inference results required targeted work on expert-parallel communication, kernel fusion and runtime behavior. In one optimization, the team collapsed an entire Mixture-of-Experts layer into a single SPMD program after a reduce-scatter operation consumed 22% of device time in a prefill-heavy test. The PyTorch interface removes much of the migration friction; it does not remove performance engineering.
What AI platform teams should test before moving
A credible evaluation should begin with the exact production workload rather than a generic tokens-per-second chart. Teams should compare time to first token, inter-token latency and total throughput at their real concurrency levels, then repeat the test with expected context lengths, tool-call patterns and cache reuse.
They should also verify numerical behavior and output quality under the intended quantization format, measure cold-start and compilation effects, and confirm that required observability, autoscaling and failure-recovery paths work on the selected topology. Multi-host serving adds operational questions around cache transfer, host failure and capacity fragmentation that a single-host benchmark will not reveal.
Finally, pricing should include engineering time and portability risk. A model that launches with unchanged Python code may still need TPU-specific kernels, sharding decisions or serving recipes to hit its target. The public roadmaps call for automated model onboarding, more routing features and broader kernel interfaces, which means important parts of the stack are still moving.
TorchTPU arrives one day after Amazon introduced TorchNeuron for native PyTorch on Trainium, while open orchestration projects such as llm-d are trying to make inference less dependent on one accelerator. The competitive shift is therefore larger than a Meta-Google deal: custom-chip vendors are attacking Nvidia’s software advantage by meeting developers inside PyTorch, vLLM and SGLang.
Meta’s deployment proves that approach can reach production. The unanswered questions—scale, product scope, availability and sustained cost—will determine whether it becomes a broad infrastructure change or a strategically useful second source.
