Muse Ember
Meta’s multimodal model handles repeated coding subtasks, tool calls, routing and document processing inside agent workflows.
TPU Sync
github.com
TPU Sync orchestrates inference caches and transfers model state between Cloud TPU workloads through native C++ and framework-specific packages.
Visit website
Pricing
Free
Available on
Google Cloud TPU, JAX, PyTorch
Category
Developer Tools
What is TPU Sync?
TPU Sync is Google’s open-source C++ library for KV-cache orchestration, multi-tier memory management and cross-node transfers on Cloud TPU. It serves ML infrastructure engineers building inference systems and reinforcement-learning pipelines.
Separate JAX and PyTorch wheels connect the library to framework environments. Official examples demonstrate prefill-to-decode cache movement, host-memory offloading and transfer microbenchmarks, giving teams starting points for evaluating their serving architecture.
Key features
Zero-copy framework access
Provides zero-copy access to JAX arrays and PyTorch tensors.
Asynchronous transfers
Performs asynchronous data transfers through native C++.
Host-memory cache offloading
Moves KV-cache blocks between accelerator memory and host RAM.
Shared-memory cache persistence
Preserves host-memory caches across serving-process restarts.
Trainer-to-sampler transfers
Supports movement of weights and activations between reinforcement-learning trainers and samplers.
Who is it for?
Inference infrastructure teams
Evaluate cache transfers between separate prefill and decode workers.
Serving platform engineers
Explore host-RAM offloading for prefix caches under accelerator-memory pressure.
Reinforcement-learning engineers
Integrate model-state transfers into trainer and sampler workflows.
How to use TPU Sync
- Prepare validated Cloud TPU hardware and a compatible Python environment.
- Choose a verified commit using the repository’s release guidance.
- Build the selected commit from source, or install the corresponding exactly pinned framework wheel. Use the verified commit-to-nightly mapping for PyTorch wheels.
- Install matching framework dependencies, including the required torch_tpu and torch versions for PyTorch.
- Run a relevant official transfer or serving example before integrating the library.
TPU Sync pricing
Free
The library is available under Apache License 2.0. Cloud TPU infrastructure is billed separately, with charges varying by product, deployment model and region. The linked page covers underlying compute costs.
Check current pricing ↗What to know before you use it
- APIs evolve rapidly. Pin a verified revision or corresponding package version when evaluating the library.
- Validated hardware is Cloud TPU v7x and v6e, with Python 3.12.
- Compiled extensions require matching framework dependencies. PyTorch requires compatible torch_tpu and torch installations, while JAX dependencies must match its compiled stack.
- The PyTorch release path is still maturing, with end-to-end performance release gating still developing.
Frequently asked questions
Which Python packages should I install?
Use tpu-sync-jax or tpu-sync-torch. The release documentation identifies the former tpu_raiden distribution names as retired and outside the project’s control on PyPI.
What can I evaluate with the examples?
Examples cover single-host and multi-host prefill-to-decode KV-cache transfers, host-RAM cache offloading, and host-to-device or device-to-host transfer microbenchmarks.
Does the free license cover TPU usage?
Cloud TPU usage has separate infrastructure charges. Google lists rates by TPU product, deployment model and region.