Together AI Unveils Production Inference Platform for Open-Weight Models

Together AI has announced a significant update to its inference platform, aiming to provide developers with comprehensive control over AI model performance, cost, and quality without the need to build custom infrastructure. This new Dedicated Model Inference platform allows models to go live in minutes, offering production-grade features from the outset. Key capabilities include managing multiple deployments behind a single stable endpoint, safe rollout strategies such as canary, blue-green, and rolling updates with automatic rollback, and sophisticated testing methods like A/B and shadow testing. The platform also supports autoscaling across multiple regions and introduces a closed beta for custom training, encompassing full-weight and LoRA reinforcement learning, as well as supervised fine-tuning.

Details: Enhanced Deployment and Control

The updated platform is built to streamline the entire model lifecycle, from experimentation to production. Users can deploy models from Together's own platform or bring their own open-weight or fine-tuned models, uploading weights or adapters from various sources like Hugging Face, S3, or local machines. The system is designed to support the full journey of a model, ensuring a smooth transition from early fine-tuning stages to serving live production traffic. A critical aspect of the update is the flexibility in how models are run; the platform abstracts away the complexity of choices regarding hardware, quantization, parallelism schemes, and serving engines, offering pre-tested and optimized deployment profiles. These profiles allow users to achieve strong results quickly, with options to further customize for latency, throughput, or a balance between the two as their needs become more specialized.

Furthermore, Together AI has significantly improved model startup times. By rebuilding the model caching and distribution layer, weights are shared across the fleet and proactively warmed, leading to approximately 4x faster warm starts for frontier models. Deployment times vary by model size, ranging from approximately 2-5 minutes for small models to 7-14 minutes for frontier-scale MoE models. The platform also offers granular control over scaling, allowing users to pin deployments to specific regions or enable flexible placement. Autoscaling can be configured based on various signals beyond generic metrics, including inflight requests, GPU utilization, time to first token, latency, decoding speed, or throughput, catering to diverse workload demands.

Context: The Rise of Open-Weight Models and Inference Challenges

The strategic shift towards open-weight models has been driven by their increasing quality, cost-effectiveness, and customizability, making them a foundation for many AI products and agents. Unlike closed-source alternatives, open-weight models offer teams greater control over performance, quality, and functionality, which is crucial for maintaining user experience and protecting proprietary intellectual property. However, optimizing inference for these models involves navigating a complex matrix of technical decisions, from quantization levels and parallelism schemes to engine parameters and model architectures. The rapid advancement of AI research means that staying current with the best techniques for achieving optimal quality and performance is a constant challenge for development teams.

This complexity often leads to teams spending significant time and resources building and managing their own inference stacks and endpoint management layers. Together AI's new platform aims to eliminate this burden by applying lessons learned from serving over 400 trillion tokens per month. The goal is to provide an inference solution that delivers frontier levels of performance, quality, and efficiency by continuously incorporating vetted research, thereby preventing lost time, wasted spend, and unintended downtime. This update consolidates advances from Together AI's research, model optimization, and platform engineering teams into a unified service.

Impact: Empowering Developers with Production-Ready Tools

This release empowers developers by abstracting away much of the underlying infrastructure complexity associated with deploying and managing AI models in production. The ability to move from experimentation to production without changing platforms or rebuilding deployments is a significant advantage. Features like stable endpoints that can host multiple evolving deployments, and safe rollout mechanisms such as canary and blue-green updates, reduce the risk associated with deploying new model versions. Max Lu at Decagon highlighted this benefit, stating, "We will go from ‘fast inference’ to ‘fast inference we can safely iterate on every week.’" This capability is crucial for teams aiming for continuous improvement and rapid iteration cycles.

Moreover, the integration of A/B testing and shadow traffic allows for real-world performance evaluation without impacting end-users. This enables developers to test new weights, configurations, or hardware using actual production traffic, moving beyond the limitations of offline benchmarks. The platform also provides robust observability through a Prometheus endpoint and an enhanced in-product analytics experience, making it easier to monitor deployment health, performance, and scaling behavior. The introduction of the custom training beta further bridges the gap between training and inference, allowing checkpoints to be deployed directly to production environments, streamlining the workflow from experimentation to serving.

What's Next: A Foundation for Integrated AI Workflows

Together AI views this launch as the initial step in building a comprehensive platform that covers the entire AI model lifecycle—from training and optimization to deployment, measurement, and continuous improvement. The company plans to release more in-depth content exploring the systems behind the platform, including details on deployment profiles, model startup optimization, autoscaling strategies, observability features, and rollout mechanisms. The ultimate vision is to make the control offered by open-weight models accessible without turning inference into a complex DIY infrastructure project, providing sensible defaults and optimized configurations while retaining deeper control for specialized needs. The new Dedicated Model Inference platform is available now for teams looking to accelerate their journey from iteration to production and maintain ongoing improvements to their deployed models.