Zero Runtime Inference Gateway

The Inference Gateway is a unified entry point for the model stack. Instead of holding a separate API key for each provider, you authenticate once with your Zero Runtime auth token and route speech-to-text, LLM, text-to-speech, turn detection, and denoise through the gateway. It manages the upstream provider connections, resampling, and streaming server-side.

Setup

Gateway providers connect using your Zero Runtime auth token, no provider API keys are required. Set it in the worker environment:

export ZERORUNTIME_AUTH_TOKEN=<token>

Usage

Import gateway providers from zeroruntime.inference (Python) or @zeroruntime/js-sdk/inference (Node JS) and drop them into the pipeline exactly like normal plugins. Every gateway class is a drop-in replacement for its plugins equivalent. Turn detection is the unified TurnDetector, imported from the same inference module, using an Echo model, which also routes through the gateway:

from zeroruntime import Pipeline
from zeroruntime.inference import AICousticsDenoise, CartesiaTTS, GoogleLLM, SarvamAISTT, TurnDetector
from zeroruntime.plugins import SileroVAD

pipeline = Pipeline(
    stt=SarvamAISTT(),        # via gateway
    llm=GoogleLLM(),          # via gateway
    tts=CartesiaTTS(),        # via gateway
    vad=SileroVAD(),          # local plugin
    turn_detector=TurnDetector(model="echo-large"),  # Echo, via gateway
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"),
)

import { Pipeline } from '@zeroruntime/js-sdk';
import { AICousticsDenoise, CartesiaTTS, GoogleLLM, SarvamAISTT, TurnDetector } from '@zeroruntime/js-sdk/inference';
import { SileroVAD } from '@zeroruntime/js-sdk/plugins';

const pipeline = Pipeline({
  stt: SarvamAISTT(),  // via gateway
  llm: GoogleLLM(),  // via gateway
  tts: CartesiaTTS(),  // via gateway
  vad: SileroVAD(),  // local plugin
  turn_detector: TurnDetector({ model: 'echo-large' }),  // Echo, via gateway
  denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }),
});

You can mix gateway and direct-provider components freely in the same pipeline. For example, run STT and TTS through the gateway while the LLM uses its own key.

Supported providers

Stage	Gateway providers
Speech-to-text	Deepgram, Google, Sarvam AI, AssemblyAI, Cartesia
LLM	Google, Sarvam AI
Text-to-speech	Cartesia, Google, Deepgram, Sarvam AI
Turn detection	Echo (echo-small, echo-large, echo-omni)
Denoise	ai-coustics, Sanas

Import paths

Every gateway class is imported from the single inference module of each SDK:

Stage	Python	Node JS
Speech-to-text	`from zeroruntime.inference import DeepgramSTT, GoogleSTT, SarvamAISTT, AssemblyAISTT, CartesiaSTT`	`import { DeepgramSTT, GoogleSTT, SarvamAISTT, AssemblyAISTT, CartesiaSTT } from '@zeroruntime/js-sdk/inference'`
LLM	`from zeroruntime.inference import GoogleLLM, SarvamAILLM`	`import { GoogleLLM, SarvamAILLM } from '@zeroruntime/js-sdk/inference'`
Text-to-speech	`from zeroruntime.inference import CartesiaTTS, GoogleTTS, DeepgramTTS, SarvamAITTS`	`import { CartesiaTTS, GoogleTTS, DeepgramTTS, SarvamAITTS } from '@zeroruntime/js-sdk/inference'`
Turn detection	`from zeroruntime.inference import TurnDetector` (Echo models route through the gateway)	`import { TurnDetector } from '@zeroruntime/js-sdk/inference'`
Denoise	`from zeroruntime.inference import AICousticsDenoise, SanasDenoise`	`import { AICousticsDenoise, SanasDenoise } from '@zeroruntime/js-sdk/inference'`

Why use it

One credential. Authenticate with a single Zero Runtime token instead of provisioning and rotating a key for every provider.
Less client work. Connection management, resampling, and streaming happen server-side.
Easy switching. Swap a gateway provider without onboarding a new API key.

What’s next

Configure a Pipeline

How gateway providers slot into a pipeline, with cascading and realtime examples.

Plugins

The full catalog of STT, LLM, TTS, and denoise providers.

Overview

Build

Plugins

Deploy

Zero Runtime Inference Gateway

Setup

Usage

Supported providers

Import paths

Why use it

What’s next

Configure a Pipeline

Plugins

​Setup

​Usage

​Supported providers

​Import paths

​Why use it

​What’s next

Configure a Pipeline

Plugins

Setup

Usage

Supported providers

Import paths

Why use it

What’s next