Agent Router Capabilities
Welcome to the Agent Router capabilities documentation! This section provides detailed information about the various features and capabilities that Agent Router offers to help you manage and optimize your AI/LLM traffic.
LLM Providers Integrations
Support for various Large Language Model providers:
- Connecting to AI Providers: Learn how to establish connectivity with any supported AI provider
- Supported Providers: Compatible AI/LLM service providers
- Supported Endpoints: Available API endpoints and operations
- Vendor-Specific Fields: Use backend-specific parameters and access provider-unique capabilities in your OpenAI-compatible requests
- Prompt Caching: Provider-agnostic prompt caching using unified cache_control API
Inference Optimization
Advanced inference optimization capabilities for AI/LLM workloads:
- InferencePool Support: Intelligent routing and load balancing for inference endpoints
- HTTPRoute + InferencePool: Basic inference routing with standard Gateway API
- AIGatewayRoute + InferencePool: Advanced AI-specific routing with enhanced features
Gateway Configuration
- GatewayConfig: Gateway-scoped configuration for the external processor (env vars, resources, shared settings)
- Scaling: Configure multiple controller replicas and horizontal pod autoscaling for production deployments
Traffic Management
Comprehensive traffic handling and routing capabilities:
- Model Virtualization: Abstract and virtualize AI models
- Provider Fallback: Automatic failover between AI providers
- Quota Policy: Token-based quota management for controlling total consumption budgets
- Usage-based Rate Limiting: Token-aware rate limiting for AI workloads
- Header and Body Mutations: Customize HTTP headers and JSON body fields per backend or route
Security
Robust security features for AI gateway deployments:
- Upstream Authentication: Secure authentication to upstream AI services
Model Context Protocol (MCP)
Connect AI agents to external tools and data sources:
- MCP Gateway: Server multiplexing, tool routing, OAuth authentication, and observability for MCP workloads
Observability
Monitoring and observability tools for AI workloads:
- Metrics: Comprehensive metrics collection and monitoring