ngrok AI Gateway

Product Introduction

1. Definition: The ngrok AI Gateway is a hosted, unified API gateway and management platform specifically designed for Large Language Model (LLM) traffic. It acts as a single point of entry for routing, securing, and monitoring requests to any AI model, whether from public cloud providers (OpenAI, Anthropic), custom API endpoints, or self-hosted models (e.g., Ollama, vLLM).
2. Core Value Proposition: It exists to eliminate the infrastructure complexity and operational overhead of managing multiple LLM integrations. Its primary value is providing unified API management for AI models, enabling developers to route AI traffic with intelligent fallback, enforce granular access control, and gain centralized observability across all model providers from one dashboard, thereby increasing developer velocity and application reliability.

Main Features

1. Unified API Endpoint & SDK Compatibility: The gateway provides a single base URL (https://gateway.ngrok.ai) that is compatible with standard AI SDKs like OpenAI, Anthropic, and Vercel AI SDK. Technically, it works by intercepting API calls, parsing the requested model identifier, and dynamically routing the request to the correct backend provider based on configuration. This requires only a change in the baseURL and API key in the client code.
2. Private Connectivity for Self-Hosted Models: This feature allows local LLMs running on private infrastructure (e.g., on-premise servers, private cloud VMs) to be securely connected without exposing them to the public internet. It works by leveraging ngrok's globally distributed edge network to establish secure, outbound-only tunnels from the model's environment to the gateway, eliminating the need for public IPs, open inbound firewall ports, or complex VPN setups.
3. Bring-Your-Own-Key (BYOK) & Cost Pass-Through: Users can provision their existing API keys from providers like OpenAI and Anthropic directly into the ngrok AI Gateway. The gateway uses these keys to authenticate requests routed to those providers. Crucially, billing for inference costs remains directly with the provider, allowing users to retain their existing rate limits, credits, and negotiated enterprise contracts, while ngrok charges a separate, flat fee for the routing and management layer.
4. Intelligent Model Routing & Fallback: This is a programmable routing layer that allows developers to define primary and fallback models for each request. If a request to the primary model fails (due to errors, high latency, or quota exhaustion), the gateway automatically and instantly reroutes the request to a pre-defined healthy alternative. This is implemented at the gateway level, requiring no application-level retry logic, thus improving application resilience.
5. Scoped Access Keys & Usage Observability: Instead of sharing a single master API key, administrators can create multiple access keys with fine-grained permissions (scopes) that restrict which providers and models each key can call. All usage—successful calls, errors, token counts, latency, and inferred cost—is logged and aggregated in a central dashboard, providing clear visibility into usage patterns and costs per application, team, or API key.

Problems Solved

1. Pain Point: LLM Vendor Lock-in and Integration Spaghetti. Developers face significant overhead when integrating and switching between multiple AI providers, each with different SDKs, authentication methods, rate limits, and error formats. This creates fragile, hard-to-maintain code.
2. Target Audience: AI Application Developers & DevOps/Platform Engineers. This includes full-stack developers building LLM-powered apps, backend engineers managing production AI services, and platform teams responsible for providing standardized, secure, and observable AI infrastructure to internal development teams.
3. Use Cases:
   • Building a resilient AI-powered SaaS application that can automatically fail over from GPT-4 to Claude 3 during OpenAI API outages.
   • Securely exposing a privately fine-tuned or proprietary model to a production application without exposing the model's server to the public internet.
   • Managing cost and usage visibility across a large engineering organization, where multiple teams use various models, requiring centralized budgeting and audit trails.
   • Simplifying development and testing by allowing developers to seamlessly switch between a local Ollama instance for development and a cloud model for production using the same code and endpoint.

Unique Advantages

1. Differentiation: Unlike building a custom proxy or using provider-specific gateways, ngrok AI Gateway is provider-agnostic and includes built-in secure tunneling for private models—a combination not offered by cloud AI providers. Compared to generic API gateways, it is pre-configured for AI-specific workflows like model routing, token counting, and AI SDK compatibility.
2. Key Innovation: The integration of its secure, outbound-only tunneling technology (core to ngrok's offering) directly into the AI Gateway workflow. This allows self-hosted models to be treated as first-class citizens alongside major cloud providers, enabling a true hybrid AI architecture without the security burden of managing public-facing endpoints.

Frequently Asked Questions (FAQ)

1. How does ngrok AI Gateway handle billing for OpenAI and Anthropic models? The ngrok AI Gateway operates on a Bring-Your-Own-Key (BYOK) model for major providers. You provide your own OpenAI or Anthropic API key, and you are billed directly by those providers for inference costs at your existing rates. ngrok charges a separate, flat fee per million tokens routed through its gateway for its routing, observability, and management services.

2. Is my AI prompt and response data secure with ngrok AI Gateway? Yes. ngrok AI Gateway is designed with security in mind. For public providers, your API key is stored securely. For private models, data is routed through ngrok's encrypted tunnels without being exposed to the public internet. The platform also offers features like data redaction for logging. You should review ngrok's security documentation and compliance certifications for specific details.

3. Can I use ngrok AI Gateway with locally hosted models like Ollama or Llama.cpp? Absolutely. This is a core strength of the ngrok AI Gateway. You can connect models running on your local machine or private server using ngrok's lightweight agent software, which creates a secure tunnel to the gateway. Your application then sends requests to the gateway, which routes them through the tunnel to your local model, keeping it private.

4. What happens if my primary AI model API goes down or times out? The ngrok AI Gateway's intelligent routing feature allows you to configure fallback models. If a request to your primary model (e.g., gpt-4) fails or exceeds a latency threshold, the gateway will automatically and immediately retry the request using your specified fallback model (e.g., claude-3-sonnet). This built-in resiliency happens at the infrastructure level, preventing application errors.

5. How does the observability dashboard help with cost management? The dashboard aggregates token usage, request counts, and errors across all providers and models. By applying the current pricing of each model (e.g., OpenAI's per-token cost), it provides an estimated cost breakdown per API key, model, and time period. This centralized view is crucial for identifying expensive models, detecting anomalous usage, and allocating costs accurately across teams or projects.