Developers are locked into single providers and hit rate limits, capacity constraints, and sudden policy changes — forced to manually switch between models.
A drop-in proxy that accepts OpenAI-compatible API calls and routes them to the cheapest/fastest available provider (Claude, GPT, Gemini, DeepSeek, open-source) based on configurable rules, with automatic fallback on rate limits.
Usage-based pricing (small markup on routed tokens) plus subscription tiers for advanced routing rules and analytics
The pain signals are real and visceral — rate limits, outages, provider lock-in, and cost anxiety are daily frustrations for AI-heavy developers. The HN thread with 875 upvotes and 679 comments confirms broad resonance. However, many developers cope with manual switching or simple retry logic, so it's painful but not always blocking.
TAM is significant and growing: estimated 2-5M developers actively using AI APIs, with enterprise AI API spend projected at $15-25B by 2027. The routing/gateway layer could capture 2-5% of throughput as margin. However, the serviceable market (teams spending $100+/mo who care enough to add a proxy layer) is narrower — likely $500M-$1B addressable.
This is the critical weakness. Developers are notoriously resistant to paying for infrastructure middleware — they expect it to be open-source (LiteLLM exists). The value proposition is cost savings, which creates a ceiling: your markup must be less than the savings you provide. OpenRouter already proved the thin-margin model works but it's a tough business. Enterprise teams will pay for observability/compliance, but that's Portkey's game.
A solo dev can absolutely build an MVP proxy in 4-8 weeks. The core is a reverse proxy that translates API formats and applies routing rules — well-understood engineering. The hard parts (intelligent routing ML, comprehensive provider support, sub-100ms routing decisions at scale) come later but aren't needed for MVP. OpenAI-compatible proxy pattern is well-documented.
This is a crowded space. OpenRouter, LiteLLM, Portkey, Martian, and Unify collectively cover most of the value prop. OpenRouter dominates unified access, LiteLLM dominates open-source self-hosted, Portkey dominates enterprise observability, and Martian/Unify are attacking intelligent routing specifically. The 'configurable rules + automatic fallback' positioning overlaps heavily with LiteLLM's existing capabilities. Differentiation would require a genuinely superior routing algorithm or a dramatically better DX.
Excellent recurring potential. Once integrated into a codebase, switching costs are moderate. Usage scales with the customer's AI usage, which is growing. Token-based pricing naturally grows with customers. This is infrastructure — it becomes more valuable and harder to remove over time.
- +Validated, growing pain point with strong signal (875 HN upvotes, real quotes about frustration)
- +Infrastructure play with natural recurring revenue and usage-based growth
- +Technical MVP is very achievable for a solo dev in 4-8 weeks
- +Multi-model is becoming the default — tailwind for the category
- !Extremely crowded market — OpenRouter, LiteLLM, Portkey, Martian, and Unify are well-funded and established
- !LiteLLM is open-source and covers 80% of the use case for free, creating a pricing ceiling
- !Thin margins on token markup make unit economics challenging without massive scale
- !AI providers themselves are adding multi-model support (Azure AI, AWS Bedrock, Google Vertex) — platform risk
- !Willingness to pay for 'yet another proxy layer' is low when free alternatives exist
Unified API gateway providing access to 200+ AI models
Open-source Python library and proxy server that provides a unified interface to 100+ LLM providers. Drop-in OpenAI SDK replacement.
AI gateway and observability platform offering unified API access, caching, load balancing, fallbacks, and detailed analytics across multiple LLM providers.
AI model router that uses a trained classifier to automatically select the best LLM for each prompt based on quality and cost tradeoffs.
LLM routing platform that benchmarks models across providers and routes requests to optimize for quality, cost, or speed based on user-defined priorities.
An opinionated CLI tool + lightweight proxy (not a platform) that does ONE thing better than LiteLLM: automatic cost-optimized routing with zero config. Install, point your OPENAI_BASE_URL at it, add API keys, and it immediately starts saving money by routing simple tasks to cheaper models. Ship with a local dashboard showing dollars saved. Differentiate on DX and 'it just works' — not features.
Free open-source proxy (self-hosted, basic routing) -> Hosted cloud version with usage-based markup ($0.10 per 1M tokens routed) -> Pro tier ($29/mo) for advanced routing rules, analytics, team management -> Enterprise ($200+/mo) for SLAs, SSO, audit logs, custom routing models. Key insight: the free tier must be genuinely useful to build community against LiteLLM.
4-6 weeks to MVP, 8-12 weeks to first paying customer if targeting hosted version. However, reaching meaningful revenue ($5K MRR) likely takes 6-12 months given the competitive landscape and thin margins. You need ~50M tokens/day routed at $0.10/1M markup to hit $5K MRR.
- “forcefully cutting myself over to one of the alternative Chinese models”
- “Claude going into stupid mode 15 times a day, constant HTTP errors”
- “can't see myself justifying $200/mo on any replacement tool”