02 / Distributed Backend Systems
In progressHigh-Throughput Asynchronous AI Gateway & Task Engine
Target role · Software Engineer (Backend)
An asynchronous gateway with distributed tasks, provider fallback, token-bucket limits, deduplication, and auditable usage.
- Target: 250+ concurrent requests
- Target: sub-50ms queue latency · >80% coverage
Problem & direction
Core problem: Direct LLM calls block web threads, time out, lack traffic controls, and create provider lock-in.
An in-progress asynchronous gateway designed to separate model requests from background execution while adding traffic controls, semantic deduplication, provider resilience, audit trails, and queue processing.
How it works
- 01
Validate through FastAPI and Pydantic v2
- 02
Apply token-bucket limits and LRU semantic deduplication
- 03
Route through LiteLLM with provider fallback
- 04
Execute with Celery and Redis, then record usage and latency in PostgreSQL
Engineering details
Token-bucket rate limiting
A token-bucket limiter controls incoming traffic while allowing bounded bursts, protecting the request pipeline from excessive demand.
Semantic caching
The caching layer targets semantically similar requests, reducing repeated model work rather than relying only on exact text matches.
Automated provider fallback
LiteLLM handles provider routing, with Groq as the primary path and OpenAI as the fallback when the primary provider cannot complete a request.
Non-blocking task execution
Celery and Redis provide background queue execution so long-running tasks can be separated from the FastAPI request path. PostgreSQL is part of the persistence stack.
Tests and continuous integration
Pytest and GitHub Actions will cover linting, testing, image builds, and registry publishing. More than 80% coverage, 250+ concurrent requests, and sub-50ms queue latency are targets.