BxGPT is a chat assistant for programming (debugging, explaining, refactoring, architecture) and general questions. A Next.js app streams answers token by token from an open-weight model (Qwen2.5-Coder 7B) running in Ollama on the same server. It needs no account, keeps conversation history in the visitor's browser & never exposes the model API to the internet.
The problem
Hosted AI assistants send every conversation to a third party & self-hosting usually assumes a GPU. The goal was a private, public-facing assistant on a single 8 GB, 4 vCPU VPS with no GPU, shared with other sites, that stays responsive and fails gracefully under load.
The approach
Nginx terminates TLS and proxies to a standalone Next.js server on loopback, which talks to Ollama on loopback too. Every chat request passes an origin check, a Zod schema, a model allow-list & a per-IP rate limit, then waits in a bounded queue for the single inference slot. History is trimmed to fit the context window, tokens stream back as NDJSON & a client disconnect aborts generation upstream. The model stays loaded in RAM, warm-up units reload it after restarts & systemd caps memory and CPU so the other sites on the box keep running.
- Token-by-token streaming with queued, loading & thinking phases
- Markdown with GFM tables, syntax-highlighted code & copy buttons
- Stop (button or Esc), regenerate & retry after interrupted replies
- Local conversation history: rename, filter, delete & export as Markdown
- Settings for theme, model, temperature, reply length & custom instructions, enforced on the server
- Per-reply stats from Ollama (tokens, tokens/s, duration) & a live model status chip
- Landing page with a working prompt box, live status & a spec sheet read from the server config
- Responsive from 320 px, keyboard accessible & checked with axe in both themes
app/ landing, /chat, /privacy, /terms, API, sitemap, OG image
api/chat|health|models route handlers (logic lives in lib/server)
components/
chat/ workspace, transcript, composer, sidebar, settings
site/ landing header/footer/prompt/status
lib/
config/server.ts Zod-validated environment
ollama/ HTTP adapter + typed errors
inference/limiter.ts concurrency limiter with bounded queue
security/ rate limiter, client key, origin check
chat/ system prompt, context-window fitting
server/ chat handler, health, model catalog
client/ browser storage, stream reader
tests/unit|integration|e2e + Ollama mock
scripts/ deploy, rollback, warm-up, monitor, verifybrowser ──POST /api/chat──▶ Nginx (TLS, coarse rate limit, no buffering)
└▶ Next.js route handler
1. origin check (CSRF) → 403
2. body size + JSON + Zod schema → 400 / 413
3. model allow-list → 400
4. per-IP rate limit + 1 active → 429
5. trim history to num_ctx → 413
6. Ollama up, model installed → 503
7. queue full? → 503
── stream starts (NDJSON) ──
8. wait for an inference slot
9. relay tokens from Ollama
10. timeouts → error event
11. client disconnect → abort upstream
12. release slot, log metadata (never content)Challenges
- • Fitting a 7B model (about 4.8 GB loaded) plus Node.js, Nginx & other sites into 8 GB of RAM without out-of-memory kills.
- • Keeping one CPU inference slot fair across visitors with a bounded queue, per-IP limits & first-token, idle and total timeouts.
- • Making failed or cancelled generations recoverable: partial text is kept, the reply is marked interrupted & Retry is offered.
Lessons learned
- • On CPU, memory bandwidth sets the speed, so honest UI states (queued, loading the model) matter as much as raw throughput.
- • Warm-up has to use the same context size as chat requests, or Ollama reloads the model and the warm-up is wasted.
- • Keeping state in the browser removes a database to run, secure & back up.