HyperAI runs on a budget that would make a hyperscaler laugh. Two A100 80GB GPUs, shared across multiple customer workloads, serving Vietnamese and English inference simultaneously. This post documents the decisions we made to get there and the ones we wish we'd made earlier.

Why bilingual inference is actually hard

The challenge with Vietnamese + English LLM serving isn't linguistic — modern large multilingual models handle both adequately. The challenge is performance parity. English performance on open-weight models is heavily optimized because the pretraining data skews overwhelmingly English. Vietnamese is there, but thinner.

For our use cases — document summarization, internal Q&A, report generation — "adequate" Vietnamese quality isn't good enough. Customers comparing the output to their human colleagues will notice. This pushed us toward models with stronger multilingual pretraining, which are generally larger, which means tighter GPU headroom.

Model selection

We evaluated six models across three dimensions: Vietnamese task quality, inference throughput at our hardware constraints, and licensing terms compatible with enterprise deployment.

The short version of what we found:

  • Qwen2.5 72B gave the best Vietnamese quality in our evaluation set, but at 72B parameters even with FP8 quantization it maxed out a single A100 and left nothing for concurrent requests.
  • Qwen2.5 32B in FP8 was the sweet spot for our setup — good Vietnamese quality, fits comfortably on one A100 with room for batching, permissive license for commercial use.
  • Llama 3.1 70B performed well on English but noticeably worse on Vietnamese, particularly on domain-specific terminology in healthcare and legal text.

Quantization choices

We run FP8 as our default. The quality-to-throughput tradeoff at FP8 is now well understood: you lose roughly 1–2% on standard benchmarks, gain 30–40% throughput improvement over BF16, and the VRAM savings let you run larger models on the same hardware.

For our specific use case, we found that FP8 quality degradation was uneven across languages. Vietnamese showed slightly more degradation than English — particularly on tonal accuracy in poetry or formal registers. For our business use cases (summarization, Q&A, data extraction), this was acceptable. For use cases requiring high-register Vietnamese output, it might not be.

We evaluated AWQ (4-bit) for tasks where throughput was critical and quality tolerances were wider. Results were usable for classification and extraction but not for generation tasks where customers would read the output directly.

Serving infrastructure

We run vLLM with PagedAttention for KV cache management. This was not a close call — vLLM's continuous batching made a meaningful difference in practical throughput at our load patterns (burst traffic during business hours, nearly zero overnight).

One non-obvious thing: Vietnamese text tokenizes differently from English in ways that affect latency predictions. Vietnamese words are often shorter tokens but sentences can be denser with meaning, meaning output token counts can be higher for equivalent-length prompts. Our initial latency budgets were calibrated on English and needed recalibration once we measured Vietnamese workloads specifically. 

# Rough throughput comparison at our hardware
# Single A100 80GB, Qwen2.5-32B-FP8, vLLM

BF16 (baseline):    ~420 tokens/sec output
FP8:                ~590 tokens/sec output  (+40%)
AWQ 4-bit:          ~780 tokens/sec output  (+86%)

# Vietnamese generation latency vs. English
# (same task, same model)
English TTFT:       ~180ms
Vietnamese TTFT:    ~210ms  (+17%)
English output:     avg 280 tokens
Vietnamese output:  avg 340 tokens  (+21%)

The RAG layer

Most of our customer workloads are RAG-based — document search, policy Q&A, report generation from structured data. The retrieval layer is where bilingual complexity really bites.

We use BGE-M3 as our embedding model — it handles Vietnamese and English in the same vector space with competitive retrieval quality on both. The alternative (separate embeddings per language with routing) is more complex to maintain and doesn't meaningfully outperform BGE-M3 on our evaluation sets.

Cross-lingual retrieval (Vietnamese query over English documents, or vice versa) is where things get tricky. Our customers often have mixed-language document corpora — English technical documentation, Vietnamese business context. Retrieval quality drops noticeably in cross-lingual scenarios. Our current mitigation is query translation before retrieval, which adds latency but improves quality significantly.

What I'd do differently

Three things:

  1. Instrument Vietnamese quality separately from day one. We spent three months with a blended quality metric before realizing Vietnamese was dragging it down in specific task types. Separate evaluation tracks from the start.
  2. Build the prompt library bilingually. We had English prompt templates that we translated to Vietnamese as needed. Vietnamese prompts need to be designed natively, not translated — the linguistic structure is different enough that direct translation produces unnatural outputs.
  3. Budget for customer calibration time. Each customer has different Vietnamese quality requirements based on their use case and audience. What's acceptable for internal tooling isn't acceptable for customer-facing outputs. Build calibration time into the deployment plan.

The overall experience has been positive. Running good bilingual AI on a constrained budget is very achievable now in a way it wasn't 18 months ago. The open-weight ecosystem has improved significantly, and the tooling around efficient inference has matured. The remaining challenges are mostly measurement and calibration, not fundamental capability.