Servidor Xincuan | Fornecedor de hardware de servidor empresarial
Dia 0: 把需求翻译成 GPU 数量
每一次 on-prem 推理部署都始于同一个问题：跑什么模型、服务多少并发、延迟承诺多少？用这三问做规模测算：
| Input | Example | Sizing impact |
|---|---|---|
| Model size | 70B params, FP8 quantized | ~70 GB weights -> needs 2 x 48GB L40S sharded or 1 x with context |
| Concurrency / tokens | 50 usuários simultâneos, 5k tokens/s | Determines GPU count, not per-GPU spec |
| Latency SLA | First token < 500 ms | Drives batching policy and GPU memory headroom |
O NVIDIA L40S (48GB, 1,466 TFLOPS FP8) is the 2026 workhorse for this envelope: a 2U node with two L40S cards serves a 70B-class model in production, and a 4-8 node cluster covers most enterprise RAG and copilot workloads. Day 0’s deliverable is a GPU count and a latency budget – nothing else.
Dia 1: 节点与网络——L40S 的标准栖身之所
Build the inference node on proven 2U hardware with 350 W GPU power delivery:
- Node: a dual-socket 2U chassis with 2 x L40S (700 W total GPU TDP) – o PowerEdge R760 (2 x 350 W double-width slots) ou xFusion 2288H V7 (4 x double-width for denser builds).
- Fabric: one E810-CQDA2 100GbE per node – tensor-parallel shards exchange activations over the fabric; 100GbE keeps multi-node latency predictable.
- Armazenar: NVMe for model weights and KV-cache checkpointing; a few hundred GB per model is enough at this scale.
- Gerenciamento: iDRAC/iBMC telemetry for GPU thermals – a 700 W GPU load changes airflow requirements vs CPU-only nodes.
For training-class density, the 6U XE9680 (8-GPU) or 4U FusionServer G5500 V7 (10-GPU) scale the same stack upward – the L40S cluster starts at 2U and grows by adding nodes.
Dia 2: 推理栈——让硬件跑满
The hardware is only half the deployment. The software stack decides real throughput:
- Runtime: vLLM or TensorRT-LLM with FP8 quantization – the L40S’s 1,466 TFLOPS FP8 is what makes small-cluster serving viable.
- Batching: continuous batching keeps GPU utilization high under mixed traffic; target 70-85% utilization at peak.
- Sharding: tensor parallelism across 2-4 GPUs per model replica; keep shard count within the node where possible.
- Multi-model: a shared GPU pool (Kubernetes with GPU scheduling) serves several small models on one cluster instead of one-model-per-node.
Measure tokens-per-second per GPU and first-token latency from Day 2 – those two numbers decide whether you add GPUs or tune batching.
Dia 3: 上线、监控与扩缩容
Go-live is the start of operations, not the end:
- Monitor: GPU utilization, KV-cache memory, token throughput, p95 first-token latency – per node and per model.
- Alert: latch onto thermal (iDRAC/iBMC) and memory pressure before users notice degradation.
- Scale: add L40S nodes to the pool as load grows; rebalance replicas via the scheduler, no downtime.
- Review: monthly – model refresh (new quantizations), traffic shape changes, and whether FP8 or smaller models cut cost further.
The Cluster Template
| Tamanho | Nodes | L40S GPUs | Model class |
|---|---|---|---|
| Pilot | 1 x 2U | 2 | 7-13B, small RAG |
| Production small | 2-3 x 2U | 4-6 | 70B-class, copilot |
| Production large | 5-8 x 2U | 10-16 | Multi-model, high concurrency |
Every node is identical (same chassis, same NIC, same stack) – the cluster scales by template replication, which is exactly how on-prem fleets stay operationally sane.
FAQ
- Why L40S over H100? For inference, FP8 density and 350W power make L40S far cheaper per token at enterprise scale.
- Two GPUs per node enough? For 7-70B models in production, yes – scale nodes out, not up.
- Need RDMA? Para 2-4 GPU tensor parallelism, 100GbE TCP is sufficient; RDMA (iWARP/RoCE) helps above 8 GPU.
- Storage requirements? NVMe for weights and KV-cache checkpoints; a few hundred GB per model.
- Where to start? Size by model+concurrency (Dia 0), then order one pilot node and measure.
Ready to build your inference cluster? We deliver L40S-ready nodes (R760 / 2288H V7 with dual-L40S and 100GbE) as a complete stack – factory-direct pricing, 3-ano de garantia, global shipping and free cluster design consultation from our authorized partner team. Contact us for a Day-0 sizing call.
Deployment note: the same Day 0-3 sequence applies whether you standardize on R760, 2288H V7 or a G5500-scale node – the sizing math and the software stack are identical.