MLOps Engineer, Model Serving on Kubernetes

• Deploy and tune serving engines such as vLLM, SGLang, TensorRT-LLM or Triton on GPU nodes.
• Pin each model's version, and log every change to a model, its settings or its route.
• Autoscale on request load and queue depth, and fit models to GPUs by memory and batch size.
• Route requests across models and replicas, with rate limits and quotas per team.
• Run retrieval services next to the models: embedding jobs, search indexes and the volumes they need.
• Run MCP servers for governed tool access, with per-tool permissions and audit logs.
• Hold each release until it passes the team's evaluation tests, and roll back any release that regresses.
• Measure latency, throughput, GPU use and cost per token for each model, and report them to the client.

• 3+ years in MLOps, ML infrastructure or backend engineering, with models in production.
• Hands-on LLM serving with vLLM, SGLang, TensorRT-LLM, Triton or similar.
• Kubernetes deployments, autoscaling, GPU requests and node selection.
• Python, plus one systems language such as Go or Rust.
• Working knowledge of how GPU memory, batching, KV cache and quantization affect latency.
• Metrics and tracing with Prometheus, Grafana or OpenTelemetry.

• KServe, Ray Serve or NVIDIA Dynamo.
• KV cache-aware routing, or disaggregated prefill and decode.
• Vector databases and retrieval pipelines.
• A model registry such as MLflow.
• Multi-GPU and multi-node serving.

$124,000 to $201,000 a year

Full-time W-2 employment with health insurance, a retirement plan and paid leave.

No application deadline. We accept applications for this role on an ongoing basis.

Remote, United States