Machine Learning Engineer, Distributed vLLM

Responsibilities

• check_circle Contribute to the design, development, and testing of new features and solutions for Red Hat AI Inference
• check_circle Innovate in the inference domain by participating in upstream communities
• check_circle Develop and maintain distributed inference infrastructure leveraging Kubernetes APIs, operators, and the Gateway Inference Extension API for scalable LLM deployments.
• check_circle Develop and maintain system components in Go and/or Rust to integrate with the vLLM project and manage distributed inference workloads.
• check_circle Develop and maintain KV cache-aware routing and scoring algorithms to optimize memory utilization and request distribution in large-scale inference deployments.
• check_circle Enhance the resource utilization, fault tolerance, and stability of the inference stack.
• check_circle Develop and test various inference optimization algorithms.
• check_circle Actively participate in technical design discussions
• check_circle Contribute to a culture of continuous improvement by sharing recommendations and technical knowledge with team members
• check_circle Collaborate with other engineering and cross-functional teams to deliver on engineering deliverables
• check_circle Communicate effectively to team members to ensure proper visibility of development efforts
• check_circle Be taught, coached, and mentored by senior members of the team
• check_circle Provide timely and constructive code reviews
• check_circle Strong proficiency in Python and/or GoLang or similar language
• check_circle Experience with cloud-native Kubernetes service mesh technologies/stacks such as Istio, Cilium, Envoy (WASM filters), and CNI.
• check_circle Working understanding of Layer 7 networking, HTTP/2, gRPC, and the fundamentals of API gateways and reverse proxies.
• check_circle Knowledge of serving runtime technologies for hosting LLMs, such as vLLM, SGLang, TensorRT-LLM, etc.
• check_circle Excellent written and verbal communication skills, capable of interacting effectively with both technical and non-technical team members.
• check_circle Ability work independently in a dynamic, fast-paced environment
• check_circle Proficiency in C, C++, or Rust
• check_circle Experience with the Kubernetes ecosystem, including core concepts, custom APIs, operators, and the Gateway API inference extension for GenAI workloads.
• check_circle Working knowledge of high-performance networking protocols and technologies including UCX, RoCE, InfiniBand, and RDMA is a plus.
• check_circle Experience with GPU performance benchmarking and profiling tools like NVIDIA Nsight or distributed tracing libraries/techniques like OpenTelemetry.
• check_circle Experience in writing high performance code for GPUs and deep knowledge of GPU hardware
• check_circle Strong understanding of computer architecture, parallel processing, and distributed computing concepts
• check_circle Bachelor's degree in computer science or related field is an advantage, though we prioritize hands-on experience
• check_circle Active engagement in the ML research community (publications, conference participation, or open source contributions) is a significant advantage

Benefits

• check_circle Comprehensive medical, dental, and vision coverage
• check_circle Flexible Spending Account - healthcare and dependent care
• check_circle Health Savings Account - high deductible medical plan
• check_circle Retirement 401(k) with employer match
• check_circle Paid time off and holidays
• check_circle Paid parental leave plans for all new parents
• check_circle Leave benefits including disability, paid family medical leave, and paid military leave
• check_circle Additional benefits including employee stock purchase plan, family planning reimbursement, tuition reimbursement, transportation expense account, employee assistance program, and more!

About the company