Senior Software Engineer, ML Infrastructure - SambaNova Systems

SambaNova Systems is at the forefront of the AI revolution with SambaNova Suite™, the first full-stack, generative AI platform from chip to model, optimized for enterprise and government organizations. Our platform, powered by the intelligent SN40L chip, is delivered on-premises or in the cloud, offering easily fine-tunable open-source models that empower customers to leverage generative AI securely and retain model ownership.

Overview:

The Senior Software Engineer, ML Infrastructure will be instrumental in designing, building, and operating the production-grade inference infrastructure for SambaNova's serving stack on our Reconfigurable Dataflow Unit (RDU) architecture. As an inference-first company, this role is central to transforming state-of-the-art inference techniques into reliable, high-throughput, low-latency services accessible via SambaStack and SambaCloud. You will own end-to-end systems including request scheduling, advanced decoding algorithms, caching layers, API surfaces, and accuracy infrastructure, partnering closely with ML, compiler, runtime, and product teams.

Qualifications:

• Bachelor’s degree in Computer Science, Electrical Engineering, or a related field.
• 5+ years of industry experience building and operating large-scale distributed systems, preferably in ML serving.
• Strong software engineering fundamentals: algorithms, data structures, concurrency, and systems design.
• Proven experience designing and maintaining production services with strict latency, throughput, and availability requirements.
• Working knowledge of modern LLM inference techniques and familiarity with open-source serving stacks (e.g., vLLM, TensorRT-LLM, SGLang).
• Proficiency in Python.
• Experience collaborating across teams to deliver complex, system-level engineering solutions.

Key Responsibilities:

• Design and productionize advanced inference techniques on RDU to optimize performance and cost, including speculative decoding, constrained decoding, function/tool calling, prompt caching, and long-context inference.
• Own SambaNova’s integration with vLLM and adjacent serving frameworks, adapting them to the RDU architecture.
• Own the public inference API surface exposed through SambaStack and SambaCloud.
• Build and maintain the accuracy verification and regression infrastructure for inference features.
• Partner with ML, compiler, runtime, and product teams to transition inference features from prototype to production.
• Contribute to technical design discussions, code reviews, and architectural decisions as a senior individual contributor.

Base Salary Range:

$200,000—$275,000 USD

EEO Policy:

SambaNova Systems is an Equal Opportunity/Affirmative Action Employer. We consider all qualified applicants without regard to age, color, disability, gender identity, genetic information, marital status, military or veteran status, national origin/ancestry, race, religion, creed, sex (including pregnancy, childbirth, breastfeeding), sexual orientation, and any other applicable status protected by law.

Benefits Summary (US-Based, Full-Time):

• Competitive total rewards package including base salary, equity, and benefits.
• 95% premium coverage for employee medical insurance; 77% for dependents.
• Health Savings Account (HSA) with employer contribution.
• Dental, Vision, Short/Long-term Disability, Basic Life, Voluntary Life, and AD&D insurance plans.
• Flexible Spending Account (FSA) options.
• Wellness benefits including Headspace subscription, Gympass+ membership, One Medical membership, and Employee Assistance Program.