Principal Engineer - AI Platform at iHerb

Job Summary

The Principal Engineer operates at the intersection of deep technical execution and broad organizational influence. This role will resolve the most complex architectural challenges, set the AI-native engineering standard, and ensure iHerb's platform decisions contribute to long-term value rather than technical debt. Reporting to the VP of Engineering, this role collaborates closely with peer Principals, Engineering Managers, and VPs across the organization.

This role owns the end-to-end technical strategy for iHerb's AI Platform and Generative AI (GenAI) product experiences. This encompasses the user-facing AI layer (Wellness Agent, LLM-driven recommendations, RAG over catalog and reviews, generative content) and the shared AI infrastructure (RAG pipelines, evaluation frameworks, MLOps, guardrails, observability) that ensures AI features are production-grade and repeatable across the engineering organization.

Job Expectations

• Define and own the AI platform architecture, including retrieval infrastructure, model lifecycle, evaluation frameworks, guardrails, and GenAI product feature design.
• Lead the development of the shared AI platform layer, which will be consumed by all AI product features and reusable by internal business teams.
• Act as a hands-on contributor, building production AI systems, creating proofs of concept, and validating architecture through functional software.
• Set and enforce technical standards for the AI Platform team, driving architecture and model quality reviews.
• Coordinate with the Personalization team to establish clear boundaries between the GenAI product layer and existing ML personalization infrastructure.
• Integrate AI platform-specific patterns and lessons learned into iHerb's shared AI-driven SDLC golden path.
• Drive the most complex cross-cutting technical decisions across multiple teams and shared platform services.
• Establish and evolve iHerb's AI-driven SDLC golden path, including shared standards, Claude Code skills, guardrails, and automation patterns.
• Lead complex, multi-team technical initiatives by coordinating architecture reviews, aligning peer Principals and Engineering Managers (EMs), and resolving competing approaches.
• Mentor and elevate the technical standard across the engineering organization through code reviews, architecture reviews, and direct coaching of senior engineers.
• Represent engineering in cross-functional discussions with product, data science, security, and infrastructure teams.
• Document architectural decisions in the shared knowledge base to foster compounding institutional knowledge across the organization.

Note: The duties and responsibilities described above may provide only a partial description of this position. Other duties may be added as necessary.

Knowledge, Skills, and Abilities

Required:

• AI-driven SDLC: Demonstrated use of AI-assisted development tools (e.g., Claude Code, GitHub Copilot, Cursor) to ship production systems. Ability to articulate workflow changes, quality trade-offs, and guardrail strategies.
• Architecture at Scale: Experience designing and evolving large-scale distributed systems across multiple teams and years, including APIs, data pipelines, event-driven architectures, or high-traffic platforms.
• Cross-Org Technical Leadership: Proven track record of driving architectural standards, technical roadmaps, or platform initiatives that span multiple teams or organizations.
• Engineering Quality Mindset: Strong convictions regarding code quality, observability, CI/CD, test automation, and maintaining velocity without accumulating hidden debt.
• Communication and Influence: Ability to write clear architecture documents, present to technical and non-technical audiences, and build consensus without formal authority.
• Experience working in distributed teams across the US, China, and Latin America.
• Production AI Systems: Production experience designing and operating AI platform systems such as RAG pipelines, vector search, embedding infrastructure, or retrieval-augmented applications at scale.
• LLM Evaluation: Hands-on experience with LLM evaluation, including building evaluation frameworks, defining quality metrics, and iterating on model and prompt quality using data.
• MLOps: Experience with model deployment, monitoring, lifecycle management, and cost governance in a production environment.
• Agentic Systems: Experience building and operating agentic systems using frameworks like LangChain, LlamaIndex, or equivalents.
• GenAI in Production: Required; prototype or research-only experience is not sufficient.
• High degree of accuracy and attention to detail.
• Excellent organization skills and ability to multi-task.

Preferred:

• Experience with Microsoft Office Suite (Word, Excel, PowerPoint).
• Experience with Google Business Suite (Gmail, Drive, Docs, Sheets, Forms) is preferred.

Experience Requirements

• Generally requires a minimum of 10+ years of software engineering experience, with a significant portion at senior, staff, or principal IC level.

Education Requirements

• Bachelor’s Degree in Computer Science or a related field is preferred, or a combination of education and equivalent work experience.

Judgment/Reasoning Ability

• Ability to identify, troubleshoot, and resolve problems quickly with sound judgment, poise, and diplomacy. Capable of determining when to escalate issues in a timely manner.

Physical Demands

• The physical demands described are representative of those required to perform essential job functions. This includes regular talking and hearing, frequent sitting, walking, climbing stairs, using hands and fingers, bending, stooping, and reaching. May occasionally lift or move office products and supplies up to 25 pounds. Reasonable accommodations may be made for individuals with disabilities.

Work Environment

• The work environment is typically moderate in noise level. It is characterized as hectic, fast-paced with multi-level distractions, professional yet casual, and may be an office/warehouse environment. Ability to work extended hours as required.

Compensation & Benefits

• Anticipated Pay Scale: $204,000—$260,000 USD
• Benefits: Eligibility for medical, dental, vision, basic life insurance, 401(k) plan, Time Off and Paid Sick Leave, paid holidays. Specific details governed by applicable plan documents. Restrict Stock Units and annual bonuses may be awarded based on eligibility and performance criteria.