AI Product Engineer

Company: ClickHouse

Location: United States (Remote options available, specific locations may have premium market ranges)

About ClickHouse:
Recognized on the Forbes Cloud 100 list, ClickHouse is a fast-growing private cloud company leading the market in real-time analytics, data warehousing, observability, and AI workloads. With over 3,000 customers and significant year-over-year ARR growth, ClickHouse is transforming how companies use data.

About the Role:
Join us in building the AI layer for Observability! ClickStack is the open-source observability platform we're building at ClickHouse. We're hiring an AI Product Engineer to build agentic capabilities on top of a petabyte-scale observability platform, focusing on developer experience. If you have experience building agents, designing skills, and wiring up MCP servers, we want to talk to you.

What You'll Do:

• Build Investigating Agents: Develop agents that investigate incidents, surface anomalies, answer critical operational questions, and use ClickStack as their foundation.
• Write Reusable Skills: Create a library of reusable skills that encapsulate team debugging strategies, root cause analysis, ClickHouse query writing, and incident response playbooks for agents.
• Own the Agent Stack End-to-End: Manage context engineering, tool design, evaluations, tracing, and cost for agent performance in production.
• Enhance ClickStack for AI: Build MCP servers, SDKs, and integrations that allow customer agents to read telemetry, take action, and remain observable.
• Collaborate in the Open: Work with OSS contributors and customers, debug issues collaboratively, and feed learnings back into the product.
• Tackle Complex Challenges: Address latency, cost, context window limits, evaluation coverage, and hallucinations in real telemetry data.

Who You Are:

• Experienced in building agents with strong opinions on context engineering, tool design, and framework limitations.
• Production-minded, focusing on p99 latency, cost per task, and system resilience.
• Fast-moving, shipping frequently, and learning from failures.
• Passionate about developer tools and excellent developer experience (DX).
• Comfortable with ambiguity and taking ownership.

What You Bring:

• 5+ years of software engineering experience, including 1-2 years on LLM-powered systems or agents in production.
• Strong backend skills in TypeScript/Node.js and/or Python.
• Hands-on experience building agents (multi-step tool use, planning, memory, error recovery).
• Experience designing skills (e.g., Markdown-based workflow encodings, Anthropic-style) and understanding when to use skills vs. tools.
• Experience with MCP: building servers, designing tools, and considering auth, scoping, and observability for agentic systems.
• Strong evaluation practice (golden sets, LLM-as-judge, regression detection).
• SQL proficiency, including writing ClickHouse queries directly.
• Comfort with Docker and Kubernetes.
• Active participation in open source and the developer community.

Bonus Points:

• Experience building or operating production agents in observability, incident response, or SRE.
• Strong opinions on agent observability (tracing, cost attribution, eval pipelines, OpenTelemetry) and ideas for improvement.
• Experience with prompt caching, context compaction, or similar techniques for handling production telemetry volumes.
• Experience with columnar databases and event ingestion pipelines.
• Contributions to open-source AI/agent projects.
• Familiarity with Go, Rust, or other systems languages.

Compensation (US):
Salary ranges are provided, with potential premium market ranges in specific locations like the San Francisco Bay Area and New York City Metro Area. Actual compensation depends on factors like qualifications, experience, location, and performance.

Perks:

• Flexible, remote-friendly work environment.
• Employer contributions towards healthcare.
• Stock options for all new team members.
• Flexible time off (US) / generous entitlement (other countries).
• $500 Home office setup stipend for remote employees.
• Opportunities for in-person connection through global gatherings and offsites.