Enhance Foundational LLMs: Data Quality & Evaluation Specialist (Remote)

Role Overview

This position is part of a project dedicated to assisting foundational LLM companies in enhancing their Large Language Models. A key focus is providing high-quality proprietary data, used both for fine-tuning models and for establishing evaluation benchmarks against competitor models.

For instance, in Supervised Fine-Tuning (SFT) data generation, you may be tasked with responding to prompts containing code and questions, or writing corresponding Python code to solve these questions.

For Reinforcement Learning from Human Feedback (RLHF) data generation, you might create prompts, ask models questions, and then evaluate and compare the outputs from two LLM versions. Your feedback is crucial for fine-tuning the models. Note: This role does not involve building or fine-tuning LLMs directly.

What Your Day-to-Day Will Look Like:

• Design, develop, and maintain efficient, high-quality Python code for AI model training and optimization support.
• Conduct evaluations (Evals) to benchmark model performance and analyze results for continuous improvement.
• Utilize familiarity with Python frameworks and libraries.
• Evaluate and rank AI model responses to user queries across diverse domains, ensuring alignment with predefined criteria.
• Develop comprehensive explanations and rationales for evaluations, demonstrating excellent reasoning and technical expertise.
• Lead efforts in Supervised Fine-Tuning (SFT), including creating and maintaining high-quality, task-specific datasets.
• Collaborate with researchers and annotators to execute Reinforcement Learning with Human Feedback (RLHF) and refine reward models.
• Design innovative evaluation strategies and processes to improve model alignment with user needs and ethical guidelines.
• Create and refine optimal responses to enhance AI performance, focusing on clarity, relevance, and technical accuracy.
• Conduct thorough peer reviews of code and documentation, providing constructive feedback.
• Collaborate with cross-functional teams to improve model performance and contribute to product enhancements.
• Continuously explore and integrate new tools, techniques, and methodologies to enhance AI training processes.

Requirements:

• 3+ years of strong experience with Python programming.
• Industry experience and knowledge of code quality, formatting, and software development best practices.
• Experience with Python’s testing ecosystem (unit, integration, property-based testing).
• Knowledge of multi-threading and asynchronous programming in Python.
• Ability to work with architectural patterns and refactor code without introducing regressions.
• Strong debugging skills, including fixing memory and concurrency issues.
• Fluent in conversational and written English.