AI Data Specialist (LLM Enhancement)

Role Overview

This position is within a project focused on assisting foundational LLM companies in enhancing their Large Language Models. We achieve this by providing high-quality proprietary data for both fine-tuning models and benchmarking their performance against competitors.

Day-to-Day Activities

• Design, develop, and maintain efficient, high-quality Python code.
• Conduct evaluations (Evals) to benchmark model performance and analyze results.
• Evaluate and rank AI model responses across diverse domains, ensuring alignment with predefined criteria.
• Develop comprehensive explanations and rationales for evaluations, demonstrating strong reasoning and technical expertise.
• Lead efforts in Supervised Fine-Tuning (SFT) data generation, creating and maintaining task-specific datasets.
• Collaborate with researchers and annotators for Reinforcement Learning with Human Feedback (RLHF) and reward model refinement.
• Design innovative evaluation strategies and processes to improve model alignment with user needs and ethical guidelines.
• Create and refine optimal responses to enhance AI performance, focusing on clarity, relevance, and technical accuracy.
• Conduct thorough peer reviews of code and documentation.
• Collaborate with cross-functional teams to improve model performance and contribute to product enhancements.
• Continuously explore and integrate new tools, techniques, and methodologies to enhance AI training processes.

Requirements

• 3+ years of strong experience with Python programming language.
• Industry experience and knowledge of code quality, formatting, and best practices in software development.
• Experience with Python’s testing ecosystem (unit, integration, property-based testing).
• Knowledge of multi-threading and asynchronous programming in Python.
• Ability to work with architectural patterns and refactor code without introducing regressions.
• Strong debugging skills, including fixing memory and concurrency issues.
• Fluent in conversational and written English communication skills.