Senior Python Data Scraping Engineer (Freelance)

About Mindrift

Mindrift connects specialists with AI projects from major tech innovators, aiming to unlock the potential of Generative AI by leveraging global real-world expertise.

The Role: Senior Python Data Scraping Engineer (AI Pilot)

This is a freelance, part-time remote role for the Tendem project. As an AI Pilot, you will collaborate with AI agents, providing critical thinking, domain expertise, and quality control for specialized data scraping workflows. You'll handle data extraction tasks requiring technical precision, utilizing tools like Apify and OpenRouter, alongside your own resourceful methods.

Key Responsibilities:

• Manage end-to-end data extraction workflows for complex websites, ensuring comprehensive coverage, accuracy, and reliable delivery of structured datasets.
• Utilize internal tools (Apify, OpenRouter) and custom workflows to expedite data collection, validation, and task execution according to defined requirements.
• Ensure reliable extraction from dynamic and interactive web sources, adapting strategies for JavaScript-rendered content and evolving site structures.
• Uphold data quality standards through validation checks, cross-source consistency controls, adherence to formatting specifications, and systematic verification.
• Scale scraping operations for large datasets using efficient batching or parallelization, monitor for failures, and maintain stability against minor site changes.

Requirements:

• Minimum 5 years of relevant experience in data engineering, web scraping, automation, or software development.
• Bachelor's or Master's Degree in Engineering, Applied Mathematics, Computer Science, or a related technical field is a plus.
• Strong technical foundation with practical experience in scripting, automation, and AI-assisted workflows.
• Proficiency in solving complex problems, working confidently with LLMs, and systematically collecting, structuring, and validating data.
• Methodical, detail-oriented approach and ability to work independently.
• Strong experience in Python web scraping (e.g., BeautifulSoup, Selenium) and handling dynamic content (JS, AJAX, infinite scroll) and APIs via proxies.
• Proven ability to extract data from complex structures (hierarchies, archived pages, inconsistent HTML).
• Solid background in data cleaning, normalization, and validation, delivering structured datasets (CSV, JSON, Google Sheets).
• Demonstrated experience handling anti-bot mechanisms and dynamic site structures at scale.
• Experience with cloud infrastructure (AWS or equivalent) and containerization (Docker) in real-world workflows.
• Hands-on experience with LLM frameworks (e.g., LangChain, OpenRouter) applied to automation tasks.
• Strong attention to detail and commitment to data accuracy.
• Self-directed work ethic with the ability to troubleshoot independently.
• GitHub profile link is a plus.
• English proficiency: Upper-intermediate (B2) or above.

Project Time Expectations:

• Estimated 10–20 hours per week during active project phases.
• This is an estimate and applies only while the project is active.

Compensation:

• Up to $25 per hour equivalent, depending on level and pace of contribution.
• Compensation varies by project scope, complexity, and required expertise.