Data Acquisition & Infrastructure Engineer

Role Overview

As a Data Acquisition & Infrastructure Engineer at MAI, you will be instrumental in establishing the company's data capabilities. Your core mission will involve developing and operating automated data collection pipelines to aggregate publicly available information from the art market ecosystem. You will also be responsible for the design and maintenance of the robust PostgreSQL and AWS database infrastructure that stores and serves this critical data.

This is a high-ownership position where you will take a partially built infrastructure to production scale. You will collaborate closely with AI engineers, a computer vision engineer, a product manager, full-stack developers, and the Head of Art Research, reporting directly to the Head of AI Engineering.

Key Responsibilities

Data Pipeline & Collection

• Architect and maintain automated pipelines for collecting, normalizing, and ingesting publicly available art market data from web-based sources.
• Build reliable and maintainable collection systems using Python (Scraping, BeautifulSoup, Playwright, or equivalent), focusing on resilience, scheduling, and data freshness.
• Manage pipeline orchestration and scheduling with tools like Apache Airflow, AWS EventBridge, or cron.
• Address practical challenges of large-scale public data collection, including access patterns, rate constraints, and source reliability.
• Clean, transform, and standardize messy, inconsistent real-world datasets for downstream consumption.

Database Engineering

• Design, build, and maintain relational database schemas in PostgreSQL (hosted on Amazon RDS) to support complex, multi-entity art market data (artists, works, transactions, provenance, valuation history).
• Develop and optimize queries, indexes, and data models for performance at scale.
• Establish and enforce data quality standards, validation rules, and integrity constraints.
• Collaborate with AI and computer vision teams to ensure the data layer supports model training and inference needs.

Infrastructure & Operations

• Deploy and manage pipeline workloads on AWS (Lambda, EC2, S3, RDS).
• Monitor pipeline health, data freshness, and system reliability, proactively addressing failures.
• Contribute to infrastructure-as-code practices as the team scales.

Core Requirements

• 3–5 years of professional experience in data engineering or a closely related field.
• Proven experience building and maintaining automated data collection pipelines from public web sources using Python (Scrapy, BeautifulSoup, Playwright, or Selenium).
• Strong data cleaning and normalization skills, with demonstrated ability to handle heterogeneous, inconsistent real-world datasets.
• Solid PostgreSQL experience: schema design, query optimization, and database maintenance.
• Hands-on AWS experience: Lambda, EC2, S3, RDS.
• Experience scheduling and orchestrating data pipelines (Apache Airflow, AWS EventBridge, or equivalent).
• Experience navigating the constraints of large-scale public data collection, including reliability, access patterns, and data freshness challenges.

Nice to Have

• Knowledge of data quality frameworks and validation pipeline design.
• Experience with containerization (Docker) and infrastructure-as-code (Terraform, AWS CDK).
• Familiarity with ETL/ELT tooling (dbt, AWS Glue, or equivalent).
• Exposure to art market platforms (Christie's, Sotheby's, Artsy, Artnet) or understanding of auction/gallery data structure.
• Background or genuine interest in the art world, collectibles, or alternative asset markets.
• Experience in a startup or early-stage environment requiring ownership and adaptability.