- Company Name
- The Trade Desk
- Job Title
- Senior Software Engineer - Data Transparency
- Job Description
-
**Job Title:** Senior Software Engineer – Data Transparency
**Role Summary:**
Full‑stack owner of data‑platform infrastructure, responsible for designing, building, and delivering scalable, high‑quality data pipelines and foundational datasets that serve internal tools and external partners. Works across data engineering, analytics, solution architecture, and business intelligence to enable trustworthy, compliant data usage.
**Expectations:**
- Lead engineering efforts, mentor a diverse team, and drive systematic improvements at scale.
- Own the data lifecycle from acquisition to production exposure, ensuring quality, lineage, and compliance for all datasets.
- Foster a culture of continuous learning and innovation, partnering with Legal, Security, and Governance to uphold data stewardship standards.
**Key Responsibilities:**
- Architect and implement large‑scale data pipelines (Spark, Airflow) for high‑velocity, petabyte‑scale workloads.
- Design core, reusable datasets that eliminate redundant downstream transformations.
- Establish and maintain metadata, data lineage, observability, and quality checks throughout the ecosystem.
- Build privacy‑centric data exports and processing layers for external stakeholders.
- Collaborate with cross‑functional teams to translate business requirements into reliable, maintainable data solutions.
- Monitor, troubleshoot, and optimize system performance and resource utilization.
**Required Skills:**
- Strong communication and stakeholder‑management abilities.
- Expert in Spark for data processing and performance tuning.
- Experience constructing complex, production‑grade pipelines in Airflow (plus).
- Proficient in data architecture, quality, observability, and lineage frameworks.
- Knowledge of data compliance, privacy regulations, and security best practices.
- Proven track record of leading engineers across varied experience levels.
**Required Education & Certifications:**
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical discipline.
- No mandatory certifications, but familiarity with Big Data or Cloud data services is a plus.