- Company Name
- Venturi
- Job Title
- Databricks Engineer
- Job Description
-
Job title: Databricks Engineer
Role Summary:
Design, develop, and maintain robust, scalable data pipelines on Azure Databricks to support enterprise‑level AI and analytics programs. Deliver high‑quality Delta Lake data models, drive performance optimization, and implement best engineering practices in CI/CD, governance, and version control.
Expectations:
- Execute end‑to‑end data engineering solutions at scale.
- Collaborate with analytics, AI/ML, and stakeholder teams to translate business requirements into technical designs.
- Operate within fast‑paced consulting environments, delivering technical excellence under tight timelines.
Key Responsibilities:
• Build, optimize, and maintain ETL/ELT pipelines using PySpark and Spark SQL on Azure Databricks.
• Design and enforce Delta Lake architectures, ensuring data quality and production readiness.
• Integrate data flows into AI/ML pipelines, facilitating seamless data access for modeling.
• Implement CI/CD pipelines (Azure DevOps or equivalent), version control, automated testing, and deployment workflows.
• Optimize cluster configuration, job execution, and resource usage to meet performance SLAs.
• Produce clear technical documentation, architecture diagrams, and run‑books.
• Lead or participate in architectural reviews, providing constructive feedback and viable alternatives.
• Troubleshoot and resolve data quality, transformation, and performance issues in production.
• Maintain governance and environment separation best practices across development, staging, and production.
Required Skills:
- Proven hands‑on experience with Azure Databricks and the Azure data ecosystem (Azure Data Lake, Azure Synapse, Azure Data Factory).
- Advanced proficiency in PySpark, Spark SQL, and distributed data processing concepts.
- Expertise in Delta Lake design, data modeling, and schema enforcement.
- Strong understanding of CI/CD (Azure DevOps preferred), version control, automated testing, and deployment strategies.
- Knowledge of data governance, security, and environment segregation in enterprise settings.
- Ability to analyze, optimize, and scale Spark workloads and cluster resources.
- Effective communication and collaboration skills for cross‑functional teams and complex stakeholder groups.
Required Education & Certifications:
- Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a closely related field.
- Azure Data Engineer Associate (DP-203) or Azure Data Scientist Associate (DP-100) certification strongly preferred.
---