- Company Name
- Yochana
- Job Title
- Data Engineer
- Job Description
-
Job title: Data Engineer
Role Summary: Design, build, and maintain scalable data pipelines and warehousing solutions on Azure and Snowflake, ensuring high performance, quality, and governance across data ingestion, transformation, and storage processes.
Expectations: • Deliver end‑to‑end data pipelines that meet business performance and SLA requirements
• Continuously optimize and troubleshoot data workflows for efficiency and cost effectiveness
• Uphold data security, lineage, and compliance standards throughout the data lifecycle
• Provide mentorship and technical guidance to junior team members
Key Responsibilities:
- Build and optimize ETL/ELT pipelines with Azure Data Factory and Azure Databricks (PySpark, SQL)
- Ingest diverse data sources into Azure Data Lake Storage (ADLS) and Snowflake
- Develop, tune, and schedule Spark jobs; leverage Delta Lake, Autoloader, and advanced optimization techniques (Z‑ordering, partitioning)
- Design and maintain Snowflake schemas, queries, and performance tuning (Snowpipe, clustering)
- Implement data quality and governance frameworks, including security policies, lineage, and validation rules
- Develop automation and processing logic in Python, PySpark, Scala, and complex SQL
- Collaborate with data architects, analysts, and stakeholders to define data requirements and solutions
- Mentor junior engineers and promote best practices in coding, testing, and documentation
- Manage CI/CD pipelines using Azure DevOps; maintain Git workflows, automated testing, and deployment pipelines
- Troubleshoot and resolve pipeline and performance issues in a production environment
Required Skills:
- Expertise in Azure services: Data Factory, Databricks, ADLS, Key Vault, and related security features
- Strong Snowflake experience: data modeling, query optimization, Snowpipe ingestion, and cost management
- Proficiency in Python, PySpark, Scala, advanced SQL, and data serialization formats (JSON, Parquet, CSV)
- Deep knowledge of Delta Lake, Autoloader, Z‑ordering, partition strategies, and performance tuning
- Familiarity with CI/CD pipelines, Git workflows, and DevOps practices for data engineering
- Strong understanding of data security, privacy, and compliance (GDPR, SOC, etc.)
Required Education & Certifications:
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field
- Azure Data Engineer Associate certification (DP-203) and/or Azure Databricks Developer certification preferred.