- Company Name
- Sharp Decisions
- Job Title
- Senior Databricks Engineer
- Job Description
-
Job title: Senior Databricks Engineer
Role Summary: Design, develop, test, and support Azure‑based data ingestion and transformation pipelines for cybersecurity data, leveraging Databricks, Azure Data Factory, and related Azure services. Lead data feed development, ensure data quality and governance, and collaborate with cross‑functional teams to align data requirements with business intelligence dashboards.
Expactations:
- Own the end‑to‑end lifecycle of cloud data solutions for security analytics.
- Mentor junior engineers and drive technical direction for data pipelines.
- Deliver high‑quality, metadata‑rich data assets that satisfy regulatory and governance standards.
Key Responsibilities:
- Build and maintain ingestion, ETL/ELT, and data quality pipelines in Azure Databricks.
- Capture and document metadata for Key Data Elements, define DQ rules, and enforce accuracy.
- Develop logical and physical data models with the architecture team, defining schemas and data types.
- Partner with data owners to define technical requirements and create dashboards.
- Configure Unity Catalog, manage data lineage, and enforce security protocols.
- Coordinate with application teams on integration and scheduling (IBM TWS, Azure DevOps).
- Resolve data quality issues, implement process improvements, and maintain production job monitoring.
- Collaborate on CI/CD pipelines (Git, Jenkins, Azure DevOps) and DevOps practices.
Required Skills:
- 10+ years in IT development, data governance, or architecture, preferably in regulated environments.
- Advanced proficiency with Azure Databricks, Azure Data Factory, Azure Functions, Azure Data Lake, Azure Event Grid, Azure Log Analytics, Azure Monitor, and Unity Catalog.
- Strong command of SQL (SQL Server, Oracle, T‑SQL), PL/SQL, Python, Java, and REST API/microservice architecture.
- Experience with CI/CD and DevOps tooling (Git, Jenkins, Azure DevOps).
- Knowledge of enterprise security data domains (phishing, identity management, privileged access, threat detection, etc.).
- Familiarity with SIEM (Microsoft Sentinel) and security frameworks (NIST, CIS, CRI, GDPR, CCPA).
- Experience with data management tools (Collibra DQIM/DQE, IBM Infosphere DA) and ETL/ELT concepts.
- Understanding of IBM Tivoli Workload Scheduler and Power BI (optional).
Required Education & Certifications:
- Bachelor’s degree (or higher) in Computer Science, Information Systems, Data Engineering, or related field.
- Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Data Engineer are preferred.
Jersey city, United states
On site
Senior
10-12-2025