Skills

Communication Python Java SQL NoSQL Data Engineering GitLab CI/CD Analytics and Reporting Programming Databases Azure Agile Analytics Spark CI/CD Pipelines PySpark Microsoft Azure

Job Specifications

Senior Data Engineer (Developer) – Pyspark

We are supporting a leading global financial markets infrastructure and data provider as they modernise and scale their core data engineering capabilities. This role sits at the centre of their transformation programme, delivering high-quality data pipelines, models, and platforms that underpin critical services across the business.

Key Responsibilities

Design, build, and optimise scalable data pipelines for both batch and streaming workloads:

Develop dataflows and semantic models aligned to analytics and reporting needs
Implement complex transformations and performance-focused data processing logic
Apply data validation, cleansing, and profiling techniques to ensure accuracy and consistency
Implement access controls, data masking, and compliance-aligned security protocols
Tune workloads and optimise performance across Spark, Fabric, and Azure components
Translate business requirements into technical solutions through close collaboration with analysts and stakeholders
Maintain clear documentation and contribute to internal knowledge repositories

Essential Skills

Strong experience developing within Microsoft Azure and Microsoft Fabric:

Proficiency in Spark programming including DataFrames, RDDs, and Spark SQL
Python and PySpark development experience, including notebook-based workflows
Hands-on experience with Spark streaming and batch processing
Delta table optimisation and Fabric Spark job development
Solid Java programming and OOP understanding
Experience working with relational and NoSQL databases
Familiarity with GitLab, unit testing, and CI/CD pipelines
Strong troubleshooting ability and experience working in Agile environments
Excellent communication skills with stakeholder-facing experience
Practical experience building ETL workflows, lakehouse architectures, dataflows, and semantic models
Exposure to time-series data, financial market feeds, transactional records, and risk-related datasets

About the Company

Gazelle Global is an international recruitment consultancy established in 2011 through a unique partnership of IT implementation know-how and head hunting expertise with a vision to create a recruitment agency that delivers a best in class service whilst being a thoughtful and conscientious employer. Our London HQ in collaboration with our offices in The Netherlands, Germany and Poland, serve clients and candidates across the UK and Europe. We pride ourselves on speed, transparency and user experience. Agility, at scale. ... Know more