This job posting has expired
Expired on March 30, 2026
Senior Software Engineer
Job Description
Dubai Bank is building a scalable data ingestion and streaming platform that ingests change data capture (CDC) events from diverse source systems, processes them in real time and lands curated data into our analytics lake. The platform uses Confluent connectors to emit Parquet files into cloud storage and leverages Databricks Auto Loader to incrementally ingest, deduplicate and write this data into Delta Lake Bronze table.
Responsibilities
- Design and develop streaming ingestion pipelines using Apache Spark and Databricks Auto Loader
- Implement CDC and deduplication logic
- Ensure data quality and fault tolerance (checkpointing, DLQ)
- Build a config-driven framework to scale ingestion through configuration
- Implement monitoring, observability and security metrics
- Participate in DevOps processes (CI/CD, Terraform)
Qualifications
- 5–8 years of experience building data pipelines using Apache Spark, Databricks or equivalent
- Hands-on expertise with streaming and messaging systems such as Apache Kafka
- Deep understanding of relational databases and CDC (SQL Server, Oracle)
- Proficiency in Python, Scala or Java and solid knowledge of SQL
- Cloud platform expertise (Azure or AWS)