SPARK Engineer

Added 31st March 2025

We are searching for a Senior Data Engineer (Spark/Kafka) who will be instrumental in building a fully configurable, Kubernetes-hosted Spark data ingestion application for our client’s AI Operations and Data Engineering teams. This solution will process data from multiple Kafka streams, apply validation and transformation logic, and deliver structured data to a Delta Lake environment. It will also support NLP workflows for advanced AI use cases.

 

Responsibilities

- Attend daily stand-ups and share progress/blockers

- Demonstrate completed work during sprint ceremonies

- Collaborate with Research team on data validation logic

- Keep documentation and runbooks up to date

- Meet weekly with Head of Data Engineering to review architecture/design

 

Essential Skills

- Experience working with Delta Tables and Spark

- Familiarity with Jinja templating and JSON configuration

- Skilled in orchestration tools like Airflow

- Solid understanding of NoSQL (e.g. MongoDB, Redis)

- Strong grasp of modular coding and abstraction principles

 

Apply now

Location

Remote

Salary

Please contact us

Industry

Retail

Contract Type

Contract

More roles that might suit you

Job title Location Type Salary

Remote

Contract

Please contact us

West Midlands

Permanent

Up to £80,000 pa

London Remote

Contract

Please contact us

Can’t find the role for you?

Upload CV