• Find a role /
• SPARK Engineer

SPARK Engineer

Added 1st April 2025

We are searching for a Senior Data Engineer (Spark/Kafka) who will be instrumental in building a fully configurable, Kubernetes-hosted Spark data ingestion application for our client’s AI Operations and Data Engineering teams. This solution will process data from multiple Kafka streams, apply validation and transformation logic, and deliver structured data to a Delta Lake environment. It will also support NLP workflows for advanced AI use cases.

Responsibilities

- Attend daily stand-ups and share progress/blockers

- Demonstrate completed work during sprint ceremonies

- Collaborate with Research team on data validation logic

- Keep documentation and runbooks up to date

- Meet weekly with Head of Data Engineering to review architecture/design

Essential Skills

- Experience working with Delta Tables and Spark

- Familiarity with Jinja templating and JSON configuration

- Skilled in orchestration tools like Airflow

- Solid understanding of NoSQL (e.g. MongoDB, Redis)

- Strong grasp of modular coding and abstraction principles