SPARK Engineer
Added 31st March 2025
We are searching for a Senior Data Engineer (Spark/Kafka) who will be instrumental in building a fully configurable, Kubernetes-hosted Spark data ingestion application for our client’s AI Operations and Data Engineering teams. This solution will process data from multiple Kafka streams, apply validation and transformation logic, and deliver structured data to a Delta Lake environment. It will also support NLP workflows for advanced AI use cases.
Responsibilities
- Attend daily stand-ups and share progress/blockers
- Demonstrate completed work during sprint ceremonies
- Collaborate with Research team on data validation logic
- Keep documentation and runbooks up to date
- Meet weekly with Head of Data Engineering to review architecture/design
Essential Skills
- Experience working with Delta Tables and Spark
- Familiarity with Jinja templating and JSON configuration
- Skilled in orchestration tools like Airflow
- Solid understanding of NoSQL (e.g. MongoDB, Redis)
- Strong grasp of modular coding and abstraction principles
Location
Remote
Salary
Please contact us
Industry
Retail
Contract Type
Contract