Lead Data Engineer (PySpark, statistics, MLflow)
Posted Updated
Description
We are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models.
Responsibilities
- Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations
- Decompose and refactor large legacy notebooks without relying on existing documentation
- Apply statistical techniques to detect outliers and events in sensor time series data
- Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning
- Implement model explainability techniques such as SHAP for existing models
- Develop unit and regression tests for PySpark
Skills
Sign in to read and join the discussion.