AI Engineer

We are looking for a talented AI Engineer who has deep expertise in running AI and ML models within Azure Databricks. The ideal candidate will be responsible for managing large-scale data processing, implementing scalable solutions, and optimizing data pipelines within the Madeline architecture framework, utilizing Bronze, Silver, and Gold catalogues. The role requires hands-on experience with PySpark, parallel processing libraries, and the ability to build and enhance AI algorithms and visualizations in Databricks.

Key Responsibilities:
Data Management: Handle and process raw data using the Madeline architecture. Create robust data pipelines for Bronze, Silver, and Gold catalogues.
AI/ML Model Execution: Run AI and ML models on Azure Databricks and optimize them for efficiency, scalability, and performance. Handle exception cases effectively during data processing and model execution.
Notebook Development: Write efficient PySpark scripts and integrate parallel processing libraries that scale seamlessly within Azure Databricks.
Data Visualization: Develop native visualizations within Databricks to present model outputs and insights. Ensure the visualizations are interactive, scalable, and easy to interpret for business users.
Algorithm Optimization: Iteratively improve AI and ML algorithms through data validation, accuracy enhancement, and performance tuning. Collaborate with the data science team to refine model outputs.
Testing and CI/CD Integration: Write unit tests to validate data pipelines and models. Ensure integration with CI/CD pipelines (preferably using GitHub Actions), enabling smooth deployment and workflow management.
Efficient Deployment: Implement scalable solutions and manage efficient deployment of workflows within the Azure Databricks environment. Optimize resource usage and performance for AI model deployment.