Data Engineer with MLOps

Data_and_MLOps_Engineer - Contract - onsite

• Experience within the Azure ecosystem, including Azure AI Search, Azure Storage Blob, Azure Postgres, with expertise in leveraging these tools for data processing, storage, and analytics tasks.
• Proficiency in data preprocessing and cleaning large datasets efficiently using Azure Tools, Python, and other data manipulation tools.
• Strong background in Data Science/MLOps, with hands-on experience in DevOps, CI/CD, Azure Cloud computing, and model monitoring.
• Expertise in healthcare data standards, such as HIPAA and FHIR, with a deep understanding of sensitive data handling and data masking techniques to protect PII and PHI.
• In-depth knowledge of search algorithms, indexing techniques, and retrieval models for effective information retrieval tasks.
• Experience with chunking techniques and working with vectors and vector databases like Pinecone.
• Ability to design, develop, and maintain scalable data pipelines for processing and transforming large volumes of structured and unstructured data, ensuring performance and scalability.
• Implement best practices for data storage, retrieval, and access control to maintain data integrity, security, and compliance with regulatory requirements.
• Implement efficient data processing workflows to support the training and evaluation of solutions using large language models (LLMs), ensuring that models are reliable, scalable, and performant.
• Proactively identify and resolve data quality issues, pipeline failures, or resource contention to minimize disruption to systems.
• Experience with large language model frameworks, such as Langchain, and the ability to integrate them into data pipelines for natural language processing tasks.
• Familiarity with Snowflake for data management and analytics, with the ability to work within the Snowflake ecosystem to support data processes.
• Knowledge of cloud computing principles and hands-on experience with deploying, scaling, and monitoring AI solutions on platforms like Azure, AWS, and Snowflake.
• Ability to communicate complex technical concepts effectively to both technical and non-technical stakeholders, and collaborate with cross-functional teams.
• Analytical mindset with attention to detail, coupled with the ability to solve complex problems efficiently and effectively.
• Knowledge of cloud cost management principles and best practices to optimize cloud resource usage and minimize costs.
• Experience with ML model deployment, including testing, validation, and integration of machine learning models into production systems.
• Knowledge of model versioning and management tools, such as MLflow, DVC, or Azure Machine Learning, for tracking experiments, versions, and deployments.
• Model monitoring and performance optimization, including tracking model drift and addressing performance issues to ensure models remain accurate and reliable.
• Automation of ML workflows through CI/CD pipelines, ensuring efficient delivery, testing, and deployment.
• Monitoring and logging of AI/ML systems post-deployment to ensure reliability and performance.
• Collaboration with data scientists and engineering teams to deliver integrated retraining, fine-tuning, and updating.
• Familiarity with containerization technologies, such as Docker and Kubernetes, for scaling machine learning models in production environments.
• Ability to implement model governance practices to ensure compliance and provide transparency into models in production environments.
• Understanding of model explainability and interpretability techniques to provide transparent insights into model behavior.

Must Have:

• Minimum of 10 years experience as a data engineer.
• Hands-on experience using Azure Cloud ecosystem.
• Deep knowledge in AI/ML and MLOps.
• Hands-on experience with DevOps, CI/CD, model monitoring.
• Hands-on experience in healthcare domain.
• Experience with containerization technologies.
• Hands-on experience working with unstructured data.

Good to Have:

• Experience with Snowflake, MLflow, DVC, experiment intelligence, Snowflake, function app, Azure AI Search.