Senior Consultant

Job Description
Note: Candidate Must have experience in
Python, Kafka Stream, Pyspark, and Azure Databricks
.

Role Overview
We are looking for a highly skilled with expertise in
Kafka, Python, and Azure Databricks
(preferred) to drive our healthcare data engineering projects. The ideal candidate will have deep experience in
real-time data streaming, cloud-based data platforms, and large-scale data processing
. This role requires strong technical leadership, problem-solving abilities, and the ability to collaborate with cross-functional teams.

Key Responsibilities

• Lead the design, development, and implementation of real-time data pipelines using Kafka, Python, and Azure Databricks.
• Architect scalable data streaming and processing solutions to support healthcare data workflows.
• Develop, optimize, and maintain ETL/ELT pipelines for structured and unstructured healthcare data.
• Ensure data integrity, security, and compliance with healthcare regulations (HIPAA, HITRUST, etc.).
• Collaborate with data engineers, analysts, and business stakeholders to understand requirements and translate them into technical solutions.
• Troubleshoot and optimize Kafka streaming applications, Python scripts, and Databricks workflows.
• Mentor junior engineers, conduct code reviews, and ensure best practices in data engineering.
• Stay updated with the latest cloud technologies, big data frameworks, and industry trends.

Required Skills & Qualifications

• 4+ years of experience in data engineering, with strong proficiency in Kafka and Python.
• Expertise in Kafka Streams, Kafka Connect, and Schema Registry for real-time data processing.
• Experience with Azure Databricks (or willingness to learn and adopt it quickly).
• Hands-on experience with cloud platforms (Azure preferred, AWS or GCP is a plus).
• Proficiency in SQL, NoSQL databases, and data modeling for big data processing.
• Knowledge of containerization (Docker, Kubernetes) and CI/CD pipelines for data applications.
• Experience working with healthcare data (EHR, claims, HL7, FHIR, etc.) is a plus.
• Strong analytical skills, problem-solving mindset, and ability to lead complex data projects.
• Excellent communication and stakeholder management skills.

About Us
Emids is a leading provider of digital transformation solutions to the healthcare industry, serving payers, providers, life sciences, and technology firms. Headquartered in Nashville, Emids helps bridge critical gaps in providing accessible, affordable, and high-quality healthcare by providing digital transformation services, custom application development, data engineering, business intelligence solutions, and specialized consulting services to all parts of the healthcare ecosystem. With nearly 2000 plusprofessionals globally, Emids leverages strong domain expertise in healthcare-specific platforms, regulations, and standards to provide tailored, cutting-edge solutions and services to its clients.

At Emids, we are guided by our core values of "Trusted guides", "Impact obsessed" and " Inclusive innovators". We expect all Emidsians, including our candidates and partners, to embody these values in their day-to-day work. We're not scared of differences. It's how we break new ground. As we scale and elevate the experience of our clients in the healthcare and life sciences space and ultimately have an impact on every patient from every walk of life, the team we build must reflect the diversity that we serve. Together, we've built and will continue to grow, a diverse and inclusive culture where everyone has a seat at the table and the space to be their most authentic self. Emids believes in being an Equal Opportunity Employer and we support, celebrate, and cherish all the things that make our teammates who they are.