(Urgent) Data Engineer

Job Description

About the Tech@Lilly Organization:u00A0

Tech@Lilly builds and maintains capabilities using cutting edge technologies like most prominent tech companies. What differentiates Tech@Lilly is that we create new possibilities through tech to advance our purpose u2013 creating medicines that make life better for people around the world, like data driven drug discovery and connected clinical trials. We hire the best technology professionals from a variety of backgrounds, so they can bring an assortment of knowledge, skills, and diverse thinking to deliver innovative solutions in every area of the enterprise.u00A0

About the Business Function:u00A0u00A0

Tech@Lilly Business Units is a global organization strategically positioned so that through information and technology leadership and solutions, we create meaningful connections and remarkable experiences, so people feel genuinely cared for. The Business Unit IDS organization is accountable for designing, developing, and supporting commercial or customer engagement services and capabilities that span multiple Business Units (Bio-Medicines, Diabetes, Oncology, International), functions, geographies, and digital channels. The areas supported by Business Unit IDS includes: Customer Operations, Marketing and Commercial Operations, Medical Affairs, Market Research, Pricing, Reimbursement and Access, Customer Support Programs, Digital Production and Distribution, Global Patient Outcomes, and Real-World Evidence.u00A0

Job Title: Data Engineer u2013 Operationsu00A0

The Data Engineer u2013 Operations is responsible for the reliability, performance, and support of data pipelines, platforms, and workflows across the enterprise data ecosystem. With 2 - 12 years of experience, this role ensures smooth daily operations, incident resolution, proactive monitoring, and support of data integration and processing activities. The ideal candidate has strong experience in managing cloud-based data platforms (e.g., AWS, Databricks, Azure), production workflows, CI/CD pipelines, and data monitoring tools. This role partners closely with data engineers, analysts, platform teams, and business users to ensure SLA adherence and platform health. The position is critical to scaling data operations as organizations shift from vendor-led to in-house models.u00A0

u201CThis role is open across experience levels, and the final designation will be determined based on the interview and assessment outcomes.

What youu2019ll be doing:u00A0u00A0

- Monitor and manage day-to-day operations of data pipelines, ETL jobs, and cloud-native data platforms (e.g., AWS, Databricks, Redshift).u00A0

- Own incident response and resolution, including root cause analysis and post-mortem reporting for data failures and performance issues.u00A0

- Perform regular system health checks, capacity planning, and cost optimization across operational environments.u00A0

- Maintain and enhance logging, alerting, and monitoring frameworks using tools like CloudWatch, Datadog, Prometheus, etc.u00A0

- Collaborate with development teams to operationalize new data workflows, including CI/CD deployment, scheduling, and support documentation.u00A0

- Ensure data quality by executing validation checks, recon processes, and business rule compliance.u00A0

- Work with vendors (if applicable) and internal teams to support migrations, upgrades, and production releases.u00A0

How You Will Succeed:u202Fu00A0

u00A0

- Identify repetitive operational tasks and implement automation using Python, Airflow, Jenkins, or similar tools.u00A0

- Enable self-service capabilities and alerting for platform users and stakeholders.u00A0

u00A0

- Explore and propose how AI can be used to detect anomalies, predict issues, and accelerate root cause analysis.u00A0

- Collaborate with internal teams to experiment with LLMs, bots, or ML models for improving operational efficiency.u00A0

- Stay informed on emerging AIOps tools and work toward integrating them gradually.u00A0

u00A0

- Monitor pipeline performance and costs, and implement changes that optimize compute, memory, and storage usage.u00A0

- Recommend and trial AI/ML-based approaches for pipeline tuning, scheduling, or resource allocation.u00A0

u00A0

- Work with data engineers, analysts, and product owners to ensure seamless data availability and usability.u00A0

- Communicate incidents and resolutions clearly and proactively across teamsu00A0

What You should Bring:u00A0

- Strong background in managing and maintaining data pipelines, preferably in AWS or Azure environments.u00A0

- Proficiency in SQL, Python, or PySpark for operational debugging and performance tuning.u00A0

- Hands-on experience with monitoring tools (e.g., CloudWatch, Datadog) and orchestration frameworks like Airflow.u00A0

- Familiarity with CI/CD processes and code deployment practices using GitHub or similar tools.u00A0

- Awareness of data governance, privacy, and security protocols.u00A0

- Proactive problem-solving mindset with the ability to identify patterns in recurring issues.u00A0

- Exposure to AI/ML concepts or a passion for learning and applying automation through AI frameworks.u00A0

Basic Qualifications and Experience Requirement:u00A0u00A0

- Bacheloru2019s degree in Computer Science, Engineering, Information Systems, or a related field.u00A0

- 2 - 12 years of experience in Data Engineering, DataOps, Platform Reliability, or equivalent roles.u00A0

- Hands-on experience managing pipelines on Databricks, AWS Glue, EMR, Snowflake, or similar platforms.u00A0

- Strong scripting skills (Python, Bash) and familiarity with version control (Git).u00A0

- Experience with orchestration tools like Apache Airflow, AWS Step Functions, or similar.u00A0

- Exposure to monitoring/observability tools like CloudWatch, Datadog, Grafana, Prometheus, etc.u00A0

- Solid understanding of data lifecycle, job dependencies, and data validation techniques.u00A0

- Eagerness to learn and apply AI/ML approaches in operational workflows.u00A0

Additional Skills/Preferences:u00A0

- Domain experience in healthcare, pharmaceutical ( Customer Master, Product Master, Alignment Master, Activity, Consent etc.u00A0 ), or regulated industries is a plus.u00A0

- AWS, Google Cloud, or Databricks Certified Data Engineer (Associate/Professional)u00A0

- ITILu00AE Foundation or SRE Foundation certifiedu00A0

- AIOps Foundation or equivalent certificationu00A0

- Certification in Apache Airflow or any orchestration platform (e.g., Prefect)u00A0

Additional Information:u00A0

- Total number of positions open for this role : 30

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form () for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lillyu00A0does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly