Sr. Data Engineer

Job Description About Position: We are currently seeking a highly proficient Sr Data Engineer to be part of a global team and function in a lead developer role. You will perform analysis, design and development of data processing ETL job streams to support the data warehouses, data marts and other data stores in support of the Optum business. You will do hands-on development and testing work as part of a scrum team that is global in nature. The primary technology stack will be Azure Databricks environment, with Delta Lake, Airflow, Python components and a Snowflake data store. You will work closely with external and internal stakeholders to design solutions to solve problems and resolve issues. - Role: Sr. Data Engineer - Location: All Persistent Locations - Experience: 7 to 10 Years - Job Type: Full Time Employment What You'll Do: - As Data Engineer, you will evaluate, design, develop, maintain and test Big Data solutions. You will be involved in data engineering activities like creating pipelines/ workflows for source to target mappings - Responsible for data ingestion from files, streams and databases. Process the data using Azure Databricks, Python, SQL. - Design and development of ETL data flows using Azure Databricks ecosystem components. Lead the development of large-scale, high-speed, and low-latency data solutions in the areas of large-scale data manipulation, long-term data storage, data warehousing, low-latency retrieval systems, real-time reporting and analytics data applications, ad-hoc analytics. Performance tune long running jobs. - Develop programs in Python and SQL as part of data cleaning and processing. - Develop efficient code for multiple use cases leveraging Python and Big Data technologies on the Azure platform in an optimal manner - Maintain high operational excellence guaranteeing high availability and platform stability - Document architecture and design components. Be able to communicate these to other engineers on the team. Guide and mentor junior engineers - Follow DevOps practices to automate the data pipelines Expertise You'll Bring: - 5+ years of overall experience in Big Data with prior experience in ETL - Familiarity with the application of AI/ML tools and models to data processing and ETL workloads - Strong, advanced expertise in Azure Databricks, Delta Lake and Python - Strong knowledge of SQL - 3+ years of DevOps and CI/CD pipeline implementation and management - Working experience with Git hub - Experience designing and developing components for a cloud application, preferably Azure - Good oral and written communication skills. Benefits: - Competitive salary and benefits package - Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications - Opportunity to work with cutting-edge technologies - Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards - Annual health check-ups - Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents Values-Driven, People-Centric & Inclusive Work Environment: Persistent Ltd. is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds. - We support hybrid work and flexible hours to fit diverse lifestyles. - Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities. - If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment Let's unleash your full potential at Persistent - persistent.com/careers Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.