Lead Data Engineer

About Delta Air Lines About the Company Delta Air Lines NYSE DAL is the U S global airline leader in safety innovation reliability and customer experience Powered by our employees around the world Delta has for a decade led the airline industry in operational excellence while maintaining our reputation for award-winning customer service With our mission of connecting the people and cultures of the globe Delta strives to foster understanding across a diverse world and serve as a force for social good In 2024 Delta was recognized by Fortune with a No 11 placement on Fortune s Top 50 Most Admired Companies list The company s strong management and commitment to providing elevated experiences and premium products also secured a No 1 ranking out of the eight airlines on the list Delta landed on TIME s inaugural list of the World s Best Companies coming in at No 12 - the only U S airline in the top 155 TIME s award is based on three criteria employee satisfaction revenue growth and sustainability Delta s people-first culture continues to be recognized earning the airline a spot on Fortune s 100 Best Companies to Work For list for the fifth year Delta is the only airline included on the 2024 list Additionally Delta earned a coveted spot on Fast Company s list of the Most Innovative Companies climbing from its No 8 spot in 2023 to No 2 in the travel category The airline was recognized for its Wi-Fi revolution that is working to ensure the future of travel is connected About the Delta Technology Hub DTH Bangalore Delta has fast emerged as a customer-focused innovation-led technology-enabled business The Delta Technology Hub will contribute directly to these objectives About Delta Tech Hub Delta Air Lines NYSE DAL is the U S global airline leader in safety innovation reliability and customer experience Powered by our employees around the world Delta has for a decade led the airline industry in operational excellence while maintaining our reputation for award-winning customer service With our mission of connecting the people and cultures of the globe Delta strives to foster understanding across a diverse world and serve as a force for social good Delta has fast emerged as a customer-oriented innovation-led technology-driven business The Delta Technology Hub will contribute directly to these objectives It will sustain our long-term aspirations of delivering niche IP-intensive high-value and innovative solutions It supports various teams and functions across Delta and is an integral part of our transformation agenda working seamlessly with a global team to create memorable experiences for customers Responsibilities Data Pipeline Development and Maintenance Design build and optimize scalable ETL ELT pipelines using AWS Serverless services Lambda Glue s3 Airflow to ingest data from diverse sources such as APIs cloud platforms and databases Ensure pipelines are robust efficient and capable of handling large volumes of data Data Integration and Harmonization Aggregate and normalize marketing performance data from multiple platforms e g Adobe Analytics Salesforce Marketing Cloud Teradata data warehouse ad platforms Implement data transformation and enrichment processes to support analytics and reporting needs Data Quality and Monitoring Develop and implement data validation and monitoring frameworks to ensure data accuracy and consistency Troubleshoot and resolve issues related to data quality latency or performance Collaboration with Stakeholders Partner with marketing teams analysts and data scientists to understand data requirements and translate them into technical solutions Provide technical support and guidance on data-related issues or projects Tooling and Automation Leverage cloud-based solutions and frameworks e g AWS serverless services - Lambda Glue s3 Airflow to streamline processes and enhance automation Maintain and optimize existing workflows while continuously identifying opportunities for improvement Documentation and Best Practices Document pipeline architecture data workflows and processes for both technical and non-technical audiences Follow industry best practices for version control security and data governance Continuous Learning and Innovation Stay current with industry trends tools and technologies in data engineering and marketing analytics Recommend and implement innovative solutions to improve the scalability and efficiency of data systems Lead and mentor other junior members within the team What you need to succeed minimum qualifications Bachelor of Science degree in Computer Science or equivalent At least 6 years of post-degree professional experience as a data engineer developing and maintaining data pipelines Extensive experience with databases and data platforms Teradata and AWS serverless - Lambda Glue s3 Airflow 5 Hands-on experience in designing implementing managing large scale data and ETL solutions utilizing AWS Compute Storage and database services S3 Lambda Redshift Glue Athena 3 years of experience as a tech lead with experience in mentorship and guidance task planning and delegation collaboration across teams and technical communication Should have experience in architecture and design decisions code quality and reviews hands-on development and tech stack and tools management Should have good working knowledge on sprint planning and execution risk management performance and stability focus Proficiency in Python SQL PySpark Glue Lambda S3 and other AWS tool sets Good understanding of data warehouses ETL AWS serverless Strong analytical and programming skills with the ability to solve data-related challenges efficiently At least 7 years of post-degree professional experience as a data engineer developing and maintaining data pipelines Collaborate with product owners to understand requirements and translate them into technical specs Perform regular code reviews and advocate for best practices testing CI CD documentation Proven ability to learn new data models quickly and apply them effectively in a fast-paced environment Excellent communication skills with the ability to present complex data findings to both technical and non-technical audiences What will give you a competitive edge preferred qualifications Desired Airline industry experience Desired experience working with marketing and media data Experience working with SAS Statistical Analysis System to develop data pipelines AWS certifications Solution Architect SAA SAP or Data Analytics Specialty DAS Experience migrating data pipelines and systems to modern cloud-based solutions Familiarity with marketing data platforms