Senior Data Engineer-AWS, Python

Embark on an exciting journey into the realm of software development with 3Pillar We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.Job SummaryFor this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.Work Location: Noida, Sec 62 Working days per week: 5Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a weekResponsibilities:Automate data workflows such as data ingestion, aggregation, and ETL processing.Logging and instrumentation of pipelines and services.Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.Advanced Support: Lead the monitoring (~2,300 jobs) and support of all scheduled and on-demand ETL jobs, acting as the primary escalation point for the team.Expert Troubleshooting: Own the most complex troubleshooting and Root Cause Analysis (RCA) efforts, ensuring detailed documentation and implementing long-term, structural fixes.Optimization: Proactively perform performance tuning to improve system efficiency and reliability. Actively identify and implement cost optimization strategies for AWS Glue usage.Automation & Development: Design and develop new automation solutions (using Glue, Lambda, and Step Functions) to reduce the need for manual support. Lead development for business enhancement requirements.Data Management: Oversee and execute complex data integrity tasks, including reloading CDC tables and managing data synchronization between external databases (like Teradata) and AWS Glue.Integration: Implement AWS Lambda functions to trigger external events, such as SAS & RMJ, as needed.Decision & Execution: Make critical, autonomous decisions under pressure to resolve incidents and execute complex tasks, ensuring all SLAs are met. Escalate to leadership only when necessary.Mentorship: Mentor junior engineers, lead knowledge transfer sessions with internal resources, and oversee the creation of high-quality technical documentation and runbooks. Qualifications:Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services.Demonstrated expertise with a minimum of 5-8 years of relevant experience.Deep understanding of Data engineering, ETL principles and data warehouse design.Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue.Automation: Proven ability to design and develop automation solutions from the ground up.Problem-Solving: Expert-level troubleshooting and RCA skills.Leadership: Ability to work independently, make critical decisions, and provide technical guidance.Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.