Senior Data Engineer-aws, Python

Embark on an exciting journey into the realm of software development with 3Pillar We extend an invitation for you to join our team and gear up for a thrilling adventure At 3Pillar our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making As a Senior Data Engineer you will hold a crucial position within our dynamic team actively contributing to thrilling projects that reshape data analytics for our clients providing them with a competitive advantage in their respective industries Job Summary For this role we are seeking a Senior Data Engineer for our Client s ETL Support Team to act as a technical pillar during our Client s transition to an AWS-native data environment You will be a key escalation point responsible for handling the most complex data pipeline issues This role may move beyond simple support you might be requested to harden and improve our Client s systems by driving performance tuning cost optimization and automation initiatives Work Location Noida Sec 62 Working days per week 5 Rotational Shift Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week Responsibilities Automate data workflows such as data ingestion aggregation and ETL processing Logging and instrumentation of pipelines and services Ensure data accuracy integrity privacy security and compliance through quality control procedures Advanced Support Lead the monitoring 2 300 jobs and support of all scheduled and on-demand ETL jobs acting as the primary escalation point for the team Expert Troubleshooting Own the most complex troubleshooting and Root Cause Analysis RCA efforts ensuring detailed documentation and implementing long-term structural fixes Optimization Proactively perform performance tuning to improve system efficiency and reliability Actively identify and implement cost optimization strategies for AWS Glue usage Automation Development Design and develop new automation solutions using Glue Lambda and Step Functions to reduce the need for manual support Lead development for business enhancement requirements Data Management Oversee and execute complex data integrity tasks including reloading CDC tables and managing data synchronization between external databases like Teradata and AWS Glue Integration Implement AWS Lambda functions to trigger external events such as SAS RMJ as needed Decision Execution Make critical autonomous decisions under pressure to resolve incidents and execute complex tasks ensuring all SLAs are met Escalate to leadership only when necessary Mentorship Mentor junior engineers lead knowledge transfer sessions with internal resources and oversee the creation of high-quality technical documentation and runbooks Qualifications Technical Expertise In-depth expert-level knowledge and extensive hands-on experience with AWS S3 Glue Lambda Step Functions and other related services Demonstrated expertise with a minimum of 5-8 years of relevant experience Deep understanding of Data engineering ETL principles and data warehouse design Optimization Experience Demonstrable experience in system performance tuning and cloud cost optimization specifically with AWS Glue Automation Proven ability to design and develop automation solutions from the ground up Problem-Solving Expert-level troubleshooting and RCA skills Leadership Ability to work independently make critical decisions and provide technical guidance Availability Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis Database Skills Strong experience with Change Data Capture CDC managing database synchronization and working with Teradata