Senior Data Engineer-AWS, Python

Embark on an exciting journey into the realm of software development with 3Pillar We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.

Job Summary

For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives.

Work Location: Noida, Sec 62 

Working days per week: 5

Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week

• Automate data workflows such as data ingestion, aggregation, and ETL processing.
• Logging and instrumentation of pipelines and services.
• Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
• Advanced Support: Lead the monitoring (~2,300 jobs) and support of all scheduled and on-demand ETL jobs, acting as the primary escalation point for the team.
• Expert Troubleshooting: Own the most complex troubleshooting and Root Cause Analysis (RCA) efforts, ensuring detailed documentation and implementing long-term, structural fixes.
• Optimization: Proactively perform performance tuning to improve system efficiency and reliability. Actively identify and implement cost optimization strategies for AWS Glue usage.
• Automation & Development: Design and develop new automation solutions (using Glue, Lambda, and Step Functions) to reduce the need for manual support. Lead development for business enhancement requirements.
• Data Management: Oversee and execute complex data integrity tasks, including reloading CDC tables and managing data synchronization between external databases (like Teradata) and AWS Glue.
• Integration: Implement AWS Lambda functions to trigger external events, such as SAS & RMJ, as needed.
• Decision & Execution: Make critical, autonomous decisions under pressure to resolve incidents and execute complex tasks, ensuring all SLAs are met. Escalate to leadership only when necessary.
• Mentorship: Mentor junior engineers, lead knowledge transfer sessions with internal resources, and oversee the creation of high-quality technical documentation and runbooks.

• Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services.
• Demonstrated expertise with a minimum of 5-8 years of relevant experience.
• Deep understanding of Data engineering, ETL principles and data warehouse design.
• Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue.
• Automation: Proven ability to design and develop automation solutions from the ground up.
• Problem-Solving: Expert-level troubleshooting and RCA skills.
• Leadership: Ability to work independently, make critical decisions, and provide technical guidance.
• Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis.
• Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.