Principal Data Engineer

Skill : Principal Data Engineer Exp range : 12 yrs to 25 yrs Job location : Bangalore PFB Job Description : Possesses hands-on technical experience working in AWS. The individual should possess a robust set of technical and soft skills and be an excellent AWS Data Engineer with a focus on complex Automation and Engineering Framework development. Being well-versed in Python is mandatory, and experience in developing complex frameworks using Python is required. Passionate about Cloud/DevSecOps/Automation and possess a keen interest in solving complex problems systematically. Drive the development and implementation of scalable data solutions and data pipelines using various AWS services. Possess the ability to work independently and collaborate closely with team members and technology leads. Exhibit a proactive approach, constantly seeking innovative solutions to complex technical challenges. Can take responsibility for nominated technical assets related to areas of expertise, including roadmaps and technical direction. Can own and develop technical strategy, overseeing medium to complex engineering initiatives. Lead the design and architecture of large-scale data systems, ensuring they meet business needs and performance requirements. Provide strategic direction for data engineering initiatives, aligning them with organizational goals. Mentor and guide senior data engineers, fostering a culture of continuous improvement and innovation. Collaborate with executive leadership to define and prioritize data engineering projects. Ensure the implementation of best practices in data management, security, and governance. Act as a System and Data Architect, designing and implementing robust, scalable, and efficient data architectures. Demonstrate capabilities to work in a fast-paced, high-pressure environment while delivering high-quality results on time. Conduct and drive multiple extensive Proof of Concepts (PoCs) simultaneously, focusing on research, automation, and framework development. Bring in the latest and greatest cloud-based solutions, ensuring the organization stays at the forefront of technology. Experience or exposure to Generative AI, Large Language Models (LLM), and Agentic AI is highly desirable. Highly technical and hands-on coding experience on multiple tech stacks related to Data Engineering and framework development on and for AWS Cloud. Essential Skills: 15-18 years of experience as a Data Engineering professional in a data-intensive environment. The individual should have strong analytical and reasoning skills in the relevant area. Proficiency in AWS cloud services, specifically EC2, S3, Lambda, Athena, Kinesis, Redshift, Glue, EMR, DMS, Event Bridge, DynamoDB, IAM, Secret Manager, KMS, Step functions, SQS, SNS, Experience in Cloud Foundation (CFN), Cloud Development Kit (CDK) and Software Development Kit (SDK).Exposure to MSK, Flink, Iceberg (good to have) Cloud Watcher, User Notification, Appflow, Fargate, ECS, EKS, Airflow, Boto3, RDS, Sagemaker, IAM, , Cookie Cutter. Excellent skills in Python-based framework development are mandatory. Proficiency in SQL for efficient querying, managing databases, handling complex queries, and optimizing query performance. Excellent automation skills are expected in areas such as: Automating the testing framework and various test cases including unit, integration, functional tests, and mock-ups using tools such as PyPy, Pytest Modularisation and automating the data pipeline and expediting tasks such as data ingestion and transformation using various tools like DBT. API-based automated and integrated calls (REST, cURL, authentication & authorization, tokens, pagination, open Api, Swagger). Implementing advanced engineering techniques and handling ad hoc requests to automate processes on demand. Implementing automated and secured file