AWS Data Engineer
2 months ago
AWS Data Engineer (Senior) with skills Data Engineering, Kafka, Python, Scala, postgreSQL Development, AWS - EKS, AWS - CloudFormation, Data Modeling, ETL, Apache Hive, AWS-Apps, AWS-Infra, Apache Airflow, SQL, Datadog, Splunk, Apache Spark, AWS DBA for location Gurugram, India
Posted on: July 30, Share on Linkedin Share on Twitter Share on Facebook
ROLES & RESPONSIBILITIES
We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in ETL, Data Modelling, and Data Architecture. Proficiency in ETL optimization, designing, coding, and tuning big data processes using Scala is essential, along with hands-on experience in stream data processing using Spark, Kafka, and Spark Structured Streaming.
Additionally, the candidate should have extensive experience in building data platforms using a variety of technologies, including Scala, SQL/PLSQL, PostgreSQL, SQL Server, Teradata, Spark, Spark Structured Streaming, Kafka, Parquet/ORC, Data Modelling (Relational Dimensional E-R Modelling), ETL, RDS (PostgreSQL, MySQL), Splunk, DataDog, Airflow, Git, CI/CD Jenkins, JIRA, Confluence, IntelliJ Idea, Agile - Scrum/Kanban, On Call & Operations, Code Review, RCP Framework, Query book, Build, Deployment CI/CD & Release Process, Backstage, PagerDuty, and Spinnaker.
Key Responsibilities:
Hands-on experience on developing Data platform and its components Data Lake, cloud Datawarehouse, APIs, Batch and streaming data pipeline Experience with building data pipelines and applications to stream and process large datasets at low latency.
· Develop and maintain batch and stream processing data solutions using Apache Spark, Kafka, and Spark Structured Streaming.
· Work on orchestration using Airflow to automate and manage data workflows.
· Utilize project management tools like JIRA and Confluence to track progress and collaborate with the team.
· Develop data processing workflows utilizing Spark, SQL/PLSQL, and Scala to transform and cleanse raw data into a usable format.
· Implement data storage solutions leveraging Parquet/ORC formats on platforms such as PostgreSQL, SQL Server, Teradata, and RDS (PostgreSQL, MySQL).
· Optimize data storage and retrieval performance through efficient data modelling techniques, including Relational, Dimensional, and E-R modelling.
· Maintain data integrity and quality by implementing robust validation and error handling mechanisms within ETL processes.
· Automate deployment processes using CI/CD tools like Jenkins and Spinnaker to ensure reliable and consistent releases.
· Monitor and troubleshoot data pipelines using monitoring tools like DataDog and Splunk to identify performance bottlenecks and ensure system reliability.
· Participate in Agile development methodologies such as Scrum/Kanban, including sprint planning, daily stand-ups, and retrospective meetings.
· Conduct code reviews to ensure adherence to coding standards, best practices, and scalability considerations.
· Manage and maintain documentation using tools like Confluence to ensure clear and up-to-date documentation of data pipelines, schemas, and processes.
· Provide on-call support for production data pipelines, responding to incidents and resolving issues in a timely manner.
· Collaborate with cross-functional teams including developers, data scientists, and operations teams to address complex data engineering challenges.
· Stay updated on emerging technologies and industry trends to continuously improve data engineering processes and tools.
· Contribute to the development of reusable components and frameworks to streamline data engineering tasks across projects.
· Utilize version control systems like Git to manage codebase and collaborate effectively with team members.
· Leverage IDEs like IntelliJ IDEA for efficient development and debugging of data engineering code.
· Adhere to security best practices in handling sensitive data and implementing access controls within the data lake environment.
Good-to-Know Skills:
· Programming Languages: Python, Bash/Unix/Linux
· Big Data Technologies: Hive, Avro, Apache Iceberg, Delta Format
· Cloud Services: EC2, ECS, S3, SNS, SQS, CloudWatch
· Databases: DynamoDB, Redis
· Containerization and Orchestration: Docker, Kubernetes
· CI/CD Tools: Github Copilot
· Additional Skills: Maven, CLI/SDK
Nice-to-Have Skills:
· Networking: Subnets, Routes
· Big Data Technologies: Flink
Key Responsibilities:
Hands-on experience on developing Data platform and its components Data Lake, cloud Datawarehouse, APIs, Batch and streaming data pipeline Experience with building data pipelines and applications to stream and process large datasets at low latency.
· Develop and maintain batch and stream processing data solutions using Apache Spark, Kafka, and Spark Structured Streaming.
· Work on orchestration using Airflow to automate and manage data workflows.
· Utilize project management tools like JIRA and Confluence to track progress and collaborate with the team.
· Develop data processing workflows utilizing Spark, SQL/PLSQL, and Scala to transform and cleanse raw data into a usable format.
· Implement data storage solutions leveraging Parquet/ORC formats on platforms such as PostgreSQL, SQL Server, Teradata, and RDS (PostgreSQL, MySQL).
· Optimize data storage and retrieval performance through efficient data modelling techniques, including Relational, Dimensional, and E-R modelling.
· Maintain data integrity and quality by implementing robust validation and error handling mechanisms within ETL processes.
· Automate deployment processes using CI/CD tools like Jenkins and Spinnaker to ensure reliable and consistent releases.
· Monitor and troubleshoot data pipelines using monitoring tools like DataDog and Splunk to identify performance bottlenecks and ensure system reliability.
· Participate in Agile development methodologies such as Scrum/Kanban, including sprint planning, daily stand-ups, and retrospective meetings.
· Conduct code reviews to ensure adherence to coding standards, best practices, and scalability considerations.
· Manage and maintain documentation using tools like Confluence to ensure clear and up-to-date documentation of data pipelines, schemas, and processes.
· Provide on-call support for production data pipelines, responding to incidents and resolving issues in a timely manner.
· Collaborate with cross-functional teams including developers, data scientists, and operations teams to address complex data engineering challenges.
· Stay updated on emerging technologies and industry trends to continuously improve data engineering processes and tools.
· Contribute to the development of reusable components and frameworks to streamline data engineering tasks across projects.
· Utilize version control systems like Git to manage codebase and collaborate effectively with team members.
· Leverage IDEs like IntelliJ IDEA for efficient development and debugging of data engineering code.
· Adhere to security best practices in handling sensitive data and implementing access controls within the data lake environment.
Good-to-Know Skills:
· Programming Languages: Python, Bash/Unix/Linux
· Big Data Technologies: Hive, Avro, Apache Iceberg, Delta Format
· Cloud Services: EC2, ECS, S3, SNS, SQS, CloudWatch
· Databases: DynamoDB, Redis
· Containerization and Orchestration: Docker, Kubernetes
· CI/CD Tools: Github Copilot
· Additional Skills: Maven, CLI/SDK
Nice-to-Have Skills:
· Networking: Subnets, Routes
· Big Data Technologies: Flink
EXPERIENCE
6-8 Years
SKILLS
Primary Skill: Data Engineering Sub Skill(s): Data Engineering Additional Skill(s): Kafka, Python, Scala, postgreSQL Development, AWS - EKS, AWS - CloudFormation, Data Modeling, ETL, Apache Hive, AWS-Apps, AWS-Infra, Apache Airflow, SQL, Datadog, Splunk, Apache Spark, AWS DBA
ABOUT THE COMPANY
Infogain is a human-centered digital platform and software engineering company based out of Silicon Valley. We engineer business outcomes for Fortune companies and digital natives in the technology, healthcare, insurance, travel, telecom, and retail & CPG industries using technologies such as cloud, microservices, automation, IoT, and artificial intelligence. We accelerate experience-led transformation in the delivery of digital platforms. Infogain is also a Microsoft (NASDAQ: MSFT) Gold Partner and Azure Expert Managed Services Provider (MSP).
Infogain, an Apax Funds portfolio company, has offices in California, Washington, Texas, the UK, the UAE, and Singapore, with delivery centers in Seattle, Houston, Austin, Kraków, Noida, Gurgaon, Mumbai, Pune, and Bengaluru.
-
Data Engineer
2 weeks ago
Gurugram, India NTT DATA Services Full timeReq ID: 282638 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Gurgaon, Haryāna (IN-HR), India (IN).Data EngineerJob SummaryWe are...
-
Data Engineer
1 week ago
Gurugram, India True Tech Professionals Full timeRole : Lead Data Engineer (Multiple Positions)Location : Remote - Gurgaon (Office visit - once in a while - as and when required)Experience : 4 to 6 yearsKey Responsibilities :- Data Pipeline Development: Design, implement, and maintain scalable data pipelines to extract, transform, and load (ETL) data from various sources into data warehouses or other...
-
AWS Data Engineer
2 months ago
Gurugram, India Infogain Full timeAWS Data Engineer (Lead) with skills Data Engineering, Python, AWS - CloudFormation, Apache Hive, MySQL, SQL, Azure SQL DBA, Azure SQL Development, AWS DBA for location Gurugram, India Posted on: July 29, Share on Linkedin Share on Twitter Share on Facebook ROLES & RESPONSIBILITIES Job Description Responsibilities Lead prototyping efforts...
-
Lead Data Engineer/Manager
3 months ago
Gurugram, India HuQuo Consulting Pvt. Ltd. Full timeJob Title: Lead Data EngineerJob SummaryLooking for a versatile professional to join our team in the role of Hybrid Data Engineer and Database Administrator. This position requires a blend of expertise in Data Engineering and Database Administration, with a strong foundation in Oracle DBA roles and a demonstrated transition into data engineering. The ideal...
-
Data Engineer
2 months ago
Gurugram, India Jobtrix Full timeRole : Data EngineerLocation : GurgaonJob Description :We are seeking a skilled Data Engineer with expertise in Python, AWS Glue, and ETL processes. You will be responsible for designing, developing, and maintaining our data infrastructure, ensuring seamless data integration and high performance.Key Responsibilities :- Develop and optimize ETL pipelines...
-
Data Engineer
3 months ago
Gurugram, India LITTLE ROBIN TECHNOLOGY PARTNERS PRIVATE LIMITED Full timeRole : Data EngineerExperience : 5+ YearsLocation : Gurgaon (WFO)Job Description :We are seeking a skilled Data Engineer with expertise in Python, AWS Glue, and ETL processes. You will be responsible for designing, developing, and maintaining our data infrastructure, ensuring seamless data integration and high performance.Key Responsibilities :- Develop and...
-
Data Engineer
3 months ago
Gurugram, India JobTrix Full timeJob Description :We are seeking a skilled Data Engineer with expertise in Python, AWS Glue, and ETL processes. You will be responsible for designing, developing, and maintaining our data infrastructure, ensuring seamless data integration and high performance.Key Responsibilities :- Develop and optimize ETL pipelines using AWS Glue.- Maintain and enhance data...
-
Customer Solutions Manager, AWS India Public Sector
3 months ago
Gurugram, India AWS India - Telangana Full timeSales, Marketing and Global Services (SMGS)AWS Sales, Marketing, and Global Services (SMGS) is responsible for driving revenue, adoption, and growth from the largest and fastest growing small- and mid-market accounts to enterprise-level customers including public sector.At Amazon Web Services (AWS) India, we are changing the future of IT. Customer Solutions...
-
Advisory AWS Data Engineer Manager
4 months ago
Gurugram, India KPMG India Full timeEvaluating, developing, maintaining and testing data engineering solutions for Data Lake and advanced analytics projects. Implement processes and logic to extract, transform, and distribute data across one or more data stores from a wide variety of sources Distil business requirements and translate into technical solutions for data systems including data...
-
AWS Data Engineer
5 months ago
Gurugram, India HuQuo Consulting Pvt. Ltd. Full timeJob Description : Responsibilities :- Proficient in designing and implementing data ingestion pipelines from various sources.- Build and enhance Python and PySpark-based frameworks for efficient and scalable data ingestion processes.- Collaborate with cross-functional teams to understand requirements and tailor frameworks to meet specific business needs.-...
-
Data Engineer
3 months ago
Gurugram, India SKYGATE CONSULTING Full timeWe are Immediately hiring Data Engineer with 4+ years of experienceLocation : Bangalore/Gurgaon/NagpurNP : Immediate(June joinees)Job Description :Skills :- Extensive development experience using snowflake or similar data warehouse technology- Having working experience with dbt and other technologies of the modern data stack, such as Snowflake, Apache...
-
Advisory AWS Data Platform Senior
3 months ago
Gurugram, India KPMG India Full timeProficient in any object-oriented/ functional scripting languages: Java, Python, Node etc. Experience in using AWS SDKs for creating data pipelines - ingestion, processing, and orchestration. Hands on experience in working with big data on AWS environment including cleaning/transforming/cataloguing/mapping etc. Good understanding of AWS components, storage...
-
Data Engineer
3 weeks ago
Gurugram, India KPMG India Full timeData Analyst / Data Engineer Location: Gurgaon, India Experience Required: 4-7 years Key Skills: AWS, PySpark Job Description: We are looking for a talented Data Analyst/Data Engineer to join our team in Gurgaon. The ideal candidate will possess strong technical skills in AWS and PySpark, with experience in managing and analyzing large-scale data...
-
Data Engineer
3 weeks ago
Gurugram, India KPMG India Full timeData Analyst / Data EngineerLocation: Gurgaon, IndiaExperience Required: 4-7 yearsKey Skills: AWS, PySparkJob Description:We are looking for a talented Data Analyst/Data Engineer to join our team in Gurgaon. The ideal candidate will possess strong technical skills in AWS and PySpark, with experience in managing and analyzing large-scale data environments....
-
Lead Data Engineer
3 months ago
Delhi NCR/Gurgaon/Gurugram, India HuQuo Consulting Pvt. Ltd. Full timeResponsibilities :- Should possess hands-on experience in working on some of the relational and non-relational databases (Oracle/SQL Server/DB2/PostgreSQL/MySQL/Golden Gate/Data Guard) on AWS and/or on-prem infrastructure.- 3+ years' Experience with AWS Services like AWS RDS, Aurora, EC2, S3, DMS, AWS Backups, and DynamoDB.- 3+ years' Experience...
-
Senior Associate Data Engineering L2 DE
1 week ago
Gurugram, India Publicis Sapient Full timeCompany DescriptionPublicis Sapient is a digital transformation partner helping established organizations get to their future, digitally enabled state, both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods, fusing strategy, consulting and customer experience with agile engineering...
-
Senior Associate Data Engineering L2 DE
7 days ago
Gurugram, India Publicis Sapient Full timeCompany Description Publicis Sapient is a digital transformation partner helping established organizations get to their future, digitally enabled state, both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods, fusing strategy, consulting and customer experience with agile...
-
AWS Cloud Engineer L2
3 months ago
gurugram, India Hitachi Systems India Pvt Ltd Full timeHitachi Systems is hiring for AWS/Azure L2/L3 Cloud Engineer Location- Gurgaon (Work from Office) Exp- 4-12 Yrs JD as Below :- AWS Cloud Engineer (L3) MAIN RESPONSIBILITIES : Ensure operational supportability of design, development, and engineering by working with operational support team in design and development such that the solution is operationally...
-
AWS Cloud Engineer L2
3 months ago
Gurugram, India Hitachi Systems India Pvt Ltd Full timeHitachi Systems is hiring for AWS/Azure L2/L3 Cloud EngineerLocation- Gurgaon (Work from Office)Exp- 4-12 YrsJD as Below :-AWS Cloud Engineer (L3)MAIN RESPONSIBILITIES:Ensure operational supportability of design, development, and engineering by working with operational support team in design and development such that the solution is operationally supportable...
-
AWS Cloud Engineer
2 weeks ago
Gurugram, India Randstad Digital Full timeHiring for AWS Cloud Engineer.Experience : 2-5 YearsLocation : GurgaonNeed only Immediate JoinersPosition Overview :We are seeking a highly skilled AWS Engineer with extensive experience in setting up AWS infrastructure using Terraform. The ideal candidate will be proficient in designing, implementing, and managing AWS services including S3, Redshift, EMR,...