Spark + Aws Emr

7 days ago

Chennai Tamil Nadu, India Cognizant Full time

**Job Summary**

**Responsibilities**
- Develop and maintain data pipelines using Apache Airflow to ensure efficient data processing and workflow automation.
- Utilize Hive for data warehousing solutions ensuring data is stored and queried effectively.
- Write and optimize SQL queries to manage and manipulate data across various databases.
- Implement and manage data processing on AWS EMR ensuring scalable and cost-effective solutions.
- Leverage Apache Spark for distributed data processing ensuring high performance and reliability.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
- Monitor and troubleshoot data pipelines and workflows to ensure smooth operation and mínimal downtime.
- Provide technical guidance and support to junior developers fostering a collaborative and knowledge-sharing environment.
- Stay updated with the latest industry trends and technologies to continuously improve data processing capabilities.
- Ensure data security and compliance with relevant regulations and standards.
- Document technical specifications and processes to maintain clear and comprehensive records.
- Participate in code reviews to ensure code quality and adherence to best practices.

**Qualifications**
- Extensive experience with Apache Airflow for workflow automation and scheduling.
- Proficiency in Hive for data warehousing and query optimization.
- Strong SQL skills for data manipulation and management.
- Hands-on experience with AWS EMR for scalable data processing.
- In-depth knowledge of Apache Spark for distributed data processing.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills.
- Ability to work in a hybrid model and adapt to changing project requirements.
- Experience with data security and compliance standards.
- Familiarity with code review processes and best practices.
- Commitment to continuous learning and improvement.
- Proven track record of delivering high-quality data solutions.

  • Aws Emr 4 to 9 Years Pan India

    7 days ago

    Chennai, Tamil Nadu, India Capgemini Engineering Full time

    **Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues around the world, and where you’ll be able to reimagine what’s possible. Join us and help the world’s leading organizations unlock the value of...

  • Aws Emr 4 to 9 Years Pan India

    1 week ago

    Chennai, Tamil Nadu, India Capgemini Engineering Full time

    Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you’d like, where you’ll be supported and inspired by a collaborative community of colleagues around the world, and where you’ll be able to reimagine what’s possible. Join us and help the world’s leading organizations unlock the value of...

  • Aws Data Architect

    2 weeks ago

    Chennai, Tamil Nadu, India Whitefield Careers Full time

    **AWS Principal Data Architect - Hadoop Migration** We are seeking an experienced AWS Principal Data Architect to lead the migration of Hadoop DWH workloads from on-premise to AWS EMR. As an AWS Data Architect, you will be a recognized expert in cloud data engineering, developing solutions designed for effective data processing and warehousing requirements...

  • Aws Data Scientist

    2 weeks ago

    Tamil Nadu, India AES Technologies Pvt. Ltd. Full time

    Mode of Work : Remote. Type of Work : Contractual [1 Year Extendable depends upon the progress]. Salary: Key Skills: AWS Core & AI/Analytics Services: - Amazon SageMaker (ML model development & deployment). - Amazon Bedrock (generative AI & foundation models). - Amazon Redshift (data warehousing). - Amazon EMR (big data processing with Spark/Hadoop). -...

  • Spark & Scala Developer (AWS EKS / AWS Kubernetes)-Bangalore

    4 days ago

    Bengaluru, Chennai, Hyderabad, India Infosys Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are seeking an experienced Spark & Scala Developer with strong expertise in big data processing, functional programming, and cloud-native deployments using AWS EKS/Kubernetes.The ideal candidate will design and develop scalable data pipelines and distributed applications, ensuring high performance and reliability in cloud environments.Key...

  • Spark Scala, Aws Kolkata&hyderabad

    1 week ago

    Chennai, Tamil Nadu, India Diverse Lynx Full time

    Hiring for Spark Scala AWS KOLKATA HYDERABAD Design and develop robust scalable data pipelines using Apache Spark Core SQL Streaming MLlib etc with Scala Write clean modular production-grade Scala code adhering to coding standards and best practices Optimize Spark jobs for performance reliability and cost-efficiency including tuning Spark configurations and...

  • AWS Data Engineer

    4 days ago

    tamil nadu, India Tata Consultancy Services Full time

    Role AWS Data EngineerRequired Technical Skill Set AWS Redshift, Glue, PySparkDesired Experience Range 4-10 YearsLocation of Requirement Chennai/PuneDesired Competencies (Technical/Behavioral Competency)Must-Have • Strong hands-on experience in Python programming and PySpark• Experience using AWS services (RedShift, Glue, EMR, S3 & Lambda)• Experience...

  • Big Data Engineer

    2 days ago

    Chennai, Tamil Nadu, India CIEL HR Full time

    Job Responsibilities Responsible for administering big data systems like EMR HDFS YARN Spark Hive Oozie and others Work closely with development teams at the design phase of big data jobs and pipelines to improve operational efficiency Troubleshoot big data systems for capacity bottlenecks of memory CPU OS storage and network Performance tuning of Hadoop...

  • Aws DevOps

    1 week ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **AWS Devops** 1. Need SA (5+ experience) with AWS DevOps experience. 2. Experience with AWS services like IAM, S3, Lambda, EC2, ECS, CloudWatch, CloudFormation,EC2,EBS,S3,EMR,IAM,GLUE,ATHENA,VPC, SUBNET, SG, RDS,Etc. 3. Experience with Docker and Kubernetes. 4. Experience with security best practices (e.g. using IAM Roles, KMS, etc.). 5. Experience in...

  • AWS Data Engineer

    4 days ago

    Chennai, India Tata Consultancy Services Full time

    Role AWS Data EngineerRequired Technical Skill Set AWS Redshift, Glue, PySparkDesired Experience Range 4-10 YearsLocation of Requirement Chennai/PuneDesired Competencies (Technical/Behavioral Competency)Must-Have • Strong hands-on experience in Python programming and PySpark• Experience using AWS services (RedShift, Glue, EMR, S3 & Lambda)• Experience...