Big Data Processing Specialist

2 days ago

Chennai, Tamil Nadu, India beBeeData Full time ₹ 9,00,000 - ₹ 12,00,000
Job Title: Big Data Processing Specialist

We are seeking a highly skilled Big Data Processing Specialist with expertise in Python programming and the PySpark framework. The ideal candidate will have hands-on experience working with AWS Glue, parallel batch processing, and popular libraries such as pandas and numpy. This role requires knowledge of other AWS services like ECS, ECR, Docker, CloudWatch, and Athena.

Roles & Responsibilities:
  1. Design and develop data processing solutions using Python programming.
  2. Apply expertise with the PySpark framework for big data processing tasks.
  3. Work extensively with AWS Glue and PySpark to build and manage data pipelines.
  4. Utilize popular Python libraries such as pandas, numpy, and joblib to enhance data manipulation and processing.
  5. Conduct parallel batch processing with Python to handle large-scale data.
  6. Work with AWS Batch and Step Functions to orchestrate and manage data workflows.
  7. Use knowledge of AWS ECS, ECR, and Docker for containerization and deployment.
  8. Leverage knowledge of CloudWatch, Athena, and other AWS services for monitoring, analytics, and data management.
  9. Collaborate with data scientists and engineers to understand requirements and deliver efficient solutions.
  10. Troubleshoot and optimize data processing jobs for performance and cost efficiency.

Required Skills & Qualifications:

  • Strong working experience in Python programming.
  • Expertise with PySpark framework.
  • Hands-on experience working with AWS Glue and PySpark.
  • Strong experience with using pandas, numpy, joblib, and other popular libraries.
  • Good working experience with parallel batch processing with Python.
  • Good working experience on AWS Batch and Step Functions.
  • Good experience with AWS ECS, ECR, Docker.
  • Good knowledge on CloudWatch, Athena, and other AWS services.
  • Strong problem-solving and analytical skills.
  • Excellent communication and collaboration abilities.

Benefits:

  • Big Data Specialist

    17 hours ago

    Chennai, Tamil Nadu, India beBeeDataSpecialist Full time US$ 1,80,000 - US$ 2,00,000

    Job Title: Big Data SpecialistWe are seeking a talented professional to join our team as a Big Data Engineer.About the Role:The ideal candidate will possess strong programming skills in Python and PySpark, along with a solid understanding of HDFS, Hive, and Spark architecture.Key Responsibilities:Building scalable ETL workflows and handling large datasets is...

  • Big Data Architecture Specialist

    4 days ago

    Chennai, Tamil Nadu, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Big Data Architecture Specialist", "We are seeking an experienced Data Engineer to join our team. The successful candidate will be responsible for designing, developing, and maintaining large-scale data systems and pipelines.", "This includes collaborating with stakeholders to identify product and technical requirements.Developing and implementing...

  • Big Data Solutions Specialist

    6 days ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 18,00,000 - ₹ 30,00,000

    Job OpportunityWe are seeking a highly skilled Big Data Engineer to join our team at Virtusa.About the Role:Design, build, and support data applications using Spark, Sqoop, and Hive.Work with large data sets and high-capacity big data processing platforms.Develop strong expertise in Unix and Shell scripting.Gain in-depth knowledge of the Hadoop ecosystem and...

  • Big Data Specialist

    4 days ago

    Chennai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title:A highly skilled Data Engineer with expertise in designing, developing, and maintaining big data applications using Apache Spark with Scala.Key ResponsibilitiesDesign and develop efficient data processing workflows using Apache Oozie to optimize data pipelines for performance, scalability, and reliability.Implement and monitor data flows using...

  • Data Engineer

    5 days ago

    Chennai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Data Engineer - Big Data SpecialistWe are seeking a skilled Data Engineer to join our team. In this role, you will collaborate with data science teams, build scalable data pipelines, and develop models that empower business decision-making through advanced analytics.Key Responsibilities:Collaborate with data science teams to implement complex algorithms and...

  • Big Data Engineer Position

    1 week ago

    Chennai, Tamil Nadu, India beBeeDataArchitecture Full time

    Job Title: Data Architecture SpecialistWe are seeking a skilled Data Architecture Specialist to join our team.Key Responsibilities:Design and develop large-scale data processing workflows using Hadoop components such as HDFS, MapReduce, Hive, Pig, and HBase.Built and maintain ETL pipelines to ingest and transform data from various sources into Hadoop...

  • Senior Big Data Python Developer

    4 days ago

    Chennai, Tamil Nadu, India Innovatz Global Full time US$ 90,000 - US$ 1,20,000 per year

    Company DescriptionInnovatz is a Global Management Consulting, Technology Services, and Business Process Outsourcing Company headquartered in Kuala Lumpur, Malaysia. With a strong presence in America, China, India, Australia, Korea, Hong Kong, and various other countries, Innovatz has a skilled workforce of over 500, including Subject Matter Experts, Project...

  • Big Data Systems Specialist

    6 days ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Overview:The Data Engineer will be responsible for designing, developing and optimizing data pipelines and workflows for large scale data systems.Key Responsibilities:Design and implement scalable data pipelinesDevelop and maintain high-quality data processing systemsCollaborate with cross-functional teams to integrate data...

  • Big Data Developer

    1 week ago

    Chennai, Tamil Nadu, India beBeeBigData Full time ₹ 9,00,000 - ₹ 12,00,000

    We're seeking experienced Big Data Developers with strong Python skills to contribute to our team's success.Key Responsibilities:Design and develop scalable data pipelines using PySpark and Python.Collaborate with cross-functional teams to implement data processing solutions.Develop expertise in Hadoop, Hive, Spark, and other Big Data technologies.Implement...

  • Big Data Development Specialist

    5 days ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title:Leading Data EngineerWe are seeking a seasoned data engineer to spearhead our big data initiatives.Key Responsibilities:Design, develop, and maintain scalable big data solutions using Hadoop ecosystem, Spark, and other distributed frameworks.Build and optimize data pipelines for batch and real-time data processing.Collaborate with data scientists,...