Data Engineering Expert

5 days ago

Bengaluru, Karnataka, India beBeeData Full time US$ 1,50,000 - US$ 2,00,000

Job Summary:

We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization.

This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques. The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices.

Key Responsibilities
  • Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
  • Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
  • Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
  • Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
  • Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
  • Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
  • Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issues, and perform routine maintenance on the Cloudera Data Platform and associated data processes.
Required Skills and Qualifications
  • PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
  • Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
  • Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
  • Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
  • Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
  • Scripting and Automation: Strong scripting skills in Linux.

Benefits:

The successful candidate will have opportunities to work closely with cross-functional teams, drive impactful business insights, and contribute to the development of innovative solutions.

About This Role:

This is an excellent opportunity to grow your career as a Data Engineer with a leading organization. If you are a motivated and detail-oriented professional with a passion for data engineering, we encourage you to apply.

  • Data Engineering Expert

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Engineering ExpertJob Title:Data Engineering ExpertRole Overview:We are seeking a Data Engineering Expert to design and implement high-performance data ingestion and transformation pipelines using Azure Databricks and Apache Spark.

  • Data Engineering Expert

    1 day ago

    Bengaluru, Karnataka, India beBeedataEngineering Full time ₹ 1,00,00,000 - ₹ 2,50,00,000

    Data Engineering ExpertWe are seeking an experienced data engineer to join our team. The successful candidate will be responsible for designing and implementing data engineering solutions using AWS services.

  • Data Engineering Expert

    5 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,80,00,000 - ₹ 2,00,00,000

    Data Engineering ExpertWe are seeking a skilled Data Engineer to join our team. In this role, you will be responsible for designing, developing, and maintaining large-scale data systems.

  • Data Engineering Expert

    1 week ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title: Data Engineering ExpertWe are seeking an experienced Data Engineering Expert to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data systems.Key Responsibilities:Design and develop scalable data pipelines using technologies like Kafka, Flink, and...

  • Data Engineering Expert

    7 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 9,00,000 - ₹ 12,00,000

    Data Engineering ExpertWe are seeking a highly skilled Data Engineering Expert to join our team.Job DescriptionThe successful candidate will be responsible for designing, implementing, and maintaining large-scale data systems that meet the needs of our business. This includes developing and deploying data pipelines, ensuring data quality, and optimizing...

  • Data Senior Engineer

    2 days ago

    Bengaluru, Karnataka, India NTT DATA Full time US$ 80,000 - US$ 1,20,000 per year

    Req ID: 337669NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Senior Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Job Duties: 8+ years of experience...

  • Expert Data Engineer

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity: Data Engineering ExpertWe are seeking a skilled data engineering expert with expertise in Python and PySpark to design, develop, and optimize large-scale data pipelines.Key Responsibilities:Design, develop, and maintain ETL/ELT pipelines for structured and unstructured data.Develop and optimize PySpark jobs for large-scale data...

  • Urgent Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India NTT Data Full time

    Job DescriptionNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer - ETL to join our team in Bangalore, Karntaka (IN-KA), India (IN).Job Duties:. Migrate ETL workflows from...

  • Data Engineer

    2 days ago

    Bengaluru, Karnataka, India Data Analytics Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Get to Know the Team Grab has been expanding into multiple lines of businesses. The Grab Data Warehouse team builds business intelligence systems using data from all business lines to provide insights to key stakeholders across the company. Get to Know the Role We are looking for a Lead Data Engineer to lead the design and enhancement of our data...

  • Data Engineering Expert

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineering ExpertJob OverviewWe're seeking a seasoned Data Engineer to design and build scalable data assets. As a key member of our team, you will be responsible for integrating diverse data from internal and external sources.Key Responsibilities:Collaborate with the team to understand business objectives and deliver quality data solutions within...