Data Engineer

3 days ago

Bengaluru, Karnataka, India ValueLabs Full time US$ 1,00,000 - US$ 1,50,000 per year

Job Title- PySpark Data Engineer

We're growing our Data Engineering team at ValueLabs and looking for a talented individual to build scalable data pipelines on Cloudera Data Platform

Experience- 5years to 9years.

Pyspark Job Description:


• Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.


• Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.


• Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.


• Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.


• Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.


• Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.


• Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issues, and perform routine maintenance on the Cloudera Data Platform and associated data processes.


• Collaboration: Work closely with other data engineers, analysts, product managers, and other stakeholders to understand data requirements and support various data-driven initiatives.


• Documentation: Maintain thorough documentation of data engineering processes, code, and pipeline configurations.

Qualifications Education and Experience


• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.


• 5+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform. Technical Skills


• PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.


• Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.


• Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).


• Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.


• Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.


• Scripting and Automation: Strong scripting skills in Linux.

  • Data Engineer

    3 days ago

    Bengaluru, Karnataka, India Data Analytics Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Get to Know the Team Grab has been expanding into multiple lines of businesses. The Grab Data Warehouse team builds business intelligence systems using data from all business lines to provide insights to key stakeholders across the company. Get to Know the Role We are looking for a Lead Data Engineer to lead the design and enhancement of our data...

  • Data Senior Engineer

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time US$ 80,000 - US$ 1,20,000 per year

    Req ID: 337669NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Senior Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Job Duties: 8+ years of experience...

  • Data Engineer

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Req ID: 338632NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer (HRIS,Tableau and BI) to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Job Duties: Key...

  • Data Engineer

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time US$ 90,000 - US$ 1,20,000 per year

    Req ID: 338630NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer (HRIS) to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Job Duties: Key...

  • Urgent Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India NTT Data Full time

    Job DescriptionNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer - ETL to join our team in Bangalore, Karntaka (IN-KA), India (IN).Job Duties:. Migrate ETL workflows from...

  • Senior, Data Engineer

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time US$ 1,04,000 - US$ 1,30,878 per year

    Make an impact with NTT DATAJoin a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.Your day at NTT DATAThe Manager,...

  • Data Engineer

    2 days ago

    Bengaluru, Karnataka, India NTT DATA Business Solutions Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    Experience: 4 to 6 YearsLocation: Hybrid/Bangalore]Employment Type: Full-TimeRole & responsibilitiesMust-Have Skills:46 years of experience in data engineering roles.Proven experience in data migration and conversion projects (SAP or non-SAP).Strong knowledge of ETL concepts, tools, and frameworks.Proficiency in SQL, Python, or other data scripting...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA North America Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karnātaka (IN-KA), India (IN)."Job Duties: Key...

  • Digital Engineering Sr. Engineer

    3 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Req ID 268395NTT DATA Services strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Digital Engineering Sr Engineer to join our team in Bangalore Karnxc4x81taka IN-KA India IN Role...

  • High Salary) Data Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT Data Full time

    Job DescriptionNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer (HRIS) to join our team in Bangalore, Karntaka (IN-KA), India (IN).Job Duties: Key ResponsibilitiesData...