Big Data Developer

3 weeks ago


Hyderabad, India Pixomania Full time
As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable streaming data systems.
Responsibilities:
Develop, test, and deploy distributed data processing systems using Apache Spark and Hive.
Design and implement scalable and efficient ETL pipelines for large datasets.
Optimize performance of Spark jobs for high throughput and low-latency processing.
Write clean, efficient, and maintainable code in Scala or Java.
Work with data engineers and other teams to integrate data solutions and ensure smooth data flow.
Implement and maintain Hive queries, partitions, and tables for batch processing.
Design and maintain data processing architectures in cloud environments (AWS, Azure, GCP).
Collaborate with business and technical stakeholders to deliver high-quality data solutions.
Work on Kafka (if applicable) to design and implement real-time data streaming applications.
Troubleshoot and optimize performance issues related to Spark and Hive-based systems.
Mentor junior developers and ensure high-quality standards are followed throughout the development lifecycle.
Required Skills and Experience:
5+ years of experience in Big Data development, with extensive hands-on experience in Apache Spark and Hive.
Strong experience with Spark for batch and real-time data processing.
Proven ability to design and implement Hive queries, partitions, and data models.
Good knowledge of Hadoop ecosystem tools and how they integrate with Spark.
Proficiency in SQL for querying and data manipulation.
Familiarity with cloud technologies and platforms (AWS, Azure, GCP).
Experience working with large-scale distributed systems.
Strong understanding of ETL pipelines, data integration, and data processing workflows.
Knowledge of Kafka for stream processing is a plus.
Experience in developing and optimizing high-performance data processing jobs.
Ability to work in a hybrid environment, with occasional on-site presence in Bengaluru/Hyderabad.
Preferred Skills:
Familiarity with Apache Kafka and real-time data stream processing.
Knowledge of data governance, security, and best practices for Big Data environments.
Experience with Hadoop, HBase, and other NoSQL databases.
Familiarity with Kubernetes or containerized environments.
Education:
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field, or equivalent work experience.
Big data developer with Spark and Hive; kafka is plus(5+ years)
Location - Bengaluru/ Hyderabad

  • Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Solutions Engineer to join our team in Hyderabad, India. As a key member of our engineering team, you will be responsible for designing and implementing complex data products components based on requirements with possible technical solutions.You will work closely with Data Scientists, Data Engineers,...

  • Big Data Developer

    3 weeks ago


    Hyderabad, India Pixomania Full time

    As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable...

  • Big Data Developer

    3 weeks ago


    Hyderabad, India Pixomania Full time

    As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable...

  • Big Data Developer

    3 weeks ago


    Hyderabad, India Pixomania Full time

    As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable...

  • Big Data Developer

    6 months ago


    Hyderabad, India Virtusa Full time

    Big Data Developer - CREQ189736 Description Job Description: 5+ years of hands-on experience in designing, building and supporting Data Applications using Spark, Sqoop and Hive Bachelors or masters degree in Computer Science or related field Strong knowledge of working with large data sets and high-capacity big data processing platform Strong experience...


  • Hyderabad, India @ Six Sigma Full time

    Job Details Requirement Type Permanent Job Title Opening for Big data Developer Job Level Executive - Non-Managerial Job Description Big data Developer No. of Openings 6 Job Domain IT Experience - Minimum - Maximum data Expected Date Of Joining 2023-11-16 Joining Time 5 Days Salary 54,167 Work Locations Hyderabad Required skills Big data

  • Big Data Expert

    6 days ago


    Hyderabad, Telangana, India Wipro Full time

    Job Title:Big Data ConsultantAbout the Role:We are seeking a highly skilled Big Data Consultant to join our team at Wipro. As a Big Data Consultant, you will be responsible for working on big data projects, utilizing your expertise in HDFS, SPARK, HIVE, and other related technologies.Key Responsibilities:Design and implement big data solutions using various...


  • Hyderabad, Telangana, India US Technology International Pvt Ltd Full time

    Job SummaryWe are seeking a highly skilled Senior Big Data Developer to join our team at US Technology International Pvt Ltd. As a Senior Big Data Developer, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Spark/Scala.Key Responsibilities:Design and develop scalable data processing systems using...

  • Big Data Architect

    2 months ago


    Hyderabad, India TECHSTAR SOFTWARE DEVELOPMENT INDIA PVT LTD Full time

    About the Role:The ideal candidate will be responsible for designing and developing scalable big data solutions that drive business growth and improve customer experiences.Key Responsibilities:Design and develop data pipelines for data ingestion and transformation using Java, Scala, or Python.Work with data scientists to build data pipelines using...


  • Hyderabad, India Virtusa Full time

    Lead - Big Data Developer - CREQ189749 Description Job Description: 5+ years of hands-on experience in designing, building and supporting Data Applications using Spark, Sqoop and Hive Bachelors or masters degree in Computer Science or related field Strong knowledge of working with large data sets and high-capacity big data processing platform Strong...

  • Big Data Developer

    5 days ago


    Hyderabad, India People Tech Group Inc Full time

    Hello Linkedin connectionsWe are hiring '' Big Data Developer '' and below are the requirement. This will be through ''People Tech Group Company .Role DescriptionExperience: 7yrs +Skill: Hadoop Eco System – Spark, Python, Hive, Impala, Unix, Python and OOPs programming languages (Scala preferrable),SQL,Hadoop / Spark ,Agile methodolgies,NiFi/Kafka...

  • Big data developer

    3 weeks ago


    Hyderabad, India People Tech Group Inc Full time

    Hello Linkedin connectionsWe are hiring '' Big Data Developer '' and below are the requirement. This will be through ''People Tech Group Company.Role DescriptionExperience: 3yrs To 8 yrsSkill: Hadoop Eco System – Spark, Python, Hive, Impala, Unix, Python and OOPs programming languages (Scala preferrable), SQL, Hadoop / Spark , Agile methodolgies, Ni...

  • Big Data Developer

    3 weeks ago


    Hyderabad, India People Tech Group Inc Full time

    Hello Linkedin connectionsWe are hiring '' Big Data Developer '' and below are the requirement. This will be through ''People Tech Group Company .Role DescriptionExperience: 3yrs To 8 yrsSkill: Hadoop Eco System – Spark, Python, Hive, Impala, Unix, Python and OOPs programming languages (Scala preferrable),SQL,Hadoop / Spark ,Agile methodolgies, NiFi/Kafka...

  • Big Data Engineer

    3 months ago


    Hyderabad, India NucleusTeq Full time

    Big Data Engineer Responsibilities: Big Data Engineer with 5+ years of experienceExperience in leading a team of engineers and a good attitude toward learning the domain and implementationStrong exposure and expertise in Spark (Primary), Scala/Java (Scala Primary), Airflow Orchestration and AWS.Finalizing the scope of the system and delivering...

  • Big data engineer

    6 months ago


    Hyderabad, India Virtusa Full time

    Big data engineer - CREQ189284 Description Position: Big data engineer Primary: big data concepts, aws, pyspark Location: HYD Create Trigger based automation framework for Data Migration Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same Integrate CDMS framework with Lake and Data bridge...

  • Big Data Developer

    2 weeks ago


    Hyderabad, Telangana, India People Tech Group Inc Full time

    Job Title: Big Data DeveloperWe are seeking a highly skilled Big Data Developer to join our team at People Tech Group Inc.About the Role:The successful candidate will have expertise in Hadoop Eco System, including Spark, Python, Hive, and Impala. They will also possess strong skills in scripting languages such as Unix, Python, and OOPs programming languages...


  • Hyderabad, Telangana, India People Tech Group Inc Full time

    We are seeking an experienced Big Data Developer Expert to join our team at People Tech Group Inc in Pune, Yerwada. This is a hybrid mode position that requires strong technical skills and excellent communication abilities.About the RoleThe ideal candidate will have 3-8 years of experience in Hadoop Eco System, Spark, Python, Hive, Impala, Unix, Python, OOPs...


  • hyderabad, India Pixomania Full time

    As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable...

  • Big Data Engineer

    1 month ago


    hyderabad, India UST Full time

    J Dear Candidate , Please share following details over email. To Proceed further . Nitin.patil@ust.com Total Years of Experience – Relevant Experience - Big Data Relevant Experience - hadoop Relevant Experience - Airflow Relevant Experience - Scala in Spark Relevant Experience - Hive Current CTC – Expected CTC – Current Location – Expected Location...

  • Big data engineer

    4 weeks ago


    Hyderabad, India Anblicks Full time

    Experience- 4+ years Experience in hadoop/big data technologies. Experience in spark. Experience working on Google or AWS cloud developing data solutions. Certifications preferred. Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required; Experience with containerization and related technologies (e.g....