Apache Spark Developer

3 weeks ago


Hyderabad, Telangana, India cranium IT Full time

Notice Period : Immediate Joiner to 15 days

Job Summary :

We are looking for an experienced On-Premises Apache Spark Developer to join our team. The ideal candidate will have 6-8 years of experience in developing and managing large-scale data processing applications using Apache Spark in on-premises environments. You will play a critical role in optimizing and scaling our data infrastructure to support high-performance data processing and analytics.

Key Responsibilities :

Apache Spark Development :

- Design, develop, and maintain complex data processing applications using Apache Spark in an on- premises environment.

- Implement Spark jobs for batch and real-time data processing, ensuring efficient use of resources and scalability.

- Optimize Spark applications for performance, including tuning Spark jobs, managing resources, and debugging issues.

Data Integration & ETL :

- Develop and maintain ETL (Extract, Transform, Load) processes to integrate data from various on- premises sources.

- Ensure data quality, consistency, and reliability across all stages of the data pipeline.

- Work with large datasets to perform data wrangling, cleaning, and transformation as needed.

System & Infrastructure Management :

- Collaborate with system administrators to manage and monitor the on-premises Hadoop and Spark infrastructure.

- Ensure the stability, security, and performance of the Spark cluster, including monitoring and troubleshooting any issues.

- Participate in capacity planning and scaling of the Spark environment based on business needs.

Collaboration & Documentation :

- Work closely with data engineers, data scientists, and other stakeholders to understand data requirements and deliver efficient solutions.

- Document the architecture, design, and implementation of Spark jobs and workflows for future reference and audits.

Performance Tuning & Optimization :

- Analyze and optimize Spark job performance, including caching, partitioning, and job scheduling.

- Identify and resolve performance bottlenecks in the Spark environment.

Security & Compliance :

- Implement security best practices for managing sensitive data within the Spark ecosystem.

- Ensure compliance with organizational policies and industry regulations.

Required Qualifications :

- 6-8 years of experience in developing and managing Apache Spark applications in an on-premises environment.

- Strong proficiency in programming languages such as Java, Scala, or Python for Spark development.


- Deep understanding of Spark architecture, including RDDs, DataFrames, and SparkSQL.

- Experience with performance tuning and optimizing Spark jobs.

- Solid understanding of data warehousing concepts, data modeling, and ETL processes.

- Excellent problem-solving skills and attention to detail.

- Strong communication skills and ability to work in a team-oriented environment.

Preferred Qualifications :

- Familiarity with DevOps practices, CI/CD pipelines, and version control systems like Git.


- Knowledge of containerization and orchestration tools such as Docker and Kubernetes.

- Experience with data security and governance practices in an on-premises environment.


- Certifications in Apache Spark or related big data technologies.

(ref:hirist.tech)

  • Hyderabad, Telangana, India cranium IT Full time

    Job Summary :We are looking for an experienced On-Premises Apache Spark Developer to join our team. The ideal candidate will have 6-8 years of experience in developing and managing large-scale data processing applications using Apache Spark in on-premises environments. You will play a critical role in optimizing and scaling our data infrastructure to support...

  • Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India Info Services Full time

    Responsibilities :Data Pipeline Production & Operations :- Possess a minimum of 4+ years of experience in data engineering.- Collaborate with the data engineering team to finalize the setup of our data pipeline for production. - Implement monitoring and alerting systems for the data pipeline using tools like AWS CloudWatch or similar solutions.- Establish...

  • Data Engineering Lead

    3 weeks ago


    Hyderabad, Telangana, India upGrad Full time

    Job Title : Data Engineering Lead. Location : Hyderabad. Experience : 6-8 Years. Company Overview : UpGrad is a leader in the online education space, providing high-quality learning experiences to individuals and businesses worldwide. We are looking for a Data Engineering Lead to drive the development of data solutions that will shape the future of our data...

  • Country Delight

    3 weeks ago


    Hyderabad, Telangana, India Country Delight Full time

    Key Responsibilities : - Design, develop, and implement data pipelines for ingesting, transforming, and loading data at scale. - Utilize Apache Spark for data processing and analysis. - Utilize AWS services (S3, Redshift, EMR, Glue) to build and manage efficient data pipelines. - Optimize data pipelines for performance and scalability, considering factors...

  • Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India DEVENIR SOFTWARE PRIVATE LIMITED Full time

    Position : Data EngineerExperience : 4 to 7 yearsLocation : HyderabadWork from office role.Fulltime positionJob Description : We are seeking a talented Data Engineer to join our dynamic team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines and infrastructure solutions. You will work closely with...


  • Hyderabad, Telangana, India Wipro Full time

    Wipro is seeking a skilled Apache Camel developer to join our team.Key Responsibilities:• Design and implement ESB routes using Apache Camel components.• Develop Spring Boot microservices with web service experience.• Identify ESB patterns and design based on interface requirements.• Have hands-on experience in Containers, Docker, Kubernetes, and...


  • Hyderabad, Telangana, India Wipro Full time

    Wipro is seeking a highly skilled Senior Apache Camel Developer to join our team.The ideal candidate will have a strong understanding of Apache Camel components and ESB routes, as well as experience with Spring Boot microservices and web service development.Key responsibilities will include:Designing and implementing ESB patterns based on interface...


  • Hyderabad, Telangana, India Virtusa Full time

    Job Title: Java Spark DeveloperWe are seeking a highly skilled Java Spark Developer to join our team at Virtusa. As a Java Spark Developer, you will be responsible for designing, developing, and deploying large-scale data processing systems using Java and Spark.Key Responsibilities:Design and develop scalable data processing systems using Java and...


  • Hyderabad, Telangana, India Virtusa Full time

    Job Title: Java Spark DeveloperWe are seeking a highly skilled Java Spark Developer to join our team at Virtusa. As a key member of our Big Data team, you will be responsible for designing, developing, and implementing large-scale data processing systems using Java Spark.Key Responsibilities:Design and develop scalable data processing systems using Java...


  • Hyderabad, Telangana, India Quess Corp Limited Full time

    Job Title: Middleware TAS Congnos Apache Tomcat admWe are seeking a highly skilled Middleware TAS Congnos Apache Tomcat adm to join our team at Quess Corp Limited.Key Responsibilities:Migrate and upgrade Apache Tomcat and Apache HTTP Server environments to ensure optimal performance and security.Configure, troubleshoot, and optimize the performance of web...

  • Big Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India V3 Staffing Solutions India p. Ltd Full time

    EXP - 5 TO 8 yearsJob Description :- 3+ years of experience in Data abbreviation, standardization, linking, rebooting- 3+ years of experience in Bigdata technologies like Spark, and Hadoop.- 3+ years of experience in RDBMS- Good understanding of Linux environment (shell scripting) - Knowledge- Understanding of data-warehousing and data-modeling techniques-...

  • Big Data Developer

    6 days ago


    Hyderabad, Telangana, India Tata Consultancy Services Full time

    Job Title: Big Data DeveloperGreetings from Tata Consultancy Services!Job Summary:We are seeking a highly skilled Big Data Developer to join our team. As a Big Data Developer, you will be responsible for designing and developing high-quality deliverables that meet business requirements.Key Responsibilities:Design and develop highly scalable and...


  • Hyderabad, Telangana, India Wipro Full time

    Job Title: Apache Camel Technical LeadWe are seeking a highly skilled Apache Camel Technical Lead to join our team at Wipro.About the Role:The successful candidate will have a strong understanding of Apache Camel components and their application in ESB routes. They will also have hands-on experience with Spring Boot microservices and web service...

  • Senior Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India METAFOLKS TECHNOLOGIES Full time

    Job Description- As a Senior Data Engineer, Your Responsibilities Will Include Design and build data pipelines to process terabytes of data Orchestrate in Airflow the data tasks to run on Kubernetes/Hadoop for the ingestion, processing and cleaning of data.- Create Docker images for various applications and deploy them on Kubernetes- Design and build best in...


  • Hyderabad, Telangana, India People First Consultants Full time

    Job Role: Weblogic Apache AWS AdminExperience: 5+ years of relevant experience and total not more than 12 years.We are seeking a highly skilled Weblogic Apache AWS administrator to join our team at People First Consultants. The ideal candidate will have a strong background in middleware technologies, including Weblogic, Apache, and AWS.Key...


  • Hyderabad, Telangana, India IT Full time

    Job Description:We are seeking a highly skilled Kafka Developer to join our team. The ideal candidate will have a strong background in Apache Kafka and experience in designing, building, and operating in-production Big Data, stream processing, and/or enterprise data integration solutions.Key Responsibilities:Design and develop scalable and efficient...


  • Hyderabad, Telangana, India Ninja Van Full time

    About Ninja VanNinja Van is a pioneering logistics company that is revolutionizing the industry with cutting-edge technology and innovative solutions. Founded in 2014 in Singapore, we have rapidly grown to become one of Southeast Asia's largest and fastest-growing express logistics companies. Our mission is to connect shippers and shoppers across the region...

  • Big Data Engineer

    6 days ago


    Hyderabad, Telangana, India Tata Consultancy Services Full time

    About the RoleGreetings from Tata Consultancy Services, a global leader in IT services and consulting.We are seeking a highly skilled Big Data Developer to join our team in Hyderabad or Pune.The ideal candidate will have a strong background in programming languages such as Java and Scala, with experience in Apache Spark frameworks and Unix Shell...


  • Hyderabad, Telangana, India 451 Discovery Comm. India Full time

    About the RoleWe are seeking a highly skilled Senior Software Engineer to join our Data Platform team at Warner Bros. Discovery. As a key member of our team, you will be responsible for designing, developing, and deploying scalable and efficient data pipelines using Databricks, Apache Spark, and other relevant technologies.Key ResponsibilitiesDesign and...

  • Solutions Architect

    6 days ago


    Hyderabad, Telangana, India Model N Full time

    Job Title: Solutions ArchitectModel N is seeking a highly skilled Solutions Architect to join our Business Services team. As a key member of our team, you will be responsible for designing and implementing complex algorithms for our clients, as well as serving as the primary liaison between our clients and internal teams.Responsibilities:Collaborate with...