Apache Spark Developer

2 months ago

Hyderabad, India cranium IT Full time

Notice Period : Immediate Joiner to 15 days

Job Summary :

We are looking for an experienced On-Premises Apache Spark Developer to join our team. The ideal candidate will have 6-8 years of experience in developing and managing large-scale data processing applications using Apache Spark in on-premises environments. You will play a critical role in optimizing and scaling our data infrastructure to support high-performance data processing and analytics.

Key Responsibilities :

Apache Spark Development :

- Design, develop, and maintain complex data processing applications using Apache Spark in an on- premises environment.

- Implement Spark jobs for batch and real-time data processing, ensuring efficient use of resources and scalability.

- Optimize Spark applications for performance, including tuning Spark jobs, managing resources, and debugging issues.

Data Integration & ETL :

- Develop and maintain ETL (Extract, Transform, Load) processes to integrate data from various on- premises sources.

- Ensure data quality, consistency, and reliability across all stages of the data pipeline.

- Work with large datasets to perform data wrangling, cleaning, and transformation as needed.

System & Infrastructure Management :

- Collaborate with system administrators to manage and monitor the on-premises Hadoop and Spark infrastructure.

- Ensure the stability, security, and performance of the Spark cluster, including monitoring and troubleshooting any issues.

- Participate in capacity planning and scaling of the Spark environment based on business needs.

Collaboration & Documentation :

- Work closely with data engineers, data scientists, and other stakeholders to understand data requirements and deliver efficient solutions.

- Document the architecture, design, and implementation of Spark jobs and workflows for future reference and audits.

Performance Tuning & Optimization :

- Analyze and optimize Spark job performance, including caching, partitioning, and job scheduling.

- Identify and resolve performance bottlenecks in the Spark environment.

Security & Compliance :

- Implement security best practices for managing sensitive data within the Spark ecosystem.

- Ensure compliance with organizational policies and industry regulations.

Required Qualifications :

- 6-8 years of experience in developing and managing Apache Spark applications in an on-premises environment.

- Strong proficiency in programming languages such as Java, Scala, or Python for Spark development.


- Deep understanding of Spark architecture, including RDDs, DataFrames, and SparkSQL.

- Experience with performance tuning and optimizing Spark jobs.

- Solid understanding of data warehousing concepts, data modeling, and ETL processes.

- Excellent problem-solving skills and attention to detail.

- Strong communication skills and ability to work in a team-oriented environment.

Preferred Qualifications :

- Familiarity with DevOps practices, CI/CD pipelines, and version control systems like Git.


- Knowledge of containerization and orchestration tools such as Docker and Kubernetes.

- Experience with data security and governance practices in an on-premises environment.


- Certifications in Apache Spark or related big data technologies.

(ref:hirist.tech)

  • Application Developer

    1 month ago

    Bangalore/Hyderabad, India Anlage Infotech Full time

    Job Description : As an Application Developer in the Data Engineering team, you will be responsible for designing, building, and configuring applications to meet business process and application requirements. Your typical day will involve collaborating with the team to develop innovative solutions and ensuring that the applications are aligned with the...

  • Apache Spark Data Processing Engineer

    4 weeks ago

    Hyderabad, Telangana, India cranium IT Full time

    Job Summary: We are seeking an experienced Apache Spark Developer to join our team at Cranium IT. The ideal candidate will have a deep understanding of Apache Spark architecture and will be able to design, develop, and maintain complex data processing applications using Apache Spark in an on-premises environment.Key Responsibilities:Apache Spark Development:...

  • Senior Apache Spark Developer

    4 weeks ago

    Noida/Bangalore/Chennai/Hyderabad/Coimbatore, India Huntsmen & Barons Full time

    Job Title: Senior Apache Spark DeveloperJob Description:5+ years of experience in Data Engineering with a strong focus on Apache Spark.Expertise in designing and developing big data platforms using Scala.Deep understanding of modern data processing technology stacks, including Spark, HBase, and Hive.Experience in streaming data architectures and real-time...

  • Cranium Information Technology

    2 months ago

    Hyderabad, India cranium IT Full time

    Job Summary :We are looking for an experienced On-Premises Apache Spark Developer to join our team. The ideal candidate will have 6-8 years of experience in developing and managing large-scale data processing applications using Apache Spark in on-premises environments. You will play a critical role in optimizing and scaling our data infrastructure to support...

  • Senior Data Analyst

    6 months ago

    Hyderabad, India Ivy comptech Full time

    About The Role As Senior Data Analyst will contribute to various phases and activities of the software development process or software testing. Candidate must be willing to work in a fast-paced, multi-tasking, team environment, the engineer works with tech Lead, software engineers, internal partners, and stakeholders. Key...

  • Data Engineering Lead

    2 months ago

    Hyderabad, Telangana, India upGrad Full time

    Job Title : Data Engineering Lead. Location : Hyderabad. Experience : 6-8 Years. Company Overview : UpGrad is a leader in the online education space, providing high-quality learning experiences to individuals and businesses worldwide. We are looking for a Data Engineering Lead to drive the development of data solutions that will shape the future of our data...

  • Data Engineering Lead

    2 months ago

    Hyderabad, India upGrad Full time

    Job Title : Data Engineering Lead. Location : Hyderabad. Experience : 6-8 Years. Company Overview : UpGrad is a leader in the online education space, providing high-quality learning experiences to individuals and businesses worldwide. We are looking for a Data Engineering Lead to drive the development of data solutions that will shape the future of our data...

  • Scala Developer

    1 week ago

    Hyderabad, India U-Set Consultancy Private Limited Full time

    Role : Scala Developer- Design, develop, and maintain scalable, high-performance backend services and APIs using Scala.- Work with a cross-functional team to build new features and improve existing ones.- Collaborate with data engineers to process large datasets in real-time using frameworks such as Apache Kafka, Akka, and Spark.- Implement functional...

  • Country Delight

    2 months ago

    Hyderabad, Telangana, India Country Delight Full time

    Key Responsibilities : - Design, develop, and implement data pipelines for ingesting, transforming, and loading data at scale. - Utilize Apache Spark for data processing and analysis. - Utilize AWS services (S3, Redshift, EMR, Glue) to build and manage efficient data pipelines. - Optimize data pipelines for performance and scalability, considering factors...

  • Country Delight

    2 months ago

    Hyderabad, India Country Delight Full time

    Key Responsibilities : - Design, develop, and implement data pipelines for ingesting, transforming, and loading data at scale. - Utilize Apache Spark for data processing and analysis. - Utilize AWS services (S3, Redshift, EMR, Glue) to build and manage efficient data pipelines. - Optimize data pipelines for performance and scalability, considering factors...

  • Apache Flink Developer

    2 months ago

    Chennai/Bangalore/Hyderabad, India Selsoftinc Full time

    Role : Apache FlinkExperience : 6-8 YearsLocation : Chennai, Hyderabad, BengaluruJob Description :- Ability to translate application storyboards and use cases into functional applications- Build near real time data pipeline with Apache Flink, DASK and Airflow & Python to extract data from NoSQL DBs and Load it into Neo4j Graph- Design, build, and maintain...

  • Big Data Developer

    4 weeks ago

    Hyderabad/Bangalore/Chennai/Bhubaneshwar, India Rigel networks Full time

    Job Title : Big Data/Spark Scala DeveloperLocation : Type : Full-TimeJob Overview : We are looking for an experienced Big Data/Spark Scala Developer to join our team. The ideal candidate will have a deep understanding of big data technologies and strong programming skills in Scala. In this role, you will be responsible for designing, developing, and...

  • Big Data Developer

    2 months ago

    Hyderabad/Bangalore/Chennai/Bhubaneshwar, IN Rigel networks Full time

    Job Title : Big Data/Spark Scala DeveloperLocation : Type : Full-TimeJob Overview : We are looking for an experienced Big Data/Spark Scala Developer to join our team. The ideal candidate will have a deep understanding of big data technologies and strong programming skills in Scala. In this role, you will be responsible for designing, developing, and...

  • Hadoop Administrator

    4 weeks ago

    Chennai/Bangalore/Hyderabad/Bhubaneshwar, India Rigel networks Full time

    Hadoop Administrator -- Excellent knowledge of UNIX/LINUX OS because Hadoop runs on Linux.- Knowledge of high degree configuration management and automation tools like Puppet or Chef for non-trivial installation.- Knowledge of cluster monitoring tools like Ambari, Ganglia, or Nagios.- Knowing of core java is a plus for a Hadoop admin but not mandatory.- Good...

  • Hadoop Administrator

    2 months ago

    Chennai/Bangalore/Hyderabad/Bhubaneshwar, IN Rigel networks Full time

    Hadoop Administrator -- Excellent knowledge of UNIX/LINUX OS because Hadoop runs on Linux.- Knowledge of high degree configuration management and automation tools like Puppet or Chef for non-trivial installation.- Knowledge of cluster monitoring tools like Ambari, Ganglia, or Nagios.- Knowing of core java is a plus for a Hadoop admin but not mandatory.- Good...

  • Scala Developer

    6 months ago

    Bangalore/Pune/Hyderabad, India T D Newton Full time

    Urgent requirement for Scala Developer.Experience : 4-7 YrsJob Location : Pune/Bangalore/HyderabadMust have : Scala, Spark, CI/CD, Airflow, Hadoop, SQL Job Description :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative...

  • Data Engineer

    1 month ago

    Noida/Bangalore/Chennai/Hyderabad/Coimbatore, IN Huntsmen & Barons Full time

    Role : Data Engineer - Apache spark/Spark/Scala/pysparkJob Description :- Relevant Experience 5 years.- The indent is created for Apache spark. But the actual demand is for Spark/ Scala.- Expertise in Data Engineering.- Hands-on experience in design and development of big data platform.- Deep understanding of modern data processing technology stacks: Spark,...

  • Data Engineer

    2 months ago

    Hyderabad, India DEVENIR SOFTWARE PRIVATE LIMITED Full time

    Position : Data Engineer Experience : 4 to 7 years Location : HyderabadWork from office role.Fulltime position Job Description : We are seeking a talented Data Engineer to join our dynamic team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines and infrastructure solutions. You will work closely...

  • Big Data Engineer

    2 months ago

    Hyderabad, India GAIUS HYPERLOCAL PRIVATE LIMITED Full time

    Job Description : Big Data Engineer - Scala, SparkYears of exp: 6 to 13 yearsJob location: Chennai / Hyderabad / Bangalore / Pune / Mumbai / Delhi / CoimbatoreAbout the Role :We are seeking a highly skilled Big Data Engineer to join our team and help us leverage the power of big data to drive innovation and business value. As a Big Data Engineer, you will be...

  • Expert Apache NiFi Developer

    3 weeks ago

    Hyderabad, Telangana, India Techwave Full time

    Job Title: Expert Apache NiFi DeveloperWe are seeking a skilled Apache NiFi Developer to join our team at Techwave Engineering Services. As a key member of our engineering team, you will be responsible for designing, implementing, and maintaining complex data pipelines using Apache NiFi.Key Responsibilities:* Design and implement data processing workflows...