Spark & Hadoop Developer - Distributed Systems
2 weeks ago
Key Responsibilities : - Design, develop, and maintain large-scale distributed data processing systems using Apache Spark and Apache Hadoop ecosystem tools (HDFS, Hive, HBase, Pig, etc.). - Build data pipelines to ingest, process, and analyze structured and unstructured data from diverse sources. - Work with data scientists, analysts, and business teams to understand data requirements and translate them into technical solutions. - Optimize Spark applications and Hadoop jobs for performance, scalability, and reliability. - Ensure the successful integration of Spark applications with Hadoop cluster management tools like YARN or Mesos. - Implement data processing solutions in a distributed computing environment, ensuring data consistency and fault tolerance. - Develop and maintain workflows for scheduling, monitoring, and managing large-scale batch and real-time data processing tasks. - Design and implement data models, storage structures, and ETL processes to support advanced analytics. - Troubleshoot and debug data processing issues across distributed systems. - Participate in code reviews, mentoring junior developers, and establishing best practices. - Stay up to date with the latest developments in the Big Data ecosystem and contribute to the evolution of our data infrastructure. Required Skills and Experience : - 6 years of experience in data engineering, with at least 3 years of experience in Apache Spark and Hadoop ecosystem technologies. - Strong proficiency in Java, Scala, and/or Python for building data processing applications. - Solid experience with Apache Spark for batch and real-time data processing. - Deep understanding of Hadoop ecosystem components such as HDFS, MapReduce, Hive, Pig, HBase, and others. - Expertise in working with distributed systems and large-scale data processing frameworks. - Strong knowledge of SQL and relational databases, and experience with NoSQL databases (HBase, Cassandra, etc.). - Familiarity with streaming frameworks like Apache Kafka, Apache Flink, or Spark Streaming for real-time data processing. - Experience with cloud platforms such as AWS, Azure, or Google Cloud for big data workloads. - Knowledge of data governance, data quality, and security best practices in big data environments. - Experience with ETL processes and data integration techniques. - Strong problem-solving and debugging skills in distributed systems. - Familiarity with containerization technologies such as Docker and orchestration tools like Kubernetes. - Excellent communication skills and ability to work in an Agile team environment. (ref:hirist.tech)
-
Hadoop Administrator
2 weeks ago
Hyderabad, Telangana, India Incedo Inc. Full timeRole : Hadoop Administrator Location: Hyderabad/Chennai Experience: 3+ years Role Description: Cloud/Hadoop Administrator The ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated...
-
Hadoop Developer
4 days ago
Hyderabad, Telangana, India longfinch Technologies Full timeWe are seeking a skilled Hadoop Developer to design, develop, and optimize big data solutions. The ideal candidate will work with large datasets, build scalable data pipelines, and implement high-performance data processing solutions using Hadoop and related technologies.Key Responsibilities :Develop, implement, and maintain Hadoop-based data...
-
Hadoop Administrator
2 weeks ago
Hyderabad, Telangana, India Incedo Inc. Full timeRole : Hadoop AdministratorLocation: Hyderabad/ChennaiExperience: 3+ yearsRole Description: Cloud/Hadoop AdministratorThe ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated...
-
Hadoop Administrator
1 week ago
Hyderabad, Telangana, India Incedo Inc. Full timeRole : Hadoop AdministratorLocation: Hyderabad/ChennaiExperience: 3+ yearsRole Description: Cloud/Hadoop AdministratorThe ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated...
-
Hadoop Developer
3 weeks ago
Hyderabad, Telangana, India Wipro Limited Full timeJob DescriptionDescriptionWe are seeking a highly skilled Hadoop Developer with 2-5 years of experience to join our team. The ideal candidate will have a strong understanding of Hadoop ecosystem and its components, and possess experience with data ingestion, processing, and analysis.Responsibilities- Design and develop scalable and reliable Hadoop...
-
Hadoop Administrator
1 week ago
Hyderabad, Telangana, India Incedo Inc. Full timeJob Summary: As a Hadoop Administrator at our company, you will play a crucial role in ensuring the smooth operation of our big data ecosystem. The ideal candidate should have 3+ years of experience in developing scalable applications and platforms using Hadoop technology. Key Responsibilities: Perform BigData Admin activities such as environment setup,...
-
Senior Hadoop Developer
4 days ago
Hyderabad, Telangana, India Tata Consultancy Services Full timeAbout the Job:We are looking for a Senior Hadoop Developer to join our team in Hyderabad. As a senior member of our big data solutions team, you will be responsible for designing, developing, and deploying complex data processing solutions using Hadoop/Spark.Key Responsibilities:Develop and implement complex data processing workflows using Hadoop/SparkDesign...
-
Java Spark Lead Developer
4 days ago
Hyderabad, Telangana, India People Prime Worldwide Full timeJava Spark LeadExperience : 8-15YearsNotice Period: Immediate JoinersLocation: PAN IndiaClient: CMMI Level 5Budget : 1.5 to 2L / Month (Including PF & TDS)Job DescriptionStrong hands-on Experience in Spark with Java with minimum 4+ years relevant experience in Java spark.Advanced experience in ETL and data wrangling using language Java is a plusProficiency...
-
Cloud Data Engineer
1 week ago
Hyderabad, Telangana, India SPG Consulting Solutions Pvt Full timeRole : Cloud Data Engineer Experience : 5-8 years Location : Hyderabad Technical Skills : Java, Spark Streaming, Spark Batch, Hadoop, AWS Key Responsibilities : - Design, develop, and implement large-scale data processing solutions using Java, Spark Streaming, Spark Batch, and Hadoop. - Build and maintain real-time and batch processing data pipelines,...
-
Java Spark Lead Developer
5 days ago
Hyderabad, Telangana, India People Prime Worldwide Full timeJava Spark LeadExperience : 8-15YearsNotice Period: Immediate JoinersLocation: PAN IndiaClient: CMMI Level 5Budget : 1.5 to 2L / Month (Including PF & TDS)Job DescriptionStrong hands-on Experience in Spark with Java with minimum 4+ years relevant experience in Java spark.Advanced experience in ETL and data wrangling using language Java is a plusProficiency...
-
Java Spark Lead Developer
3 days ago
Hyderabad, Telangana, India People Prime Worldwide Full timeJava Spark Lead Experience: 8-15 years of experience in Java Spark with a strong background in data wrangling and ETL processes. Notice Period: Immediate joiners are preferred, allowing for a seamless transition to the team. Location: PAN India, with opportunities for remote work and collaboration across various locations. Client: A CMMI Level 5 client,...
-
Java Spark Lead Developer
2 days ago
Hyderabad, Telangana, India People Prime Worldwide Full timeJava Spark Lead Experience: 8-15 years of experience in Java Spark with a strong background in data wrangling and ETL processes. Notice Period: Immediate joiners are preferred, allowing for a seamless transition to the team. Location: PAN India, with opportunities for remote work and collaboration across various locations. Client: A CMMI Level...
-
Java Developer
2 weeks ago
Hyderabad, Telangana, India Bharat Head Hunters Full timeJob Title : Java Developer with Hadoop, Spark, and AWSLocation : HyderabadRequired Skills & Qualifications :- Strong programming skills in Java.- Hands-on experience with Hadoop ecosystem (HDFS, MapReduce, Hive, HBase, etc.).- Proficiency in Apache Spark (Core, SQL, Streaming, and MLlib).- Experience with AWS cloud services (EMR, S3, Lambda, EC2, etc.).-...
-
Java Developer
2 weeks ago
Hyderabad, Telangana, India Bharat Head Hunters Full timeJob Title : Java Developer with Hadoop, Spark, and AWSLocation : HyderabadRequired Skills & Qualifications :- Strong programming skills in Java.- Hands-on experience with Hadoop ecosystem (HDFS, MapReduce, Hive, HBase, etc.).- Proficiency in Apache Spark (Core, SQL, Streaming, and MLlib).- Experience with AWS cloud services (EMR, S3, Lambda, EC2, etc.).-...
-
AWS Data Engineer
2 weeks ago
Hyderabad, Telangana, India Tech Mahindra Full timeWe are looking for 4 to 8 Years of experience.AWS expertise (EMR, EC2, Athena, Glue)Tech Skills: Java, Spark Streaming and Batch, Hadoop, AWSDesign, develop, and implement large-scale data processing solutions using Java, Spark Streaming, Spark Batch, and Hadoop.Build and maintain data pipelines for real-time and batch processing, ensuring high performance...
-
Cloud Hadoop Professional
2 days ago
Hyderabad, Telangana, India Tata Consultancy Services Full timeAbout the RoleThis is an exciting opportunity for a talented professional to work on large-scale big data projects, leveraging skills in Hadoop, Spark, and cloud-based architectures.
-
Distributed Systems Developer
3 days ago
Hyderabad, Telangana, India Phenom People Full timeJob DescriptionWe are seeking an experienced Distributed Systems Developer to join our team. As a key member of our engineering team, you will be responsible for architecting and designing business-critical, large-scale distributed systems that are scalable, secure, and maintainable.Architect and design complex distributed systems that meet business...
-
Senior Backend Developer
1 day ago
Hyderabad, Telangana, India Tata Consultancy Services Full timeWe are seeking an experienced Senior Backend Developer to join our team at Tata Consultancy Services in Singapore. As a key member of our high-performance, globally scaled platform development team, you will contribute to the conception, building, and operation of highly distributed systems.Key Responsibilities:Design and develop scalable, high-performance...
-
Distributed Systems Developer
4 days ago
Hyderabad, Telangana, India NetSuite Full timeAre you a skilled Senior Software Developer looking for a new challenge? Do you have expertise in distributed systems, OS, and networking skills? We are seeking an exceptional developer to join our team and contribute to the development of our cloud-based observability platform.About NetSuiteNetSuite is a leading provider of cloud-based enterprise resource...
-
Distributed Systems Developer
4 days ago
Hyderabad, Telangana, India techjays Full timeDeveloping Scalable SolutionsWe're seeking an experienced .NET Developer to join our team in building and maintaining high-performance, scalable distributed systems. The ideal candidate will have extensive experience in modern cloud-native development practices and a proven track record of delivering robust enterprise solutions.Key ResponsibilitiesDesign and...