Apache Spark Expert
7 days ago
Position Title:
- Apache Spark Expert**Location**:
- Pune, MH, OnsitePosition Type:
- Full-TimeInterview Process:
- Webcam/SkypeReq Id:
- 3314Salary/Rate:
- MarketPosition Duration:
- Permanent**Key Responsibilities**:
- Provide architectural consultancy to clients and develop implementation roadmaps for Apache Spark and Kafka solutions.
- Translate architectural strategies into code and facilitate knowledge transfer to junior developers.
- Demonstrate in-depth knowledge of AWS services, particularly MSK (Managed Streaming for Kafka), Spark EMR, and working with various data sources, sinks, and transformations.
- Proficiency in utilizing S3, Athena, Glue, and Step Functions for efficient data processing and analysis.
- Experience with Kinesis, Firehose, and the Kafka Connect Framework, including the development of custom connectors.
**Qualifications and Skills**:
- Strong expertise in Apache Spark and Kafka with a focus on architectural understanding.
- Proven hands-on experience in developing and implementing solutions using Apache Spark and Kafka Streaming.
- Sound knowledge of AWS services, including MSK, Spark EMR, S3, Athena, Glue, Step Functions, Kinesis, and Firehose.
- Proficiency in the Kafka Connect Framework, especially in the development of custom connectors.
**Preferred**:
- Familiarity with Apache Airflow is considered a plus.
- Experience in (Real-time) Data analytics is also welcome.
-
Senior Cloudera Developer
1 week ago
Pune, India Deutsche Telekom Full timeYour Role :We are looking for a Senior Cloudera Developer (Data Engineer) with extensive experience in Spark, the Hadoop ecosystem, and GCP services to design and implement data :- Design, develop, and implement data solutions using Cloudera technologies such as Hadoop, Spark, and Hive.- Collaborate with data engineers to optimize data pipelines and data...
-
Pune, India Coditas Technologies Pvt. Ltd. Full timeWe are looking for Senior data engineers who have the right attitude, aptitude, skills, empathy, compassion, and hunger for learning.Build products in the data analytics space.Roles and Responsibilities :- Develop and manage robust ETL pipelines using Apache Spark (Scala).- Understand Spark concepts, performance optimization techniques, and governance...
-
Scala/Spark Developer
3 days ago
Bengaluru, Chennai, Pune, India Photon Full time ₹ 10,00,000 - ₹ 25,00,000 per yearAbout The Role Develop, test, and deploy data processing applications using Apache Spark and Scala.Optimize and tune Spark applications for better performance on large-scale data sets.Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions.Collaborate with data scientists, business...
-
Data Streaming
1 week ago
Pune, India ACE Talent Consulting Full timeJob Description: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. **Responsibilities**: Create Scala/Spark/Pyspark jobs for data transformation and aggregation Produce...
-
Data Streaming
1 week ago
Pune, India ACE Talent Consulting Full timeob Description: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. **Responsibilities**: Create Scala/Spark/Pyspark jobs for data transformation and aggregation Produce...
-
Tcs data Streaming
5 days ago
Pashan, Pune, Maharashtra, India Response Informatics Full timeJOB DESCRIPTION: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. SKILLS: Pyspark Scala (with a focus on the functional programming paradigm) Apache Spark 2.x, 3.x -...
-
Java Spark
3 days ago
Pune, India Citi Full timeRequirements - 6+ years of progressive experience in development with Java/J2EE technologies. - Expertise in development with Apache Spark streaming and batch framework - Proficiency in programming SparkSQL, Java for Spark. - Experience with Kafka / high performance streaming frameworks - Should have experience working with Hadoop ecosystem components -...
-
Lead Data Engineer
1 week ago
Pune, India CliqHR Full timeWe are seeking an experienced Lead Data Engineer to design, build, and optimize large-scale data solutions. This role involves working with modern cloud and big data technologies to enable data-driven decision-making, while leading initiatives that ensure scalability, reliability, and efficiency.Key Responsibilities :- Lead the design and deployment of...
-
Spark/Scala Developer
3 weeks ago
Pune, India Cyanous Software Private Limited Full timeDetailed JD (Roles and Responsibilities) :- At least 8+ years of experience and strong knowledge in Scala programming language.- Able to write clean, maintainable and efficient Scala code following best practices.- Good knowledge on the fundamental Data Structures and their usage - At least 8+ years of experience in designing and developing large scale,...
-
Spark Admin
2 weeks ago
Pune, Maharashtra, India Virtusa Full timeSpark Administration: Install, configure, and manage Apache Spark 3.x. clusters. Monitor and optimize Spark cluster performance. Troubleshoot and resolve Spark-related issues. Manage Spark job scheduling and resource allocation. Ensure high availability and reliability of Spark clusters. DevOps Practices: Implement and manage CI/CD pipelines using tools like...