
Spark ( Pyspark ) Developer
2 weeks ago
- The developer must have sound knowledge in Apache Spark and Python programming.
- Deep experience in developing data processing tasks using pySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.
- Experience in deployment and operationalizing the code is added advantage Have knowledge and skills in Devops/version control and containerization. Preferable having deployment knowledge.
- Create Spark jobs for data transformation and aggregation Produce unit tests for Spark transformations and helper methods
- Write Scaladoc-style documentation with all code
- Design data processing pipelines to perform batch and Real- time/stream analytics on structured and unstructured data
- Spark query tuning and performance optimization Good understanding of different file formats (ORC, Parquet, AVRO) to optimize queries/processing and compression techniques.
- SQL database integration (Microsoft, Oracle, Postgres, and/or MySQL)
- Experience working with (HDFS, S3, Cassandra, and/or DynamoDB)
- Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)
- Experience in building cloud scalable high-performance data lake solutions
- Hands on expertise in cloud services like AWS, and/or Microsoft Azure.
Required Skills
- Hive
- Spark
- SQL
-
Data Engineer
7 hours ago
Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 1,04,000 - ₹ 1,30,878Job Title: Data Engineer (Spark & PySpark)About the RoleWe are seeking a highly skilled and experienced Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Apache Spark and Python.Your primary focus will be on developing and...
-
Urgent) Azure, Spark,PySpark,Scala
3 weeks ago
Bengaluru, Karnataka, India Infosys Limited Full timeJob DescriptionJob Description:- Azure Spark PySpark Scala- Azure Synapse- Azure knowledge mustKey Responsibilities:- Engage with Business Stakeholders with status update on progress of development and issue fix- Setup Standards related to Coding Pipelines and Documentation- Understand the requirements from the business and translate it into an appropriate...
-
Experienced PySpark Engineer
3 days ago
Bengaluru, Karnataka, India beBeePySparkEngineer Full time ₹ 9,00,000 - ₹ 12,00,000Job Title: Experienced PySpark EngineerJob Summary:Our organization is seeking an experienced and highly skilled PySpark engineer to join our team. As a key member of the data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using PySpark.Key Responsibilities:Design and implement complex...
-
Python/pyspark
3 weeks ago
Bengaluru, Karnataka, India CIEL HR Full timeHiGreetings of the Day We have a critical opening with our client for the SAP ABAP Development for Python Pyspark for Bangalore Location Experience 8-12 years - Work Form Office Payroll Company Name CIEL Mode of Employment ContractNotice Period Immediate 15 days 30 days MaxJob descriptionBig data technologies Hadoop data lakes SQL scripting...
-
SDE-Python/Pyspark
3 weeks ago
Bengaluru, Karnataka, India VidPro Consultancy Services Full timeJob DescriptionExperience: 4 - 7 YearsLocation: Whitefield - BangaloreWork Mode: HybridMandatory Skills: Python, PySpark, AWS, SQL, DSA, Databricks,Hadoop,spark,mapreduce,HBase.Responsibilities: Good development practices- Hands on coder with good experience in programming languages likePython or Pyspark .- Hands-on experience on the Big Data stack like...
-
PySpark Scala Developer
2 weeks ago
Bengaluru, Karnataka, India Talent Worx Full timeRole: Spark Scala DeveloperExperience: 3–8 yearsLocation: BengaluruEmployment Type: Full-time⸻What We're Looking ForWe're hiring a Spark Scala Developer who has real-world experience working in Big Data environments, both on-prem and/or in the cloud. You should know how to write production-grade Spark applications, fine-tune performance, and work...
-
Java Developer(with Spark SQL)
2 days ago
Bengaluru, Karnataka, India Affine Full timeExperience - 4 - 9 yearsWork Location: India - Remote, Bengaluru will be preferred.Work Timings: 1:00pm to 10:00pm ISTWe are seeking experienced Java Developers with strong Spark SQL skills to join a fast-paced project for a global travel technology client. The role focuses on building API integrations to connect with external data vendors and creating...
-
Java Developer(with Spark SQL)
4 days ago
Bengaluru, Karnataka, India Affine Full timeExperience yearsWork Location: India - Remote, Bengaluru will be preferred.Work Timings: 1:00pm to 10:00pm ISTWe are seeking experienced Java Developers with strong Spark SQL skills to join a fast-paced project for a global travel technology client. The role focuses on building API integrations to connect with external data vendors and creating...
-
Pyspark Architect
3 days ago
Bengaluru, Karnataka, India Talent Worx Full time ₹ 15,00,000 - ₹ 20,00,000 per yearJob Description: PySpark ArchitectLocation: Bangalore,Pune,Kolkata Experience Required: 7 to 13 YearsRole Overview:We are seeking a highly skilled and motivated PySpark Manager to lead our dataengineering initiatives. The ideal candidate will have deep expertise in PySpark,Hadoop, AWS, Teradata, Scala, and data visualization tools. You will be responsible...
-
Mainly Scala+spark
3 weeks ago
Bengaluru, Karnataka, India TapTalent Full timePrimary Skills Mainly Scala Spark OR Java SparkNotice period immediate to 30 days Above 45 days notice period do not share profiles Location Pune Chennai Bangalore Hyderabad Mumbai GurgoanExp - 4-10 yrs Mainly Scala Spark OR Java Spark experience needed 5 years of experience Expertise in Data Engineering Hands-on experience in design and development...