Pyspark Developer

1 day ago


Noida, Uttar Pradesh, India Zigsaw Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Roles & Responsibilities:
Design development and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR
Writing reusable testable and efficient code
Integration of data storage solutions in spark & ndash especially with AWS S3 object storage.
Performance tuning of pySpark scripts.
Need to ensure overall build delivery quality is good and on time delivery is done at all times.
Should be able to handle meetings with customers with ease.
Need to have excellent communication skills to interact with customer.Education/Experience
Technical/Functional Skills (Mandatory skills):
5+ years of experience in programming with python. Strong proficiency in python
Familiarity with functional programming concepts
3+ years of Hands on experience in developing ETL data pipelines using pyspark on AWS EMR
Hands-on experience of JSON processing using python
Good understanding of Spark& rsquo s RDD API
Good understanding of Spark& rsquo s Dataframe and API
Experience in configuring EMR clusters on AWS
Experience and good understanding of Apache Spark Data sources API.
Experience of dealing with AWS S3 object storage from Spark.
Experience in troubleshooting spark jobs. Knowledge of monitoring Spark jobs using Spark UI
Performance tuning of Spark jobs.
Understanding fundamental design principles behind a business processes
 
Nice to have skills:
Knowledge of AWS SDK CLI
Experience of setting up continuous integration/deployment of spark jobs to EMR clusters
Knowledge of scheduling spark applications in AWS EMR cluster.
Understanding the differences between Hadoop MapReduce and Apache Spark
Proficient understanding of code versioning tools as Git SVN
Experience minimum 4-7years


  • Developer - PySpark

    3 days ago


    Noida, Uttar Pradesh, India Compunnel Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for skilled PySparkDevelopers to design and develop scalable data processing solutions. Therole involves working with big data platforms, building ETL pipelines,and collaborating with cross-functional teams to ensure data availability,performance, and quality.Qualification:BE / B. TechLocation:Pune,Key ResponsibilitiesDesign, develop, and...


  • Noida, Uttar Pradesh, India Zigsaw Full time ₹ 1,00,00,000 - ₹ 2,00,00,000 per year

    Roles & Responsibilities: Design, development and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR Writing reusable, testable, and efficient code Integration of data storage solutions in spark – especially with AWS S3 object storage. Performance tuning of pySpark scripts. Need to ensure overall build delivery...


  • Noida, Uttar Pradesh, India NTT DATA North America Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Business Consulting-Technical analyst with ETL,GCP using Pyspark to join our team in Noida, Uttar Pradesh (IN-UP), India...

  • Data Engineer

    2 weeks ago


    Noida, Uttar Pradesh, India Source-Right Full time ₹ 5,00,000 - ₹ 12,00,000 per year

    Position: Data Engineer (AWS QuickSight, Glue, PySpark) (Noida) (CE46SF RM 3386)Education Required: Bachelor's / Masters / PhD:Bachelor's or master's in computer science, Statistics, Mathematics, Data Science, EngineeringAWS certification (e.g., AWS Certified Data Analytics – Specialty, AWS Certified Developer)Must Have SkillsProficiency in AWS cloud...


  • Noida, Uttar Pradesh, India Topcoder Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Primary skill: Azure Databricks, SQL, PySparkAdditional skills: Power BIDetailed JD: -Extensive experience in designing and building data pipelines within an Azure Databricks Lakehouse environment.Strong understanding and practical experience with Spark and other big data technologies.Integrated spark jobs in Databricks to read and write structured, batch...


  • Noida, Uttar Pradesh, India EXL Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    DescriptionJob DescriptionJob Title: Data Engineer - PySpark, Python, SQL, Git, AWS Services – Glue, Lambda, Step Functions, S3, Athena. Job Description:We are seeking a talented Data Engineer with expertise in PySpark, Python, SQL, Git, and AWS to join our dynamic team. The ideal candidate will have a strong background in data engineering, data...


  • Noida, Uttar Pradesh, India Product Company Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    We're Hiring: Senior ETL/Automation Test lead in Pharmacovigilance ITWe are looking for anexperienced Senior ETL/Automation TesterKey Requirements:Developer Skills– Strong experience in writing SQL codeETL Testing Expertise– Ability to design test cases, scenarios, and scripts; execute and analyze results; and contribute to analysis, design,...

  • Associate- BIM

    1 week ago


    Noida, Uttar Pradesh, India Axtria - Ingenious Insights Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Position SummaryTo be a driven business analyst who can work on complex Analytical problems and help the customer in better business decision making especially in the area of pharma/life sciences (domain).Job ResponsibilitiesEngage with Client to participate in requirement gathering, Status update on work, UAT and be the key partner in the overall...

  • Data Engineer

    1 week ago


    Noida, Uttar Pradesh, India EICE Technology Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Role Overview:We are looking for a highly skilledSenior Data Engineer 7+ years of experience as a Microsoft Fabric Specialistto lead the design, development, and optimization of modern data solutions. The role involves working extensively withAzure Fabric, ADF, Databricks, SSIS, Power BI, and PySparkto build scalable, high-performance data pipelines and...

  • Senior Data Engineer

    2 weeks ago


    Noida, Uttar Pradesh, India Eice Technology Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Job Title: Senior Data Engineer, Microsoft Fabric + SSIS SpecialistLocation: Noida, Sector 62 (Onsite, Full-Time, Long-Term Project)Experience Required: 7 + YearsAvailability: Immediate Joiners OnlyRole Overview:We are looking for a highly skilled Senior Data Engineer 7+ years of experience as a Microsoft Fabric Specialist to lead the design, development,...