PySpark/Databricks Engineer

4 months ago

Pune, India KPI Partners Full time

We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims to build a data standardized and curation-based Hadoop cluster. This high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.

Key Responsibilities:

  • Ability to design, build and unit test applications on Spark framework on Python.
  • Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
  • Develop and execute data pipeline testing processes and validate business rules and policies.
  • Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
  • Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
  • Ability to design & build real-time applications using Apache Kafka & Spark Streaming
  • Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
  • Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation
  • Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
  • Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
  • Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
  • Work collaboratively with onsite and offshore team.
  • Develop & review technical documentation for artifacts delivered.
  • Ability to solve complex data-driven scenarios and triage towards defects and production issues
  • Ability to learn-unlearn-relearn concepts with an open and analytical mindset
  • Participate in code release and production deployment.
  • Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

  • Databricks Engineer

    2 months ago

    Pune, India HNM Solutions Full time

    Role : DatabricksLocation : pune onlyExperience : 2 to 4 yearsNotice Period : Immediate joinersJob Description :A Data Engineer understands the client's requirements and develops and delivers data engineering solutions as per the scope. The role requires good skills in the development of solutions using various services required for data architecture on...

  • Senior Azure Data Engineer

    3 weeks ago

    Pune, India Techno Wise Full time

    Job Description :1. Design, develop, and maintain scalable data pipelines and ETL processes using Databricks and PySpark.2. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement data solutions that align with business needs.3. Optimize and tune existing data pipelines for performance and...

  • Data Engineer

    4 months ago

    Pune, India EDGESOFT Full time

    Job Description :The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.Responsibilities :- Design,...

  • Python Developer

    4 months ago

    Pune, India IT Full time

    Total Yrs. of Experience : 8+ Relevant Yrs of experience.Roles and Responsibilities :- 5+ years of hands-on Python development experience with excellent programming skills - 3+ years of experience with cloud-based platform, ideally Microsoft Azure - Working experience and skills on big data technologies such as PySpark, Databricks, Kafka - Strong solution...

  • PySpark/Databricks Engineers

    2 weeks ago

    pune, India KPI Partners Full time

    Location: Bangalore / Hyderabad / PuneJob Type: Full-time  Introduction: We are seeking a highly skilled PySpark Engineer with over 6 years of experience in big data processing, particularly with a strong background in Python, Spark and SQL. As part of our dynamic team, you will play a crucial role in designing and developing scalable data pipelines and...

  • PySpark/Databricks Engineers

    1 month ago

    Pune, India KPI Partners Full time

    Location: Bangalore / Hyderabad / PuneJob Type: Full-time Introduction:We are seeking a highly skilled PySpark Engineer with over 6 years of experience in big data processing, particularly with a strong background in Python, Spark and SQL. As part of our dynamic team, you will play a crucial role in designing and developing scalable data pipelines and...

  • Databricks Developer

    2 months ago

    Pune, India Tata Technologies Full time

    Bachelor’s degree in Computer Science, Engineering, or related field.· 6+ years of Overall Experience and 4+ years of hands-on experience as a Databricks· Strong proficiency in Apache Spark and Databricks.· Strong knowledge on PySpark· Experience with Scala and/or Python programming languages.· Solid understanding of data warehousing concepts and ETL...

  • Databricks Developer

    2 months ago

    Pune, India Tata Technologies Full time

    Bachelor’s degree in Computer Science, Engineering, or related field.· 6+ years of Overall Experience and 4+ years of hands-on experience as a Databricks· Strong proficiency in Apache Spark and Databricks.· Strong knowledge on PySpark· Experience with Scala and/or Python programming languages.· Solid understanding of data warehousing concepts and ETL...

  • Big Data Engineer

    2 months ago

    Pune, India Techno Wise Full time

    Position : Big Data EngineerRelevant Experience : 5+ yearsLocation : Navi Mumbai /Bengaluru/PuneNotice Period : Immediate or serving Notice PeriodPrimary Skills : Big Data, PySpark, Cloud- Azure or AWS, Databricks, SQL, PythonOverview : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a...

  • Pyspark Developer

    2 weeks ago

    Pune, India NewVision Software Full time

    Position Summary: We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform. Responsibilities: Design, develop, and implement data pipelines using...

  • Pyspark Developer

    2 weeks ago

    Pune, India NewVision Software Full time

    Position Summary:We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.Responsibilities:Design, develop, and implement data pipelines using Azure...

  • Pyspark Developer

    3 days ago

    pune, India NewVision Software Full time

    Position Summary: We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform. Responsibilities: Design, develop, and implement data pipelines using...

  • Pyspark Developer

    3 days ago

    pune, India NewVision Software Full time

    Position Summary:We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.Responsibilities:Design, develop, and implement data pipelines using Azure...

  • Pyspark Developer

    2 weeks ago

    Pune, India NewVision Software Full time

    Position Summary:We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.Responsibilities:Design, develop, and implement data pipelines using Azure...

  • Pyspark Developer

    2 weeks ago

    Pune, India NewVision Software Full time

    Position Summary:We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.Responsibilities:Design, develop, and implement data pipelines using Azure...

  • AWS DataBricks

    1 month ago

    Pune, India Capgemini Engineering Full time

    EducationBachelor’s or Master’s degree in Computer Science, Information Technology, Data Science, Bioprocess Engineering, Chemical Engineering, or a related field.ExperienceProven experience of 5-7 years as a Data Engineer or in a similar role, with previous experience in the pharmaceutical industry being highly regarded.Hands-on experience with...

  • AWS DataBricks

    1 month ago

    Pune, India Capgemini Engineering Full time

    EducationBachelor’s or Master’s degree in Computer Science, Information Technology, Data Science, Bioprocess Engineering, Chemical Engineering, or a related field.ExperienceProven experience of 5-7 years as a Data Engineer or in a similar role, with previous experience in the pharmaceutical industry being highly regarded.Hands-on experience with...

  • Evnek - Azure Data Engineer - SQL/PySpark

    2 weeks ago

    Pune, India Evnek Technologies Full time

    Job Description : Data Engineer. Location : Pune, India (Work from Office). Experience : 5+ Years. Notice Period : Immediate Joiner. Job Type : Contract. Role Overview : We are seeking an experienced Data Engineer with over 5 years of experience to join our team in Pune. The ideal candidate will be an expert in SQL, with at least 3 years of hands-on...

  • Data Engineer

    2 months ago

    Pune, India Www.Huquo.com Full time

    Position : Data Engineer / Managed ServiceWork Location : Pune/ Hyderabad / Gurgaon / HybridExperience : 5+ YearsMust have Skills : - Experience working with large data sets, building an optimising pipelines and ETL/ELT workflows.- Experience with investigating variety of data frequencies (streaming, batch), formats (JSON, Parquet, CSV) and schemas...

  • Big Data Engineer

    4 months ago

    Mumbai/Pune/Bangalore, India Techno Wise Full time

    Job Description : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering...