PySpark/Databricks Engineer

2 weeks ago

Anywhere in IndiaMultiple LocationsHyderabadSrinagarJaipur Aricent Full time

Job : PySpark/Databricks Engineer

Open for Multiple Locations with WFO and WFH

Job Description :

We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims to build a data standardized and curation-based Hadoop cluster

This high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer s critical systems

Key Responsibilities :

- Ability to design, build and unit test applications on Spark framework on Python.

- Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.

- Develop and execute data pipeline testing processes and validate business rules and policies.

- Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDDs.

- Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.

- Ability to design build real-time applications using Apache Kafka Spark Streaming

- Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.

- Build data tokenization libraries and integrate with Hive Spark for column-level obfuscation

- Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.

- Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories

- Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings

- Work collaboratively with onsite and offshore team.

- Develop review technical documentation for artifacts delivered.

- Ability to solve complex data-driven scenarios and triage towards defects and production issues

- Ability to learn-unlearn-relearn concepts with an open and analytical mindset

- Participate in code release and production deployment.

- Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

- BE/B.Tech/ B.Sc. in Computer Science/Statistics, Econometrics from an accredited college or university.

- Minimum 3 years of extensive experience in design, build and deployment of PySpark-based applications.

- Expertise in handling complex large-scale Big Data environments preferably (20Tb+).

- Minimum 3 years of experience in the following: HIVE, YARN, HDFS preferably on Hortonworks Data Platform.

- Good implementation experience of OOPS concepts.

- Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.

- Ability to build abstracted, modularized reusable code components.

- Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses

(ref:hirist.tech)

  • Azure Data Lead

    2 weeks ago

    Anywhere in India/Multiple Locations Etaash Consulting Full time

    Years of experience : 7 to 15 Years Role : Sr. Tech Lead Job Description : - Experience in Perform Design, Development & Deployment using Azure Services (Databricks, PySpark, SQL, Data Factory,)- Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.- Experience in...

  • Azure Databricks Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations IT Source Global Full time

    We have Immediate Openings on Azure Databricks EngineerJob Description :- Design, develop, and maintain scalable data processing solutions using Azure Databricks and Azure Data Factory.- Build and optimize end-to-end data pipelines for batch and real-time data ingestion, transformation, and loading.- Develop complex ETL processes using PySpark on Databricks,...

  • Azure Databricks Developer

    1 month ago

    Anywhere in India/Multiple Locations SAN Engineering Solutions Full time

    About the PositionAt SAN Engineering Solutions, we are seeking a skilled Azure Databricks Engineer to join our dynamic team. The ideal candidate will have extensive experience in Azure services, particularly Databricks, and will be responsible for designing, developing, and maintaining data pipelines and ETL processes.Key Responsibilities• Design and...

  • Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations PureSoftware Pvt Ltd. Full time

    We are seeking a talented and experienced Azure Data Engineer to join our team. As an Azure Data Engineer, Mandatory Skills : - Databricks(PySpark, Scala) - Data Factory/Synapse - SQL DB and DW - Working knowledge on Git Roles and Responsibilities : Requirements : - 4+ years of experience working as a Data Engineer, with a focus on Azure cloud...

  • Azure Databricks Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations SAN Engineering Solutions Full time

    Job Description : Position : Azure Databricks EngineerExperience Level : 5 - 9 YearsLocation : Pan India (Remote Work Available)Job Type : Full-TimeAvailability : Immediate / Early Joiners PreferredAbout the Role :We are seeking a skilled Azure Databricks Engineer to join our dynamic team. The ideal candidate will have extensive experience in Azure services,...

  • Pyspark Databricks

    3 months ago

    Hyderabad, India Risk Resources LLP Full time

    OverviewThe positionof Pyspark Databricks is crucial to our organization as it involvesdeveloping and implementing data processing pipelines using Pysparkon the Databricks platform ensuring efficient and scalable dataprocessing and analysis.KeyresponsibilitiesDesign and developdata processing pipelines using Pyspark andDatabricks.Optimize and tune the...

  • Pyspark With Azure Databricks

    4 weeks ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate, We have an opportunity of PySpark with DataBricks position with HTC Global Services Please share your updated resume on devanshi.nigam@htcinc.com with below details - We are Looking for Immediate or 15days joiners only Candidate‘s Full Name (As Per Aadhar) - Total Exp. - Rel. Exp. (in Pyspark) - Rel. Exp. (in Databricks) - Notice...

  • PySpark with Azure Databricks

    4 weeks ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate, We have an opportunity of PySpark with DataBricks position with HTC Global Services Please share your updated resume on devanshi.nigam@htcinc.com with below details - We are Looking for Immediate or 15days joiners only Candidate's Full Name (As Per Aadhar) - Total Exp. - Rel. Exp. (in Pyspark) - Rel. Exp. (in Databricks) - Notice...

  • Pyspark With Azure Databricks

    4 weeks ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate, We have an opportunity of PySpark with DataBricks position with HTC Global Services Please share your updated resume on devanshi.nigam@htcinc.com with below details - We are Looking for Immediate or 15days joiners only Candidate‘s Full Name (As Per Aadhar) - Total Exp. - Rel. Exp. (in Pyspark) - Rel. Exp. (in Databricks) - Notice...

  • PySpark with Azure Databricks

    1 month ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate, We have an opportunity of PySpark with DataBricks position with HTC Global Services Please share your updated resume on devanshi.nigam@htcinc.com with below details - We are Looking for Immediate or 15days joiners only Candidate's Full Name (As Per Aadhar) - Total Exp. - Rel. Exp. (in Pyspark) - Rel. Exp. (in Databricks) - Notice Period- If...

  • ML Engineer

    6 months ago

    Hyderabad, India Tiger Analytics Full time

    Job Description ML Engineer (Databricks + PySpark) Locations: Chennai / Hyderabad / Bangalore Tiger Analytics is a global leader in AI and analytics, helping Fortune companies solve their toughest challenges. We are on a mission to push the boundaries of what AI and analytics can do to help enterprises navigate uncertainty and move forward decisively....

  • PySpark with Azure Databricks

    1 month ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate,We have an opportunity of PySpark with DataBricks position with HTC Global ServicesPlease share your updated resume on with below details -We are Looking for Immediate or 15days joiners onlyCandidate's Full Name (As Per Aadhar) -Total Exp. -Rel. Exp. (in Pyspark) -Rel. Exp. (in Databricks) -Notice Period-If serving Notice or not working,...

  • PySpark with Azure Databricks

    1 month ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate,We have an opportunity of PySpark with DataBricks position with HTC Global ServicesPlease share your updated resume on with below details -We are Looking for Immediate or 15days joiners onlyCandidate's Full Name (As Per Aadhar) -Total Exp. -Rel. Exp. (in Pyspark) -Rel. Exp. (in Databricks) -Notice Period-If serving Notice or not working,...

  • PySpark with Azure Databricks

    1 month ago

    Hyderabad, India HTC Global Services Full time

    Dear Candidate,We have an opportunity of PySpark with DataBricks position with HTC Global ServicesPlease share your updated resume on with below details -We are Looking for Immediate or 15days joiners onlyCandidate's Full Name (As Per Aadhar) -Total Exp. -Rel. Exp. (in Pyspark) -Rel. Exp. (in Databricks) -Notice Period-If serving Notice or not working,...

  • Experienced Data Engineer

    6 days ago

    Hyderabad, Telangana, India HTC Global Services Full time

    We are seeking a highly skilled Data Engineer to join our team at HTC Global Services. This role involves working on exciting projects utilizing PySpark and Azure Databricks.About the RoleAs a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems. Your expertise in PySpark and Azure Databricks...

  • PySpark with Azure Databricks

    2 months ago

    Hyderabad, India HTC Global Services Full time

    Dear Candidate, We have an opportunity of PySpark with DataBricks position with HTC Global Services Please share your updated resume on with below details - We are Looking for Immediate or 15days joiners only Candidate's Full Name (As Per Aadhar) - Total Exp. - Rel. Exp. (in Pyspark) - Rel. Exp. (in Databricks) - Notice Period- If serving...

  • PySpark with Azure Databricks

    2 months ago

    Hyderabad, India HTC Global Services Full time

    Dear Candidate,We have an opportunity of PySpark with DataBricks position with HTC Global ServicesPlease share your updated resume on devanshi.nigam@htcinc.com with below details -We are Looking for Immediate or 15days joiners onlyCandidate's Full Name (As Per Aadhar) -Total Exp. -Rel. Exp. (in Pyspark) -Rel. Exp. (in Databricks) -Notice Period-If serving...

  • PySpark with Azure Databricks

    2 months ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate, We have an opportunity of PySpark with DataBricks position with HTC Global Services Please share your updated resume on with below details - We are Looking for Immediate or 15days joiners only Candidate's Full Name (As Per Aadhar) - Total Exp. - Rel. Exp. (in Pyspark) - Rel. Exp. (in Databricks) - Notice Period- If serving Notice or not...

  • PySpark with Azure Databricks

    2 months ago

    Hyderabad, India HTC Global Services Full time

    Dear Candidate,We have an opportunity of PySpark with DataBricks position with HTC Global ServicesPlease share your updated resume on devanshi.nigam@htcinc.com with below details -We are Looking for Immediate or 15days joiners onlyCandidate's Full Name (As Per Aadhar) -Total Exp. -Rel. Exp. (in Pyspark) -Rel. Exp. (in Databricks) -Notice Period-If serving...

  • PySpark with Azure Databricks

    2 months ago

    hyderabad, India HTC Global Services Full time

    Dear Candidate,We have an opportunity of PySpark with DataBricks position with HTC Global ServicesPlease share your updated resume on devanshi.nigam@htcinc.com with below details -We are Looking for Immediate or 15days joiners onlyCandidate's Full Name (As Per Aadhar) -Total Exp. -Rel. Exp. (in Pyspark) -Rel. Exp. (in Databricks) -Notice Period-If serving...