PySpark/Databricks Engineer

5 months ago

Anywhere in IndiaMultiple LocationsHyderabadSrinagarJaipur, IN Aricent Full time

Job : PySpark/Databricks Engineer

Open for Multiple Locations with WFO and WFH

Job Description :

We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims to build a data standardized and curation-based Hadoop cluster

This high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer s critical systems

Key Responsibilities :

- Ability to design, build and unit test applications on Spark framework on Python.

- Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.

- Develop and execute data pipeline testing processes and validate business rules and policies.

- Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDDs.

- Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.

- Ability to design build real-time applications using Apache Kafka Spark Streaming

- Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.

- Build data tokenization libraries and integrate with Hive Spark for column-level obfuscation

- Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.

- Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories

- Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings

- Work collaboratively with onsite and offshore team.

- Develop review technical documentation for artifacts delivered.

- Ability to solve complex data-driven scenarios and triage towards defects and production issues

- Ability to learn-unlearn-relearn concepts with an open and analytical mindset

- Participate in code release and production deployment.

- Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

- BE/B.Tech/ B.Sc. in Computer Science/Statistics, Econometrics from an accredited college or university.

- Minimum 3 years of extensive experience in design, build and deployment of PySpark-based applications.

- Expertise in handling complex large-scale Big Data environments preferably (20Tb+).

- Minimum 3 years of experience in the following: HIVE, YARN, HDFS preferably on Hortonworks Data Platform.

- Good implementation experience of OOPS concepts.

- Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.

- Ability to build abstracted, modularized reusable code components.

- Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses

(ref:hirist.tech)

  • Azure Data Lead

    5 months ago

    Anywhere in India/Multiple Locations, IN Etaash Consulting Full time

    Years of experience : 7 to 15 Years Role : Sr. Tech LeadJob Description :- Experience in Perform Design, Development & Deployment using Azure Services (Databricks, PySpark, SQL, Data Factory,)- Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.- Experience in creating...

  • Azure Databricks Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN Hum Technologies Full time

    We are hiring for Azure Databricks Engineer Data Engineering experience on AWS/Azure and Databricks Strong Experience in Databricks, AWS/Azure, and SQL , creation of jobs using Pyspark. good exposure to Python.For one of the Fortune 500 Clients.Company Name : Hum TechnologiesClient : One of the Fortune 500 CompaniesLocation : Remote/ HybridRole : Azure...

  • Azure Databricks Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN IT Source Global Full time

    We have Immediate Openings on Azure Databricks EngineerJob Description :- Design, develop, and maintain scalable data processing solutions using Azure Databricks and Azure Data Factory.- Build and optimize end-to-end data pipelines for batch and real-time data ingestion, transformation, and loading.- Develop complex ETL processes using PySpark on Databricks,...

  • Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN PureSoftware Pvt Ltd. Full time

    We are seeking a talented and experienced Azure Data Engineer to join our team. As an Azure Data Engineer,Mandatory Skills :- Databricks(PySpark, Scala)- Data Factory/Synapse- SQL DB and DW- Working knowledge on Git Roles and Responsibilities :Requirements :- 4+ years of experience working as a Data Engineer, with a focus on Azure cloud platform.- Good...

  • Azure Databricks Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN SAN Engineering Solutions Full time

    Job Description : Position : Azure Databricks EngineerExperience Level : 5 - 9 YearsLocation : Pan India (Remote Work Available)Job Type : Full-TimeAvailability : Immediate / Early Joiners PreferredAbout the Role :We are seeking a skilled Azure Databricks Engineer to join our dynamic team. The ideal candidate will have extensive experience in Azure services,...

  • Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN Vysystems Full time

    Data Engineer Need short joinersJob Location : Bangalore - Manyata Tech ParkMode : Hybrid.Experience : 4-8 yearsJob Description : We are looking for a skilled Data Engineer with expertise in Azure Data Factory, Azure Databricks, PySpark, Snowflake, and SQL. The ideal candidate will play a key role in designing, building, and maintaining scalable data...

  • Databricks Developer

    1 month ago

    Anywhere in India/Multiple Locations, IN Gen Full time

    About the Company :Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose the relentless pursuit of a world that works better...

  • Data Engineer/Architect/Manager

    1 month ago

    Anywhere in India/Multiple Locations, IN Spectrum Consulting Full time

    Job Description Roles and Responsibilities :- Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines...

  • Data Engineer

    2 weeks ago

    Anywhere in India/Multiple Locations, IN Apidel Technologies Full time

    Data Engineer | 100% Remote | 12-Month Contract. We are looking for an experienced Data Engineer to join our team on a 12-month remote contract. This role offers a great opportunity to work with Azure services, including Azure Databricks, Azure DataFactory, and Azure DevOps, to build secure, scalable data pipelines. What You'll Do :- Develop data...

  • Lead Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN Gen Full time

    Job Description :Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose - the relentless pursuit of a world that works better...

  • Azure Databricks/Python Developer

    4 weeks ago

    Anywhere in India/Multiple Locations, IN AIMDek Technologies Pvt. Ltd. Full time

    Job Title : Azure Databricks + Python Developer for FHIR Interface DevelopmentLocation : India (Remote)Employment Type : Full-timeJob Overview :We are seeking a highly skilled and motivated Azure Databricks + Python Developer to join our team. This role will focus on designing, developing, and implementing healthcare data integration interfaces using FHIR...

  • Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN IT Source Global Full time

    Role : Data EngineerJob Description :- Design, build, and maintain large-scale data pipelines and ETL/ELT workflows using PySpark, Python, and SQL for data extraction, transformation, and loading.- Work extensively with AWS services such as Amazon S3 for data storage and Athena for querying and analyzing structured and unstructured data.- Develop and manage...

  • Data Engineer

    5 months ago

    Pune/Hyderabad, IN EDGESOFT Full time

    Job Description :The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.Responsibilities :- Design,...

  • Lead Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN Gen Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose - the relentless pursuit of a world that works better for people - we...

  • Big Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN Stratosphere IT Services PVT Ltd Full time

    Job Description : - Azure data Engineer - Azure Data Factory - azure databricks- python - sql- Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database- Optional: Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB and Purview.- Strong programming, unit testing & debugging skills in SQL, Python or Scala/Java.- Some experience...

  • Genpact - Databricks Developer - Python/Scala

    1 month ago

    Anywhere in India/Multiple Locations, IN GENPACT India Private Limited Full time

    Job Description :Inviting applications for the role of Principal Consultant- Databricks Developer AWS!In this role, the Databricks Developer is responsible for- solving the real world cutting edge problem to meet both functional and non-functional requirements.Responsibilities :- Maintains close awareness of new and emerging technologies and their potential...

  • Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN TalenTECH Solutions Private Limited Full time

    Technical/Functional Skills :Must have :- 5+ years of experience working in data warehousing systems- Strong experience in Oracle Fusion ecosystem, with strong data-extracting experience using Oracle BICC/BIP.- Must have good functional understanding of Fusion data structures.- Must have strong and proven data engineering experience in big data / Databricks...

  • AWS Data Engineer

    1 month ago

    Anywhere in India/Multiple Locations, IN IT Source Global Full time

    We have Immediate Openings on AWS data engineerJob Description :- Data Pipeline Development: Design and develop ETL processes using AWS Glue, Python, and PySpark to extract, transform, and load data from various sources into data lakes or data warehouses.- Data Integration: Integrate data from multiple sources, ensuring data quality, consistency, and...

  • Azure Data Engineer

    3 weeks ago

    Anywhere in India/Multiple Locations, IN estrel.ai Full time

    Job Description :- 5 - 8 years of experience in IT Industry- - 4/5+ years of experience with Azure Data Engineering Stack (Event Hub, Data Factory , Cosmos DB, Synapse, SQL DB, Databricks, Data Explorer) - 3+ years of experience with Python / Pyspark, Spark, Scala, Hive, Impala - Excellent knowledge of SQL and coding skills - Good understanding of other...

  • Data Architect

    1 month ago

    Anywhere in India/Multiple Locations, IN RAPINNO TECH SOLUTIONS PRIVATE LIMITED Full time

    Position Overview :We are looking for an experienced Data Architect with expertise in Databricks, Apache Spark, and ETL processes. The ideal candidate will have a proven track record of designing and building robust data applications from the ground up. You will play a key role in shaping our data strategy and ensuring our data solutions meet business needs...