Data Engineer

2 months ago


PuneHyderabadRemote, IN HARP Technologies and Services Full time

Job Location : Pune & Hyderabad (Initial 1 -2 months hybrid and then complete remote job. Client will offer accommodation(stay), food & travel exp. )

Exp range : 6+ years

Shift timings : General 9am - 6pm IST or 10am - 7pm IST

Mandatory skills : Data engineering (4+years), Databricks (3.5+ years), Pyspark (3+ years), Python (preferred) or bash, Data pipeline, Azure, ETL.

Job Type : C2H long term (within 6 months based on your performance/cultural fit you might be onboarded by the end client as a permanent employee)

We are seeking a skilled Data Engineer with 5+ years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks.


As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.

Responsibilities :

- Design, develop, and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.

- Proven experience in optimizing Apache Spark batch processing workflows.

- Extensive experience in building and maintaining streaming data pipelines.

- Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.

- Troubleshoot issues related to data pipelines, identify bottlenecks, and implement effective solutions.

- Implement best practices for data governance, security, and compliance within DataBricks environments.

- Work closely with Data Scientists and Analysts to support their data requirements and enable efficient access to relevant datasets.

- Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.

- Demonstrated expertise in optimizing systems for low-latency and high-throughput performance.

- Proficiency in using Spark SQL and DataFrame API for dynamic data transformations.

- Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.

- Familiarity with the principles of distributed systems and their application in message broking.

- Collaborate with cross functional teams to gather requirements, understand data needs, and implement scalable solutions.

Requirements :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 4 to 5 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.

- Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.

- Solid understanding of different components within DataBricks such as clusters, notebooks, jobs, and libraries.

- Strong knowledge of SQL, data modeling, and ETL processes.

- Ability to analyzxcellent communication skills with the ability to collaborate with cross functional teams.

(ref:hirist.tech)
  • Data Engineer

    2 days ago


    Remote, IN Suitable.AI Full time

    Role Objective :Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our software...

  • Snowflake Developer

    4 days ago


    Remote, IN Flexiventures Pvt. Ltd Full time

    We are seeking a skilled Data Engineer with expertise in Snowflake to join our dynamic team. The ideal candidate will have a strong background in data engineering and a deep understanding of Snowflake architecture and development. You will be responsible for designing, building, and optimizing data pipelines and warehouse solutions within the Snowflake...

  • Senior Data Engineer

    2 months ago


    Remote, IN Nthinsight Full time

    Job Description :We are seeking a skilled and experienced Data Engineer with expertise in Python, data warehousing, ETL (Extract, Transform, Load), and SQL. As a Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure to support the organization's data needs. You will collaborate with cross-functional...

  • Senior Data Engineer

    3 weeks ago


    Remote, IN Nthinsight Full time

    Job Description :We are seeking a skilled and experienced Data Engineer with expertise in Python, data warehousing, ETL (Extract, Transform, Load), and SQL. As a Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure to support the organization's data needs. You will collaborate with cross-functional...

  • Data Engineer

    3 weeks ago


    Remote, IN Protiviti India Member Private limited Full time

    Job Description :Roles and Responsibilities :- Design, develop, and maintain robust data pipelines using Azure Data Factory.- Create and manage complex SQL queries and stored procedures in SQL Server.- Implement data integration solutions leveraging Databricks, Azure Synapse Analytics, and ADLS.- Ensure data quality and integrity by implementing best...

  • Data Engineer

    2 months ago


    Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full time

    Role : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...

  • Data Engineer

    3 weeks ago


    Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full time

    Role : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...


  • Remote, IN Prime Infosoft Full time

    What Does a Data Scientist Do?Data scientist roles and responsibilities include: - Data mining or extracting usable data from valuable data sources - Using machine learning tools to select features, create and optimize classifiers - Carrying out preprocessing of structured and unstructured data - Enhancing data collection procedures to include all relevant...


  • Remote, IN Prime Infosoft Full time

    What Does a Data Scientist Do?Data scientist roles and responsibilities include: - Data mining or extracting usable data from valuable data sources - Using machine learning tools to select features, create and optimize classifiers - Carrying out preprocessing of structured and unstructured data - Enhancing data collection procedures to include all relevant...


  • Remote, IN Suitable.AI Full time

    Role Objective :Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our software...


  • Remote, IN Avalara Technologies Pvt ltd Full time

    - ONLY ENGINEERING GRADUATES ( COMPUTER SCIENCE AND RELATED FIELDS ONLY )- Candidates should be from product firms only ( No services firms )- Candidates from top tier institutes ( IIT / REC NIT ) & BITSLead Cloud Data Analytics Engineer We are looking for a talented, highly motivatedLead Cloud Data Engineer to join our team! Responsibilities : - Design,...

  • Data Modeler

    4 days ago


    Remote, IN MindBrain Full time

    Key Responsibilities :- Data Modeling: Design and develop data models for ETL processes to support data warehousing and business intelligence solutions.- Warehouse Architecture: Create and maintain data warehouse architecture, ensuring it aligns with business needs and performance standards.- SQL Proficiency: Utilize strong SQL skills to write, optimize, and...

  • Data Engineer

    3 weeks ago


    Pune/Hyderabad, IN EDGESOFT Full time

    Job Description :The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.Responsibilities :- Design,...

  • Data Engineer

    3 weeks ago


    Pune/Hyderabad, IN EDGESOFT Full time

    Job Description :The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.Responsibilities :- Design,...

  • Data Engineer

    1 month ago


    Hyderabad/Chennai/Pune, IN Freelancer HR Full time

    Description :Job Description :- Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Data bricks and/or HDInsights- Experience with other Open Source big data products Hadoop (incl. Hive, Pig, Impala)- Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,...

  • Data Engineer

    3 weeks ago


    Hyderabad/Chennai/Pune, IN Freelancer HR Full time

    Description :Job Description :- Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Data bricks and/or HDInsights- Experience with other Open Source big data products Hadoop (incl. Hive, Pig, Impala)- Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,...

  • Aptus Data Labs

    4 days ago


    Mumbai/Bangalore/Hyderabad/Gurgaon/Gurugram, IN Aptus Data Labs Full time

    Role - Data Scientist (Azure)Location - Bangalore/Hyderabad/Mumbai/Gurgaon (Onsite)Apply if you can join us within 30 days.The ideal candidate's favorite words are learning, data, scale, and agility. You will leverage your strong collaboration skills and ability to extract valuable insights from highly complex data sets to ask the right questions and...

  • Aptus Data Labs

    2 weeks ago


    Bangalore/Hyderabad/Gurgaon/Gurugram/Mumbai, IN Aptus Data Labs Full time

    Description :We are seeking an experienced and talented individual for the position of Data Governance with Azure experience. In this role, you will be responsible for managing the data governance framework using Azure tools and technologies, and collaborating closely with cross-functional teams to implement and maintain best practices.Location :...

  • Azure Data Engineer

    2 months ago


    Bangalore/Hyderabad/Pune, IN Kezan Consulting Full time

    Work Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...

  • Azure Data Engineer

    3 weeks ago


    Bangalore/Hyderabad/Pune, IN Kezan Consulting Full time

    Work Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...