DBX Data Engineer

1 month ago

PuneBangaloreGurgaonGurugramNoida, India HARP Technologies and Services Full time

Job Description :


We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks.


As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.

Responsibilities :

- Design, develop, and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.

- Proven experience in optimizing Apache Spark batch processing workflows.

- Extensive experience in building and maintaining streaming data pipelines.

- Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.

- Troubleshoot issues related to data pipelines, identify bottlenecks, and implement effective solutions.

- Implement best practices for data governance, security, and compliance within DataBricks environments.

- Work closely with Data Scientists and Analysts to support their data requirements and enable efficient access to relevant datasets.

- Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.

- Demonstrated expertise in optimizing systems for low-latency and high-throughput performance.

- Proficiency in using Spark SQL and DataFrame API for dynamic data transformations.

- Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.

- Familiarity with the principles of distributed systems and their application in message broking.

- Collaborate with cross functional teams to gather requirements, understand data needs, and implement scalable solutions.

Requirements :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 4 to 5 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.

- Experience in cloud environment AWS tools services is mandatory.

- Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.

- Solid understanding of different components within DataBricks such as clusters, notebooks, jobs, and libraries.

- Strong knowledge of SQL, data modeling, and ETL processes.

- Ability to analyse excellent communication skills with the ability to collaborate with cross functional teams.

(ref:hirist.tech)

  • Data Engineer

    2 weeks ago

    pune, India EDGESOFT Full time

    Data Engineer - DBX We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and...

  • Data Engineer

    3 weeks ago

    Pune, India EDGESOFT Full time

    Data Engineer - DBX We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining...

  • Data Engineer

    4 weeks ago

    Pune, Maharashtra, India EDGESOFT Full time

    Data Engineer - DBX We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining...

  • Data Engineer

    2 weeks ago

    Pune, India EDGESOFT Full time

    Data Engineer - DBX We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and...

  • Senior Data Engineer

    4 days ago

    Pune, India Data Axle Full time

    Job DescriptionAbout Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and...

  • Senior Data Engineer

    1 month ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Senior Data Engineer

    2 weeks ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Data Engineer

    1 month ago

    Gurgaon,Gurugram, India E-Qube Digital Services Full time

    Position Summary :- Data Engineers are primarily responsible for designing, building, managing, and operationalizing data pipelines in support of key data and analytics use cases. - They are accountable for guaranteeing compliance with data governance, data privacy, and data security requirements.- Data Engineers are also responsible for the Enterprise...

  • Machine Learning Engineer

    1 month ago

    Bangalore, India Staffington Consulting Pvt Ltd Full time

    JOB DESCRIPTION : Responsibilities : - Set up Databricks workspaces for data analysis and machine learning tasks.- Import and explore data using Databricks, ensuring data quality and integrity.- Collaborate with cross-functional teams to design, develop, and deploy AI solutions leveraging GPT, LLMOps, and Small Language Models.- Research and experiment with...

  • Data Engineer

    1 month ago

    Pune,Gurgaon,Gurugram,Bangalore,Noida, India HuQuo Consulting Pvt. Ltd. Full time

    Data Engineer Role Mandate skills : SSIS + Datawarehouse + Azure Data Factory + SQL Job Description :- Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights.- Constructing infrastructure for efficient ETL processes from various sources and storage systems.- Collaborating closely...

  • Senior Data Scientist

    2 weeks ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the USA. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based...

  • Senior Data Scientist

    1 month ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the USA. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Sr. Storage Developer

    4 weeks ago

    Pune, India Corporate Resources Full time

    Job DescriptionStrong Storage background with experience in one or more of SAN, Block level protocols (SCSI, iSCSI, Fibre Channel, Fibre Channel over Ethernet (FCoE)), Replication, Data Deduplication, Storage VirtualizationStrong system programming skills in C.Strong UNIX system programming skills, Kernel development experience is highly desirableStrong OS...

  • OfBusiness - Data Engineer - Data Pipeline

    3 days ago

    Gurgaon/Gurugram, India OFB Tech Pvt Ltd (OfBusiness) Full time

    We are currently seeking a talented and motivated Data Engineer to join our dynamic team and play a crucial role in shaping the future of our data : - Design, develop, and maintain robust data pipelines for efficient and scalable data processing.- Implement data integration solutions using Python and Java programming languages.- Utilize Airflow for...

  • OfBusiness - Data Engineer - Data Pipeline

    3 days ago

    Gurgaon/Gurugram, IN OFB Tech Pvt Ltd Full time

    We are currently seeking a talented and motivated Data Engineer to join our dynamic team and play a crucial role in shaping the future of our data : - Design, develop, and maintain robust data pipelines for efficient and scalable data processing.- Implement data integration solutions using Python and Java programming languages.- Utilize Airflow for...

  • Data Engineer

    1 month ago

    Gurgaon,Gurugram,Bangalore, India Huquo Full time

    Role and Responsibilities :- To work on Data Governance - Build a robust data quality validation framework, business rules to identify any data quality issues.- Liaise with different client stakeholders on ad-hoc analyses related to monitoring the data- Build ETL & data pipelines to help feed the data into different business facing data products /...

  • Data Engineer

    4 weeks ago

    Gurgaon/Gurugram/Bangalore, IN Huquo Full time

    Role and Responsibilities :- To work on Data Governance - Build a robust data quality validation framework, business rules to identify any data quality issues.- Liaise with different client stakeholders on ad-hoc analyses related to monitoring the data- Build ETL & data pipelines to help feed the data into different business facing data products /...

  • Data Engineer

    2 weeks ago

    Bangalore/Gurgaon/Gurugram, IN Intuitive Apps Full time

    Key Skills : Minimum 6 years Exp in Data Engineer , 3yrs in Azure Data Factory, Pyspark, DatabricksJob Description :- Create and maintain optimal data pipeline architecture- Knowledge on Data Modelling, Data warehouse, and Data Analysis is also required.- Design and develop Data Warehouse solutions- Assemble large, complex data sets that meet functional /...

  • Data Engineer

    2 weeks ago

    Bangalore/Gurgaon/Gurugram, India Intuitive Apps Full time

    Key Skills : Minimum 6 years Exp in Data Engineer , 3yrs in Azure Data Factory, Pyspark, Databricks Job Description : - Create and maintain optimal data pipeline architecture - Knowledge on Data Modelling, Data warehouse, and Data Analysis is also required. - Design and develop Data Warehouse solutions- Assemble large, complex data sets that meet...

  • Paytm Insurance

    2 weeks ago

    Noida/Gurgaon/Gurugram/Bangalore, India Paytm Insurance Broking Private Limited Full time

    About the Role : You would be responsible to lead a team of Strong Data Engineers including Leads; understand the strategic, financial, and operational goals of the organization and identify opportunities to leverage Big Data solutions towards the same. Execution Own the execution of Data Engineering-based products/platforms with an impact-focused output....