Azure Databricks Developer

2 months ago

Kolkata, India SELSOFT Full time

ROLE : Data Bricks Developer

EXPERIENCE : 5+Years

LOCATION : Kolkata , Mumbai , Hyderabad

Responsibilities :

- Design and implement efficient data pipelines and ETL processes

- Collaborate with cross-functional teams to develop and implement data-driven solutions

- Optimize data models and queries for performance

- Develop and maintain data documentation and data lineage

- Mentor and provide technical leadership to junior team members

Requirements :

- 3+ years of software engineering experience

- Strong experience in Databricks

- Experience with AWS, Python, Big Data

- Hands-on knowledge of Spark

- Good leadership skills

- Upper-Intermediate level of English (B2+)

The key responsibilities of an Azure data engineer include the following :

1. Designing and implementing data storage solutions on Azure :

One of the primary responsibilities of an Azure data engineer is to design and implement data storage solutions on Azure. That includes selecting the appropriate Azure data storage services for the specific use case, such as Azure SQL Database, Azure Cosmos DB, or Azure Data Lake Storage. The data engineer must also design the data storage schema and ensure it is optimized for the specific use case.

2. Building and maintaining data pipelines for data integration and processing :

Another key responsibility of an Azure data engineer is to build and maintain data pipelines for data integration and processing. Accordingly, data is collected from various sources, transformed into a practical format, and loaded into the appropriate data storage solution on Azure using tools such as Azure Data Factory and Azure Databricks. The data engineer must also ensure the data pipeline is scalable and efficient.

3. Ensuring data quality and accuracy through testing and validation :

Data quality and accuracy are critical to the success of any data management system. As an Azure data engineer, you must ensure that the data stored and processed on Azure is highly accurate. It involves testing and verifying the data at different phases of the data pipeline, from extraction through loading.

4. Optimizing data processing performance through tuning and monitoring :

Data processing performance is another critical aspect of Azure data engineering. The data engineer must ensure the data processing is efficient and scalable by tuning the data pipeline and monitoring performance. That involves identifying and resolving bottlenecks in the data pipeline and optimizing the data processing algorithms.

5. Developing and maintaining data models and schemas :

Data modeling and schema design are important aspects of Azure data engineering. The data engineer must develop and maintain data models and schemas optimized for the specific use case. That involves selecting the appropriate data modeling techniques and ensuring the data schema is scalable and efficient.

6. Collaborating with other teams to provide data for analytics and reporting :

As an Azure data engineer, you will work closely with other teams, such as data analysts, data scientists, and software developers, to provide the data they need to perform their job functions. That involves understanding the specific data requirements of each team and ensuring that the data is available in the appropriate format and location on Azure.

7. Ensuring data security and privacy prerequisites are followed :

The security and privacy of knowledge are important elements of any information management system. The data engineer oversees ensuring that the data stored and processed on Azure conforms with applicable data security and privacy requirements, such as HIPAA and GDPR.

(ref:hirist.tech)

  • Azure Databricks

    3 weeks ago

    Kolkata, India People Prime Worldwide Full time

    AboutClient: Our client is globaltechnology consulting and digital solutions company that enablesenterprises to reimagine business models and accelerate innovationthrough digital technologies. Powered by more than 84000entrepreneurial professionals across more than 30 countries itcovers to over 700 clients. With its extensive domain andtechnology expertise...

  • Databricks Developer

    3 weeks ago

    Kolkata, India Selsoftinc Full time

    Job Description :- Design, develop, and maintain data pipelines and workflows using Databricks.- Create and manage ETL processes to ingest, process, and store large volumes of data.- Develop and optimize Spark jobs and notebooks for efficient data processing.Data Engineering :- Work with large datasets and perform data transformations, aggregations, and...

  • Databricks Developer

    2 months ago

    Kolkata, India NUSTAR TECHNOLOGIES INDIA PRIVATE LIMITED Full time

    Exp: 5+yrsLocation : KolkataJob Description : - Minimum 5 years + working exp as Databricks Developer.- Minimum 3 years + working exp on PySpark and AWS.- 5+ years relevant and progressive data engineering experience.- Deep Technical knowledge and experience in Databricks, Python, Scala, Microsoft Azure architecture and platform including Synapse, ADF (Azure...

  • Lead Consultant

    2 months ago

    Kolkata, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people –...

  • Lead Consultant-Databricks Engineer

    4 months ago

    Kolkata, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people –...

  • Senior Principal Consultant

    4 months ago

    Kolkata, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people – we...

  • Lead Consultant

    2 months ago

    Kolkata, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people –...

  • Data Engineer

    4 weeks ago

    Kolkata, India Webologix Ltd INC Full time

    Job Position: Azure Cloud DB EngineerJob Type: Full-TimeLocation: Kolkata, INExperience: - 3 to 8+ YearsMust Have: - Azure DATA Factory; Azure SQL; Azure Databricks; Azure Cosmos DB; PySpark; PythonResponsibilities:Data Pipeline Development:Design and implement data integration workflows using Azure Data Factory to ingest, transform, and load data from...

  • Azure Databricks

    4 months ago

    Kolkata, West Bengal, India Cognizant Full time

    **Lead Administrator** **Qualification**: Bachelors in science, engineering or equivalent ** Responsibility**: Database/ tool Monitoring - Identify the specific parameters (Disk usage, performance metrics ) to be monitored and threshold values for those parameters. - Based on the criticality and understanding of the requirement, prioritize the specific...

  • Lead DataOps Engineer

    3 weeks ago

    Kolkata, India RS Consulting Full time

    Job Title: Lead DataOps EngineerLocation: Pune, IndiaExperience: 10+ YearsJob Description: As a DataOps Engineer, you will be responsible for managing data operations in a Cloud Data Lakehouse environment. You will lead the DataOps team and ensure the smooth functioning of the Azure Databricks Data & Analytics platform.Key Responsibilities:Build the DataOps...

  • Azure Engineer

    5 months ago

    kolkata, India Corporate Resources Full time

    Job Description Design, deploy and maintain Azure-based solutions: The Azure engineer should have a deep understanding of Azure cloud services and be able to design, deploy and maintain Azure-based solutions. This includes virtual machines, web applications, storage solutions, and other Azure services. ...

  • Data Engineer

    4 weeks ago

    Kolkata, India Webologix Ltd INC Full time

    Job Position: Azure Cloud DB EngineerJob Type: Full-TimeLocation: Kolkata, INExperience: - 3 to 8+ YearsMust Have: - Azure DATA Factory; Azure SQL; Azure Databricks; Azure Cosmos DB; PySpark; PythonResponsibilities: Data Pipeline Development:Design and implement data integration workflows using Azure Data Factory to ingest, transform, and load data from...

  • Data Engineer

    4 weeks ago

    Kolkata, India Webologix Ltd INC Full time

    Job Position: Azure Cloud DB EngineerJob Type: Full-TimeLocation: Kolkata, INExperience: - 3 to 8+ YearsMust Have: - Azure DATA Factory; Azure SQL; Azure Databricks; Azure Cosmos DB; PySpark; PythonResponsibilities: Data Pipeline Development:Design and implement data integration workflows using Azure Data Factory to ingest, transform, and load data from...

  • Data Engineer

    4 weeks ago

    Kolkata, India Webologix Ltd INC Full time

    Job Position: Azure Cloud DB Engineer Job Type: Full-Time Location: Kolkata, IN Experience: - 3 to 8+ Years Must Have: - Azure DATA Factory; Azure SQL; Azure Databricks; Azure Cosmos DB; PySpark; Python Responsibilities:  Data Pipeline Development: Design and implement data integration workflows using Azure Data Factory to ingest, transform, and...

  • Azure Admin

    3 months ago

    Kolkata, India ZINFI SOFTWARE SYSTEMS Full time

    **What you will be doing**: - Plan, design, and implement solutions on the Azure platform to meet business requirements. - Manage and maintain the company’s Azure infrastructure. - Design & develop cost-effective solutions and implement cost-optimization strategies for the Azure environment. - Monitor and manage billing, reporting, and provide regular...

  • Senior Associate

    3 weeks ago

    KOLKATA, India Cognizant Full time

    Job SummaryWe are seeking a highly skilled Sr. Developer with 8 to 11 years of experience to join our team. The ideal candidate will have extensive experience in Databricks SQL Databricks Workflows and PySpark. Experience in the Property & Casualty Insurance domain is a plus. The role involves developing and optimizing data workflows ensuring data quality...

  • Principal Consultant

    3 weeks ago

    Kolkata, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people – we...

  • Lead Consultant-Data Bricks Senior Engineer

    2 months ago

    Kolkata, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people –...

  • Azure DevOps Engineer

    1 month ago

    Kolkata, India ANB IT Solutions Pvt Ltd Full time

    Responsibilities : - Design and Implement Continuous Integration Pipelines for Application Development projects, primarily through Azure DevOps. - Work with the Cloud based Platforms, primarily Microsoft Azure. - Enforce the proper execution of Agile paradigms in tandem with Management Tools, primarily Azure DevOps formerly VSTS - Establish Automated Code...

  • Azure DevOps Engineer

    4 weeks ago

    Kolkata, India Amazon Web Services (AWS) Full time

    Job Position: Azure DevOps EngineerJob Type: Full-TimeLocation: KolkataExperience: - 5+ to 10+ YearsMust Have: DevOps, Azure AKS, CI/CD architecture, Docker, PowerShell, TerraformResponsibilities:Key Responsibilities:Design, implement, and manage Azure AKS (Azure Kubernetes Service) environments.Develop and maintain robust CI/CD pipelines and architecture...