Databricks + Pyspark

1 week ago

Indore Madhya Pradesh, India Cognizant Full time

**Job Summary**

**Responsibilities**
- Lead the development and implementation of clinical data systems using Scala and PySpark to ensure efficient data processing.
- Oversee the setup and maintenance of CI/CD pipelines in Azure DevOps to streamline development workflows.
- Provide expertise in Azure Repos for version control and collaboration on code repositories.
- Manage and optimize Databricks and Azure Databricks environments for advanced analytics and data engineering tasks.
- Implement Azure Stream Analytics to process real-time data streams and integrate them into clinical systems.
- Utilize Data Factory and Azure Data Factory for orchestrating data workflows and ensuring seamless data integration.
- Leverage Azure Storage solutions to store and manage large volumes of clinical data securely.
- Develop and maintain Azure Services to support various clinical data management and processing needs.
- Collaborate with cross-functional teams to ensure the successful implementation of clinical systems.
- Provide technical guidance and support to junior developers and team members.
- Ensure compliance with industry standards and regulations in clinical data management.
- Conduct regular code reviews and ensure adherence to best practices in software development.
- Monitor system performance and implement optimizations to enhance efficiency and reliability.

**Qualifications**
- Possess a strong background in Scala and PySpark for data processing and analytics.
- Demonstrate expertise in setting up and managing CI/CD pipelines in Azure DevOps.
- Have extensive experience with Azure Repos for version control and collaboration.
- Show proficiency in Databricks and Azure Databricks for data engineering and analytics.
- Be skilled in implementing Azure Stream Analytics for real-time data processing.
- Have hands-on experience with Data Factory and Azure Data Factory for data orchestration.
- Be knowledgeable in Azure Storage solutions for secure data management.

  • Aws Databricks

    5 days ago

    Andhra Pradesh, India Virtusa Full time

    Must have 7plus years of experience total IT experience in Data engineering DW area Must have 3plus yrs of exposure on AWS ecosystem specifically DataBricks, Redshift Lambda Glue Starbust Minimum 5 yrs of technical expertise with data models data mining and segmentation techniques Expert on programming languages SQL Python Pyspark Must be good at...

  • Data Engineer

    1 week ago

    Indore, Madhya Pradesh, India Vitric Business Solutions Pvt. Ltd. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Job Title: Data EngineerExperience: 5+ YearsLocation: IndoreEmployment Type: Full-TimeMandatory Tech Stack: Programming: Python Big Data Processing: PySpark Database: SQL ETL Tools & Processes Platform: DatabricksKey Responsibilities: Design, develop, and maintain scalable data pipelines and ETL workflows. Work with large datasets using...

  • Data Engineer

    16 hours ago

    Indore, India Vitric Business Solutions Pvt. Ltd. Full time

    Job Title: Data Engineer Experience: 5+ Years Location: Indore Employment Type: Full-Time Mandatory Tech Stack:  Programming: Python  Big Data Processing: PySpark  Database: SQL  ETL Tools & Processes  Platform: Databricks Key Responsibilities:  Design, develop, and maintain scalable data pipelines and ETL workflows.  Work with large...

  • Big Data Cloud Engineer

    6 days ago

    Bhubaneswar, Indore, Nagpur, India beBeeDatabricks Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Title: Azure Databricks Developer">Experience with data warehousing, data integration and analytics is desirable.A strong proficiency in big data processing using Azure services such as Databricks and Data Factory.Proficiency in SQL or TSQL for efficient data processing and reporting tasks.">Responsibilities:">Design and develop scalable and efficient...

  • Data Modeller

    2 weeks ago

    Indore, India TEANSOFT Full time

    Mandatory experience in Advance Data & Analytics Architecture, ETL, Data Engineering solution using the following Skills, tools and technologies: AWS Data & Analytics Services: Athena, Glue, DynamoDB, Redshift Databricks Lakehouse Platform PySpark, Spark SQL, Experience with any NoSQL Database Expert-level skills in writing and optimizing SQL. Experience...

  • Subcontractor-data Scientist

    2 weeks ago

    Noida, Uttar Pradesh, India BirlaSoft Full time

    Country/Region: IN - Requisition ID: 25471 - Work Model: - Position Type: - Salary Range: - Location: INDIA - NOIDA- BIRLASOFT OFFICE **Title**:Subcontractor-Data Scientist**: - Description: **Area(s) of responsibility**: Databricks with Data Scientist experience (8 to 10) - 5B 4 years of relevant work experience as a data scientist Minimum 2 years of...

  • Principal Backend Software Engineer

    2 weeks ago

    Indore, Madhya Pradesh, India beBeeCloud Full time ₹ 20,00,000 - ₹ 25,00,000

    Backend Architect LeadThe Backend Architect Lead will be responsible for designing and implementing scalable and secure backend systems. This role requires strong leadership skills, deep technical expertise, and strategic collaboration across teams.Key Responsibilities:Architect and build scalable RESTful APIs and microservices using Python & DjangoImplement...

  • Aws Engineer

    2 weeks ago

    Bhubaneswar, Coimbatore, Indore, India Ltimindtree Full time ₹ 5,00,000 - ₹ 10,00,000 per year

    Role & responsibilitiesAWS,Glue or DataBricks,SQL,Python,Pyspark.

  • Solution Architect

    6 days ago

    Indore, India Impetus Technologies Full time

    The Databricks Architect role will be part of our client’s Data Engineering practice team and will be focused on building Data pipelines and Data processing solutions based on AWS platform for Digital Marketing domain. The Databricks Architect will be working closely with Enterprise Architect, Data Scientist and Product Manager’s for creating technical...

  • Big Data Developer

    2 weeks ago

    Indore, Madhya Pradesh, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Cloud Data EngineerJob Description:As a Cloud Data Engineer, you will design and develop large-scale data systems that drive business growth. Your primary responsibility is to architect, implement, and optimize scalable data platforms using cloud-based technologies like Azure Databricks.Key Responsibilities:Our ideal candidate will have expertise...