Pyspark+databricks

2 days ago

Chennai Tamil Nadu, India Cognizant Full time

**Exp: 4 to 13 years**

**Skill: Data Bricks+Pyspark**

**Location : Bangalore/Hyderabad/Kolkota/Pune/Chennai**

**Technical Skills**:
Python,PySpark,Azure Data Lake Store,Databricks Workflows,Databricks SQL

**Responsibilities**:

- Develop and optimize data solutions using Azure Data Lake Store to support business needs.
- Utilize Python to create efficient and scalable data processing scripts.
- Implement and manage Databricks SQL for querying and analyzing large datasets.
- Design and maintain Databricks Workflows to automate data processing tasks.
- Leverage PySpark to perform large-scale data transformations and analytics.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Ensure data integrity and quality through rigorous testing and validation processes.
- Provide technical guidance and support to junior developers and team members.
- Monitor and troubleshoot data pipelines to ensure smooth operation.
- Stay updated with the latest industry trends and technologies to continuously improve data solutions.
- Document all development processes and workflows for future reference.
- Contribute to the overall data strategy and architecture of the organization.
- Drive innovation by exploring new tools and techniques to enhance data capabilities.
- Qualifications
- Possess strong experience in Azure Data Lake Store, demonstrating the ability to manage and optimize large data repositories.
- Have extensive knowledge of Python, with a proven track record of developing efficient data processing scripts.
- Show proficiency in Databricks SQL, capable of querying and analyzing complex datasets.
- Demonstrate expertise in designing and maintaining Databricks Workflows for automated data processing.
- Exhibit strong skills in PySpark, with experience in large-scale data transformations and analytics.
- Nice to have experience in Investment Banking Operations, providing valuable domain insights.

  • Pyspark

    4 days ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **Job Summary** **Responsibilities** - Develop and maintain data solutions using Databricks SQL Databricks Delta Lake and Databricks Workflows. - Optimize and enhance existing data workflows to improve performance and efficiency. - Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs. -...

  • Databrick

    6 days ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Key Responsibilities: Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates,...

  • PySpark Developer

    2 days ago

    Bengaluru / Bangalore, Chennai, Pune, India beBeeData Full time US$ 1,04,000 - US$ 1,30,878

    Job SummaryWe are seeking a highly skilled PySpark expert with hands-on experience in Databricks to join our data engineering team. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on the Databricks Unified Analytics Platform.Key Responsibilities:Develop and manage data pipelines using PySpark on...

  • Pyspark Dev

    7 days ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Pyspark Dev & QA Mandatory Skills BIG Data technology mentioned below Hadoop / Big Data (HDFS, PYTHON, SPARK-SQL, MapReduce) with PySpark Build CI/CD pipelines is required Outstanding coding, debugging and analytical skills Spark APIs to cleanse, explore, aggregate, transform, store & analyse available data Knowledge of installing, configuring, debugging...

  • PySpark Developer

    6 days ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job DescriptionROLE RESPONSIBILITIESData Engineering and Processing:Develop and manage data pipelines using PySpark on Databricks.Implement ETL/ELT processes to process structured and unstructured data at scale.Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.Databricks Platform Expertise:Experience in Perform Design,...

  • Azure Databrick

    4 days ago

    Chennai, India LatentView Analytics Pvt. Ltd Full time

    Role : Azure Databricks Experience:5 to 9 yrs Location: Chennai Position : Full time Good to have : - 5+ years of relevant experience in azure Data brick. - Strong proficiency in Microsoft SQL Server development, including i)ability to query and investigate data and to write views, stored procedures and other objects using SQL. ii)understanding and...

  • Data Architect- Databricks

    7 days ago

    Chennai, Tamil Nadu, India InfoCepts Full time

    **Position**: Data Architect (Databricks, Snowflake) **Purpose of the Position**: Design and optimize cloud-native data architectures on platforms like Databricks and Snowflake, enabling scalable data engineering, advanced analytics, and AI/ML solutions aligned with business needs. **Location**: Nagpur, Pune, Chennai, Bangalore **Type of Employment**:Full...

  • Pyspark developer

    3 days ago

    Bengaluru, Chennai, Pune, India MFX Infotech Private Limited Full time

    Job DescriptionROLE SUMMARYWe are seeking a highly skilledPySpark Developerwith hands-on experience inDatabricksto join Sompo's IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on theDatabricks Unified Analytics Platform. The ideal candidate will...

  • Databricks SQL Engineer(Pharma Background)

    7 days ago

    Chennai, Tamil Nadu, India Mastek Full time

    Job Title: Databricks SQL Engineer (with Pharma/Life Sciences background)About the Role:We are seeking a highly skilled Databricks SQL Engineer with Pharma background to join our Data Engineering team. The ideal candidate will have strong expertise in Databricks, Pyspark, SQL, Spark SQL, Delta Lake, and data lakehouse architectures, and will be responsible...

  • Databricks Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India NRM Analytix Full time

    We are hiring a Databricks Data Engineer for a remote, full-time role. The ideal candidate should meet the following requirements:1 to 5 years of overall experience in data engineering.At least 1 year of hands-on experience with Databricks technology.Strong working knowledge of Databricks Delta Lake and PySpark.Experience on cloud platforms like AWS or...