Pyspark+databricks

4 days ago

Chennai Tamil Nadu, India Cognizant Full time

**Exp: 4 to 13 years**

**Skill: Data Bricks+Pyspark**

**Location : Bangalore/Hyderabad/Kolkota/Pune/Chennai**

**Technical Skills**:
Python,PySpark,Azure Data Lake Store,Databricks Workflows,Databricks SQL

**Responsibilities**:

- Develop and optimize data solutions using Azure Data Lake Store to support business needs.
- Utilize Python to create efficient and scalable data processing scripts.
- Implement and manage Databricks SQL for querying and analyzing large datasets.
- Design and maintain Databricks Workflows to automate data processing tasks.
- Leverage PySpark to perform large-scale data transformations and analytics.
- Collaborate with cross-functional teams to understand data requirements and deliver solutions.
- Ensure data integrity and quality through rigorous testing and validation processes.
- Provide technical guidance and support to junior developers and team members.
- Monitor and troubleshoot data pipelines to ensure smooth operation.
- Stay updated with the latest industry trends and technologies to continuously improve data solutions.
- Document all development processes and workflows for future reference.
- Contribute to the overall data strategy and architecture of the organization.
- Drive innovation by exploring new tools and techniques to enhance data capabilities.
- Qualifications
- Possess strong experience in Azure Data Lake Store, demonstrating the ability to manage and optimize large data repositories.
- Have extensive knowledge of Python, with a proven track record of developing efficient data processing scripts.
- Show proficiency in Databricks SQL, capable of querying and analyzing complex datasets.
- Demonstrate expertise in designing and maintaining Databricks Workflows for automated data processing.
- Exhibit strong skills in PySpark, with experience in large-scale data transformations and analytics.
- Nice to have experience in Investment Banking Operations, providing valuable domain insights.

  • Databricks + Pyspark

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines using PySpark and Databricks for ETL processing of large volumes of data. Cloud Integration: Develop solutions leveraging Databricks on cloud platforms (AWS/Azure/GCP) to process and analyze data in a distributed computing environment. Data Modeling: Build robust...

  • databricks/pyspark

    3 days ago

    Chennai, Tamil Nadu, India NRM Analytix Full time ₹ 4,00,000 - ₹ 12,00,000 per year

    Responsibilities:* Design, develop & maintain PySpark solutions using Databricks platform* Optimize performance through efficient data processing techniques* Collaborate with cross-functional teams on project deliveryAccessible workspace

  • Databrick Pyspark

    2 weeks ago

    Chennai, India Virtusa Full time

    Key Responsibilities: Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates,...

  • Databrick Pyspark

    1 week ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Key Responsibilities:Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates,...

  • Etl Databricks with Aws

    1 week ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Develop and maintain a metadata driven generic ETL framework for automating ETL code Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS. Ingest data from a variety of structured and unstructured sources (APIs, RDBMS, flat files, streaming). Develop and maintain robust data pipelines for batch and streaming data using Delta...

  • Pyspark

    6 days ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **Job Summary** **Responsibilities** - Develop and maintain data solutions using Databricks SQL Databricks Delta Lake and Databricks Workflows. - Optimize and enhance existing data workflows to improve performance and efficiency. - Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs. -...

  • Databrick

    7 days ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Key Responsibilities: Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates,...

  • Pyspark/databricks

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Bachelors degree or equivalent in Computer Engineering, Computer Science, or a related field. 5+ years of experience in data / software engineering role. 3+ years of experience with building AWS or Azure cloud-based data pipelines and AI solutions. 3+ years of experience with Python and Spark. Strong experience with Databricks, including Spark-based...

  • Pyspark & Databricks

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Bachelors degree or equivalent in Computer Engineering, Computer Science, or a related field. 7+ years of experience in data / software engineering role. 3+ years of experience with building AWS or Azure cloud-based data pipelines and AI solutions. 3+ years of experience with Python and Spark. Strong experience with Databricks, including Spark-based...

  • Data Engineer

    3 hours ago

    Chennai, Tamil Nadu, India NielsenIQ Full time

    Junior Data Engineer - Azure Databricks Pyspark Python Airflow Chennai Pune India 1- 2 years exp only YOU LL BUILD TECH THAT EMPOWERS GLOBAL BUSINESSES Our Connect Technology teams are working on our new Connect platform a unified global open data ecosystem powered by Microsoft Azure Our clients around the world rely on Connect data and insights to innovate...