Azure Databricks+ Spark

1 week ago

Bengaluru Karnataka, India Cognizant Full time

**Skill: Azure Databricks+ Spark**

**Experience: 6 to 9 years**

**Location: AIA Kochi**

**Responsibilities**
- Develop and implement scalable data processing solutions using Spark in Scala to enhance data-driven decision-making.
- Manage and administer Databricks Unity Catalog to ensure data governance and security compliance.
- Utilize Delta Sharing to facilitate secure and efficient data sharing across various platforms.
- Configure and maintain Databricks CLI for seamless integration and automation of workflows.
- Design and execute Delta Live Pipelines to streamline data ingestion and transformation processes.
- Implement Structured Streaming solutions to handle real-time data processing and analytics.
- Collaborate with cross-functional teams to integrate risk management strategies into data solutions.
- Leverage Apache Airflow for orchestrating complex data workflows and ensuring timely execution.
- Optimize data storage and retrieval using Amazon S3 and Amazon Redshift to improve performance.
- Develop Python scripts to automate data processing tasks and enhance operational efficiency.
- Utilize Databricks SQL for querying and analyzing large datasets to derive actionable insights.
- Implement Databricks Delta Lake to ensure data reliability and consistency across the platform.
- Manage Databricks Workflows to automate and streamline data engineering processes.

**Qualifications**
- Possess strong expertise in Spark in Scala and Databricks Unity Catalog Admin.
- Demonstrate proficiency in Delta Sharing and Databricks CLI for data management.
- Have experience with Delta Live Pipelines and Structured Streaming for real-time analytics.
- Show capability in risk management and integration with data solutions.
- Be skilled in using Apache Airflow Amazon S3 and Amazon Redshift for data orchestration.
- Exhibit strong Python programming skills for automation and data processing.
- Have a solid understanding of Databricks SQL and Delta Lake for data analysis and reliability.
- Experience with PySpark for distributed data processing is highly desirable.

  • Azure Databricks

    1 week ago

    Bengaluru, Karnataka, India PradeepIT Consulting Services Full time

    **About the job Azure Databricks**: **Position**: Lead - Azure Databricks **Overall IT experience**:8 to 13 years **Primary Skills**:Azure Databricks, Hadoop, Hive, Pig, Impala, ETL, Data warehouse **Secondary Skills**:U-SQL, Spark, Scala, Python **Work Location**:Bengaluru, Hyderabad **Notice Period**:0 to 30 Days **Essential experience**: Experience...

  • Sr Backline Engineer

    4 weeks ago

    Bengaluru, India Databricks Full time

    Mission Databricks helps data teams solve the world’s toughest problems. As the leader in Unified Data Analytics, Databricks helps organizations make all their data ready for analytics, empower data-driven decisions across the organization, and rapidly adopt machine learning to outpace the competition Databricks is venture-backed and founded by the...

  • Sr Backline Engineer

    4 weeks ago

    Bengaluru, India Databricks Full time

    Mission Databricks helps data teams solve the world’s toughest problems. As the leader in Unified Data Analytics, Databricks helps organizations make all their data ready for analytics, empower data-driven decisions across the organization, and rapidly adopt machine learning to outpace the competition Databricks is venture-backed and founded by the...

  • Senior Technical Solutions Engineer

    4 weeks ago

    Bengaluru, India Databricks Full time

    P-1377 Databricks helps data teams solve the world’s toughest problems. As the leader in Unified Data Analytics, Databricks helps organizations make all their data ready for analytics, empower data-driven decisions across the organization, and rapidly adopt machine learning to outpace the competition Databricks is venture-backed and founded by the...

  • Senior Technical Solutions Engineer

    4 weeks ago

    Bengaluru, India Databricks Full time

    P-1377 Databricks helps data teams solve the world’s toughest problems. As the leader in Unified Data Analytics, Databricks helps organizations make all their data ready for analytics, empower data-driven decisions across the organization, and rapidly adopt machine learning to outpace the competition Databricks is venture-backed and founded by the original...

  • Staff Technical Solutions Engineer

    4 weeks ago

    Bengaluru, India Databricks Full time

    P-1364 As a Spark Staff Technical Solutions Engineer, you will provide a deep dive technical and consulting related solutions for the challenging Apache Spark™/ML/AI/Delta/Streaming/Lakehouse reported issues by our customers and resolve any challenges involving the Databricks unified analytics platform with your highly comprehensive technical and customer...

  • Staff Technical Solutions Engineer

    4 weeks ago

    Bengaluru, India Databricks Full time

    P-1364 As a Spark Staff Technical Solutions Engineer, you will provide a deep dive technical and consulting related solutions for the challenging Apache Spark™/ML/AI/Delta/Streaming/Lakehouse reported issues by our customers and resolve any challenges involving the Databricks unified analytics platform with your highly comprehensive technical and customer...

  • Azure Databricks

    1 week ago

    Bengaluru, India Pradeepit Consulting Services Full time

    Job Description - - Experience in building ETL / data warehouse transformation processes Direct experience of building data pipelines using Azure Data Factory and Apache Spark (preferably Databricks) - Experience with other Open-Source big data products Hadoop (incl. Hive, Pig, Impala) Evaluating the performance and applicability of multiple tools against...

  • Data Engineer

    4 days ago

    Bengaluru, India Novotree Minds Full time

    Description :Responsibilities :Data Engineering & Architecture :- Design and implement scalable and optimized data pipelines on Databricks using Delta Lake, PySpark, and SQL.- Develop ETL/ELT frameworks for batch and streaming data processing.- Ensure data quality, governance, and observability using Unity Catalog, Great Expectations, or custom validations.-...

  • Data Engineer

    1 week ago

    Bengaluru, India SKS Enterprises Full time

    Position Name : Data EngineerLocation : BangaloreExperience Range : 3+ YearsMandatory Requirement : Our client is seeking a skilled Data Engineer with a strong foundation in Spark, Databricks, and Data Factory to join their dynamic team.The ideal candidate will have a proven track record in building and maintaining data pipelines, working with large...