Databrick Pyspark

20 hours ago

Chennai, Tamil Nadu, India Virtusa Full time ₹ 5,00,000 - ₹ 15,00,000 per year

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates, and schema evolution. Optimize data processing workflows for performance, scalability, and cost efficiency. Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality datasets. Implement data quality checks, validation routines, and logging mechanisms. Monitor and debug production jobs using Databricks jobs, notebooks, and clusters. Ensure security, privacy, and compliance standards are met throughout the data lifecycle. Provide guidance and mentorship to junior team members.

Required Skills & Qualifications:

5 to 7 years of experience in Big Data development. Hands-on experience with Databricks (including Workflows, Notebooks, Delta Live Tables, Unity Catalog). Strong programming skills in PySpark and/or Scala. Solid understanding of Delta Lake architecture. Proficient in SQL for data analysis and transformation. Experience with cloud platforms such as Azure (Azure Data Lake, Data Factory, Synapse) or AWS (S3, Glue, Redshift). Familiarity with CI/CD for Databricks deployments (e.g., using GitHub Actions, Azure DevOps). Knowledge of data governance, cataloguing, and security best practices. Experience working in an Agile/Scrum environment.

Preferred Skills:

Experience with Databricks Unity Catalog and Delta Live Tables. Exposure to machine learning workflows in Databricks. Experience with Apache Airflow, Kafka, or other orchestration/messaging tools. Certifications such as Databricks Certified Data Engineer Associate/Professional, Azure, or AWS certification.

About Virtusa

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

  • Azure DataBricks

    1 day ago

    Chennai, Tamil Nadu, India Hexaware Technologies Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    We're on the lookout for passionate Azure Data Engineers who want to redefine the future of data with Databricks & Azure.Role: ADB – Data Engineer (Azure, Databricks, ADF, PySpark/Python)Date: Saturday, 13/09/2025Mode: F2F Discussion at Hexaware Technologies , Siruseri ,Chennai ,Bring your expertise in:Datawarehousing & ETL (10+ yrs)Azure, ADF, Synapse,...

  • Databricks Developers

    24 hours ago

    Chennai, Tamil Nadu, India Cynosure Corporate Solutions Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Key Responsibilities:Develop and optimize data pipelines in Databricks for transforming and processing data from various sources.Integrate data using Unity Catalog and external data sources (data lakes, APIs, etc.).Write Spark SQL and PySpark scripts for data transformations, optimizations, and creating views/procedures.Perform data analysis to identify...

  • Databricks spark

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job SummaryThe Sr. Developer role is crucial for driving innovation and efficiency in our hybrid work model. With a focus on Kafka Python Databricks SQL Databricks Workflows and PySpark the candidate will enhance our data processing capabilities. Experience in Claims and Billing domains is advantageous. This position requires 8 to 9 years of experience and...

  • Databricks Developer

    1 day ago

    Chennai, Tamil Nadu, India VySystems Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Job Title: Databricks DeveloperNotice Period: Immediate to 15 DaysLocation: Pan India (Remote)Experiences: 5 to 9 YearsJob SummaryWe are seeking a highly skilledDatabricks Developerwith expertise inPython, PySpark, API Design, Azure, Airflow, and SQL. The ideal candidate will design, build, and optimize large-scale data processing solutions, ensuring high...

  • Pyspark Data Engineer

    21 hours ago

    Chennai, Tamil Nadu, India Prowesstics Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for a skilledData Engineerwith hands-on experience inPySparkandDatabricksto design, develop, and optimize large-scale data pipelines. The ideal candidate should have strong problem-solving skills, experience working in cloud-based environments, and the ability to collaborate effectively with cross-functional teams.Key Responsibilities:Develop...

  • Databricks Data Engineer

    7 days ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    About the RoleThe Databricks Data Engineer will play a crucial role in migrating and optimizing data workflows, ensuring high performance and reliability in our data operations.Experience- 3 to 6 yearsLocation - Kolkata /Chennai/TrivandrumRequired Technical Skill Set ------ Databricks, AWS, Python & PySparkGood to have- Experience in AWS platform (S3, EC2,...

  • Azure Data Factory,Azure Python, Pyspark, Databricks, SQL Professional

    2 weeks ago

    Chennai, Tamil Nadu, India Suzva Software Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Experience with Azure datafactory,Azure ,Python, Pyspark, Databricks, SQL Key ResponsibilitiesDesign and build ETL/ELT pipelines to ingest, transform, and load data from clinical, omics, research, and operational sources.Optimize performance and scalability of data flows using tools like Apache Spark, Databricks, or AWS Glue.Collaborate with domain experts...

  • Azure Databricks Lead

    5 days ago

    Chennai, Tamil Nadu, India GIG Consultants Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description : Key Responsibilities : - Lead the design, development, and optimization of large-scale data solutions using Azure Databricks, Azure Data Factory (ADF), and Azure Synapse Analytics. - Architect and implement efficient ETL/ELT pipelines for ingesting, transforming, and processing structured and unstructured data. - Develop...

  • Pyspark - Machine Learning

    4 days ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    7+ years of experience in Big Data with strong expertise in Spark and ScalaMandatory Skills: Big Data Primarily Spark and ScalaStrong Knowledge in HDFS, Hive, Impala with knowledge on Unix , Oracle, Autosys,Good to Have : Agile Methodology and Banking ExpertiseStrong Communication SkillsNot limited to Spark batch, need Spark streaming experienceNo SQL DB...

  • DataBricks / AWS Redshift

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key responsibilities include designing developing and maintaining scalable data pipelines using Apache Spark on Databricks Writing efficient and production ready PySpark or Scala code for data transformation and ETL processes Integrating data from various structured and unstructured sources into a unified platform Implementing Delta Lake and managing data...