Databrick Pyspark

2 days ago

Chennai, India Virtusa Full time

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates, and schema evolution. Optimize data processing workflows for performance, scalability, and cost efficiency. Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality datasets. Implement data quality checks, validation routines, and logging mechanisms. Monitor and debug production jobs using Databricks jobs, notebooks, and clusters. Ensure security, privacy, and compliance standards are met throughout the data lifecycle. Provide guidance and mentorship to junior team members.

Required Skills & Qualifications:

5 to 7 years of experience in Big Data development. Hands-on experience with Databricks (including Workflows, Notebooks, Delta Live Tables, Unity Catalog). Strong programming skills in PySpark and/or Scala. Solid understanding of Delta Lake architecture. Proficient in SQL for data analysis and transformation. Experience with cloud platforms such as Azure (Azure Data Lake, Data Factory, Synapse) or AWS (S3, Glue, Redshift). Familiarity with CI/CD for Databricks deployments (e.g., using GitHub Actions, Azure DevOps). Knowledge of data governance, cataloguing, and security best practices. Experience working in an Agile/Scrum environment.

Preferred Skills:

Experience with Databricks Unity Catalog and Delta Live Tables. Exposure to machine learning workflows in Databricks. Experience with Apache Airflow, Kafka, or other orchestration/messaging tools. Certifications such as Databricks Certified Data Engineer Associate/Professional, Azure, or AWS certification.

About Virtusa

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

  • Etl Databricks

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Develop and maintain a metadata driven generic ETL framework for automating ETL code Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS. InsureMO rating engine experience required. Ingest data from a variety of structured and unstructured sources (APIs, RDBMS, flat files, streaming). Develop and maintain robust data pipelines...

  • PySpark Developer

    5 days ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job Description ROLE RESPONSIBILITIES Data Engineering and Processing: Develop and manage data pipelines using PySpark on Databricks. Implement ETL/ELT processes to process structured and unstructured data at scale. Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks. Databricks Platform Expertise: Experience in...

  • PySpark Developer

    3 weeks ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job DescriptionROLE RESPONSIBILITIESData Engineering and Processing:Develop and manage data pipelines using PySpark on Databricks.Implement ETL/ELT processes to process structured and unstructured data at scale.Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.Databricks Platform Expertise:Experience in Perform Design,...

  • Databricks

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Job SummaryJoin our dynamic team as a Developer where you will leverage your expertise in Spark in Scala Databricks and Amazon technologies to drive innovative solutions. With a hybrid work model and day shifts you will collaborate with cross-functional teams to optimize data processes and enhance our data infrastructure. Your contributions will directly...

  • Databricks

    2 days ago

    Chennai, India Cognizant Technology Solutions Full time

    Job Summary Join our dynamic team as a Developer where you will leverage your expertise in Spark in Scala Databricks and Amazon technologies to drive innovative solutions. With a hybrid work model and day shifts you will collaborate with cross-functional teams to optimize data processes and enhance our data infrastructure. Your contributions will directly...

  • Databricks

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **Job Summary** **Responsibilities** - Lead the design and implementation of data processing pipelines using Spark in Scala and PySpark - Oversee the administration of Databricks Unity Catalog and manage data governance - Provide expertise in Delta Sharing and ensure secure data sharing practices - Develop and maintain Databricks CLI scripts for automation...

  • Azure DataBricks

    2 weeks ago

    Chennai, Tamil Nadu, India Hexaware Technologies Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    We're on the lookout for passionate Azure Data Engineers who want to redefine the future of data with Databricks & Azure.Role: ADB – Data Engineer (Azure, Databricks, ADF, PySpark/Python)Date: Saturday, 13/09/2025Mode: F2F Discussion at Hexaware Technologies , Siruseri ,Chennai ,Bring your expertise in:Datawarehousing & ETL (10+ yrs)Azure, ADF, Synapse,...

  • Azure DataBricks

    2 days ago

    Chennai, India Hexaware Technologies Full time

    We're on the lookout for passionate Azure Data Engineers who want to redefine the future of data with Databricks & Azure. Role: ADB – Data Engineer (Azure, Databricks, ADF, PySpark/Python) Date: Saturday, 13/09/2025 Mode: F2F Discussion at Hexaware Technologies , Siruseri ,Chennai , Bring your expertise in: Datawarehousing & ETL (10+ yrs) Azure, ADF,...

  • Databricks

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **Job Summary** Join our dynamic team as a Developer where you will leverage your expertise in Spark in Scala Databricks and Amazon technologies to drive innovative solutions. With a hybrid work model and day shifts you will collaborate with cross-functional teams to optimize data processes and enhance our data infrastructure. Your contributions will...

  • Pyspark developer

    5 days ago

    Bengaluru, Chennai, Pune, India MFX Infotech Private Limited Full time

    Job Description ROLE SUMMARY We are seeking a highly skilledPySpark Developerwith hands-on experience inDatabricksto join Sompo's IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on theDatabricks Unified Analytics Platform. The ideal candidate...