Etl Databricks with Aws

3 days ago

Chennai Tamil Nadu, India Virtusa Full time

Develop and maintain a metadata driven generic ETL framework for automating ETL code
Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark/SQL) on AWS.
Ingest data from a variety of structured and unstructured sources (APIs, RDBMS, flat files, streaming).
Develop and maintain robust data pipelines for batch and streaming data using Delta Lake and Spark Structured Streaming.
Implement data quality checks, validations, and logging mechanisms.
Optimize pipeline performance, cost, and reliability.
Collaborate with data analysts, BI, and business teams to deliver fit for purpose datasets.
Support data modeling efforts (star, snowflake schemas) de norm tables approach and assist with data warehousing initiatives.
Work with orchestration tools Databricks Workflows to schedule and monitor pipelines.
Follow best practices for version control, CI/CD, and collaborative development

**Skills**:Hands-on experience in ETL/Data Engineering roles.
Strong expertise in Databricks (PySpark, SQL, Delta Lake), Databricks Data Engineer Certification preferred
Experience with Spark optimization, partitioning, caching, and handling large-scale datasets.
Proficiency in SQL and scripting in Python or Scala.
Solid understanding of data lakehouse/medallion architectures and modern data platforms.
Experience working with cloud storage systems like AWS S3
Familiarity with DevOps practices Git, CI/CD, Terraform, etc.
Strong debugging, troubleshooting, and performance-tuning skills.

**About Virtusa**

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

  • Python/aws/databricks

    3 days ago

    Chennai, India Resource Tree Full time

    Job Name : Data Engineers with AWS and Databricks experience Role : Data Engineer Min years of experience : 3+ Location : Chennai/WFH, Open Positions : 2 Technology Requirement **:Databricks, SQL, Python, AWS **(Glue, S3, Lambda, EC2, SageMaker) Responsibilities: - Develop and implement data lake solutions and analytics solution using AWS and...

  • Databrick

    1 day ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Key Responsibilities: Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks. Write efficient and production-ready PySpark or Scala code for data transformation and ETL processes. Integrate data from various structured and unstructured sources into a unified platform. Implement Delta Lake and manage data versioning, updates,...

  • Databricks SQL Engineer

    2 weeks ago

    Chennai, India Mastek Ltd Full time

    Job Title : Databricks SQL Engineer (with Pharma/Life Sciences background)About the Role :We are seeking a highly skilled Databricks SQL Engineer with Pharma background to join our Data Engineering team. The ideal candidate will have strong expertise in Databricks, Pyspark, SQL, Spark SQL, Delta Lake, and data lakehouse architectures, and will be responsible...

  • 15h Left! Aws Data Engineer

    2 days ago

    Chennai, Tamil Nadu, India virtusa Full time

    AWS Data Engineer Design and build scalable data pipelines using AWS services like AWS Glue Amazon Redshift and S3 Develop efficient ETL processes for data extraction transformation and loading into data warehouses and lakes Create and manage applications using Python SQL Databricks and various AWS technologies Automate repetitive tasks and build...

  • Databricks with Aws(Datadog)

    3 days ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Job Description: Were looking for an Engineer to support AWACs SRE tool migration initiative. Youll help configure and maintain new monitoring tools, replacing Datadog and LogicMonitor, and ensure visibility across AWS, Azure, and Databricks environments. Ideal for someone with hands on experience in observability and a passion for modern tooling. Key...

  • Databricks + Pyspark

    1 week ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines using PySpark and Databricks for ETL processing of large volumes of data. Cloud Integration: Develop solutions leveraging Databricks on cloud platforms (AWS/Azure/GCP) to process and analyze data in a distributed computing environment. Data Modeling: Build robust...

  • Etl Tester

    1 week ago

    Chennai, Tamil Nadu, India Ambc Technologies Full time

    **Position**: ETL tester **Location**: Chennai **Experience**: 6-8 Years **Responsibilities**: - Create and execute test plans to ensure the accuracy of data processing pipelines in Data Bricks. - Collaborate with different teams to understand requirements and validate data transformation. - Test ETL workflows for functionality, integration, and...

  • Databricks & AWS Lead

    6 days ago

    Bengaluru, Chennai, Hyderabad, India Krazy Mantra HR Solutions Pvt. Ltd Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    We are looking for a skilled Databricks & AWS Lead with 7-14 years of experience to join our team in Pune, Mumbai, Chennai, Hyderabad, Bangalore, and Noida. The ideal candidate will have expertise in Databricks and AWS, Unity Catalog, Spark, SQL, and Python.Roles and ResponsibilityDesign and implement data pipelines using Databricks and AWS.Develop and...

  • Databricks Developers

    2 weeks ago

    Chennai, India Cynosure Corporate Solutions Full time

    Key Responsibilities: Develop and optimize data pipelines in Databricks for transforming and processing data from various sources. Integrate data using Unity Catalog and external data sources (data lakes, APIs, etc.). Write Spark SQL and PySpark scripts for data transformations, optimizations, and creating views/procedures. Perform data analysis to identify...

  • Databricks Developers

    1 day ago

    Chennai, Tamil Nadu, India Cynosure Corporate Solutions Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Key Responsibilities:Develop and optimize data pipelines in Databricks for transforming and processing data from various sources.Integrate data using Unity Catalog and external data sources (data lakes, APIs, etc.).Write Spark SQL and PySpark scripts for data transformations, optimizations, and creating views/procedures.Perform data analysis to identify...