Databricks Engineer

2 days ago

Bengaluru, Karnataka, India Teamware Solutions ( A division of Quantum Leap Co Full time ₹ 20,00,000 - ₹ 25,00,000 per year

Job Description :

- Assess and understand current-state Informatica ETL jobs, mappings, workflows, and dependencies.

- Redesign and implement equivalent data pipelines using Databricks (PySpark/Scala/SQL).

- Optimize performance and scalability of the migrated workflows in the Databricks environment.

- Collaborate with data architects, DevOps, QA, and business stakeholders to validate functionality and ensure successful migration.

- Develop automated testing and validation scripts to verify data integrity post-migration.

- Establish CI/CD practices for deployment of Databricks notebooks/jobs.

- Document design patterns, migration strategy, and data lineage for audit and compliance.

Roles and Responsibilities :

Assess and Analyze Informatica ETL Workflows :

- Evaluate existing Informatica PowerCenter jobs, mappings, workflows, and interdependencies to understand business logic and data flow.

- Redesign ETL Pipelines in Databricks

- Re-engineer Informatica-based ETL processes into scalable, maintainable data pipelines using Databricks (PySpark, Scala, SQL), aligning with modern data architecture principles.

Performance Tuning and Optimization :

- Enhance performance, scalability, and resource utilization of Spark-based jobs through optimization techniques such as caching, partitioning, and efficient joins.

Cross-functional Collaboration :

- Work with data architects, DevOps, QA, and business stakeholders to validate functionality and ensure the successful migration of ETL workloads.

Automated Testing and Data Validation :

- Design and implement automated test scripts to validate data quality, transformation logic, and integrity post-migration.

CI/CD Implementation :

- Establish robust CI/CD pipelines for Databricks notebooks and jobs using tools like Azure DevOps, GitHub Actions, or Jenkins, enabling version control and seamless deployments.

- Documentation and Compliance

- Create comprehensive documentation covering design patterns, migration strategies, data lineage, and operational runbooks to support audit, compliance, and ongoing maintenance.

  • Engineering Manager

    2 weeks ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 15,00,000 - ₹ 30,00,000 per year

    Sr Engineering Manager - Backend P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Engineering Manager

    2 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 2,00,00,000 - ₹ 2,50,00,000 per year

    Sr Engineering Manager - Backend P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Engineering Manager

    2 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Engineering Manager (Search Platform) P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Engineering Manager

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Engineering Manager - Backend P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers...

  • Engineering Manager

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    P-995At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Engineering Manager

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    P-995At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Senior Engineering Manager

    2 weeks ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 60,00,000 per year

    Senior Engineering Manager - Backend P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Senior Backline Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Sr Backline Engineer (Apache Spark) Mission Databricks helps data teams solve the world's toughest problems. As the leader in Unified Data Analytics, Databricks helps organizations make all their data ready for analytics, empower data-driven decisions across the organization, and rapidly adopt machine learning to outpace the competition Databricks is...

  • Staff Software Engineer

    2 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 2,00,00,000 - ₹ 2,50,00,000 per year

    P-1346At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Staff Software Engineer

    2 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Staff Software Engineer - Data Platform P-375 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...