Databricks Engineer

3 weeks ago

Bengaluru, India TeamWare Solutions Full time

Job Description : - Assess and understand current-state Informatica ETL jobs, mappings, workflows, and dependencies.- Redesign and implement equivalent data pipelines using Databricks (PySpark/Scala/SQL).- Optimize performance and scalability of the migrated workflows in the Databricks environment.- Collaborate with data architects, DevOps, QA, and business stakeholders to validate functionality and ensure successful migration.- Develop automated testing and validation scripts to verify data integrity post-migration.- Establish CI/CD practices for deployment of Databricks notebooks/jobs.- Document design patterns, migration strategy, and data lineage for audit and and Responsibilities : Assess and Analyze Informatica ETL Workflows : - Evaluate existing Informatica PowerCenter jobs, mappings, workflows, and interdependencies to understand business logic and data flow.- Redesign ETL Pipelines in Databricks- Re-engineer Informatica-based ETL processes into scalable, maintainable data pipelines using Databricks (PySpark, Scala, SQL), aligning with modern data architecture principles.Performance Tuning and Optimization : - Enhance performance, scalability, and resource utilization of Spark-based jobs through optimization techniques such as caching, partitioning, and efficient joins.Cross-functional Collaboration : - Work with data architects, DevOps, QA, and business stakeholders to validate functionality and ensure the successful migration of ETL workloads.Automated Testing and Data Validation : - Design and implement automated test scripts to validate data quality, transformation logic, and integrity post-migration.CI/CD Implementation : - Establish robust CI/CD pipelines for Databricks notebooks and jobs using tools like Azure DevOps, GitHub Actions, or Jenkins, enabling version control and seamless deployments.- Documentation and Compliance- Create comprehensive documentation covering design patterns, migration strategies, data lineage, and operational runbooks to support audit, compliance, and ongoing maintenance. (ref:hirist.tech)

  • Engineering Manager

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 2,00,00,000 - ₹ 2,50,00,000 per year

    Sr Engineering Manager - Backend P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Engineering Manager

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Engineering Manager (Search Platform) P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Sr Engineering Manager

    6 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    P-995At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Staff Software Engineer

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 2,00,00,000 - ₹ 2,50,00,000 per year

    P-1346At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Staff Software Engineer

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Staff Software Engineer - Data Platform P-375 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our...

  • Staff Software Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 2,00,000 - ₹ 4,00,000 per year

    P-375At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Senior Software Engineer

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    P-1346At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Engineering Manager

    2 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 15,00,000 - ₹ 30,00,000 per year

    P-995At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Senior Backline Engineer

    2 days ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    P-1386At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Staff Software Engineer

    1 week ago

    Bengaluru, Karnataka, India Databricks Full time ₹ 2,00,00,000 - ₹ 2,50,00,000 per year

    P-375At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...