Azure Databricks Platform Lead Engineer

7 days ago


Kanpur, Uttar Pradesh, India Birlasoft Full time

Birlasoft, a powerhouse where domain expertise, enterprise solutions, and digital technologies converge to redefine business processes. We take pride in our consultative and design thinking approach, driving societal progress by enabling our customers to run businesses with unmatched efficiency and innovation. As part of the CK Birla Group, a multibillion-dollar enterprise, we boast a 12,500+ professional team committed to upholding the Group's 162-year legacy. Our core values prioritize Diversity, Equity, and Inclusion (DEI) initiatives, along with Corporate Sustainable Responsibility (CSR) activities, demonstrating our dedication to building inclusive and sustainable communities. Join us in shaping a future where technology seamlessly aligns with purpose About the Role - Azure Databricks Platform Lead Engineer Location - Noida /Pune Responsibilities - Design and Maintain Data Pipelines: Develop and optimize data pipelines using Azure Data Factory and Databricks. Ensure smooth and efficient data flow through the pipelines. Troubleshooting and Optimization: Identify and resolve issues related to cluster creation, performance, and resource allocation. Optimize existing data pipelines for performance and reliability. Data Quality and Integrity: Implement measures to ensure data quality and integrity across various data sources. Develop custom error handling in notebooks to ensure proper failure reporting. Collaboration and Communication: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. Provide processed and usable data for business decision-making. Security and Compliance: Implement data security best practices, including encryption and role-based access control (RBAC). Ensure compliance with data governance and regulatory requirements. Monitoring and Performance Management: Use tools like Azure Monitor to track data pipeline performance and troubleshoot issues. Manage and monitor cluster performance to avoid overload and service throttling. Managing Databricks Notebooks Across Environments: Use version control systems like Git to manage notebooks across different environments (dev, homologation, prod). Modularize code by breaking down notebooks into reusable modules for easier testing and deployment. Implement a CI/CD pipeline to automate the deployment of notebooks from development to production using tools like Azure DevOps or GitHub Actions. Token Management: Avoid hardcoding tokens; instead, store them in Azure Key Vault and retrieve them programmatically within notebooks. Use service principals for authentication, which is more secure and scalable than using personal access tokens. Compute Engine Selection: Use job clusters instead of all-purpose clusters for running production jobs to reduce costs. Consider using serverless compute options if supported, as they automatically scale according to workload and can be more cost-effective. Implement compute policies to preconfigure clusters for specific purposes, such as development, testing, and production. Streaming Pipelines: Develop and manage streaming pipelines using Azure IoT Hub and Stream Analytics to analyze data for efficiency and handle data coming in from IoT devices. Required Skills - Azure Data Factory Azure Databricks Azure Monitor, APIM Azure Key Vault Azure IoT Hub Azure Stream Analytics Git (Version Control) CI/CD Tools (e.g., Azure DevOps, GitHub Actions) Cluster Management (Job clusters, all-purpose clusters) Serverless Compute Options Data Security (Encryption, Role-Based Access Control) Data Governance and Compliance Databricks Notebooks Service Principals for Authentication



  • Kanpur, Uttar Pradesh, India Birlasoft Full time

    **Birlasoft Overview**Birlasoft is a powerhouse where domain expertise, enterprise solutions, and digital technologies converge to redefine business processes.We take pride in our consultative and design thinking approach, driving societal progress by enabling our customers to run businesses with unmatched efficiency and innovation.As part of the CK Birla...


  • Kanpur, Uttar Pradesh, India Tavant Full time

    About Tavant: With 24+ years of experience building innovative digital products and solutions, Tavant provides impactful results to its customers. It has been the frontrunner in driving digital innovation and tech-enabled transformation across a wide range of industries such as Consumer Lending, Manufacturing, Agtech, Media & Entertainment, and Retail in...

  • Azure Data Engineer

    2 weeks ago


    Kanpur, Uttar Pradesh, India Ampstek Full time

    Job Title: Azure Data EngineerLocation: 100% RemoteJob Type: ContractRequired Skills & Experience:• 8+ years of experience in data engineering, with a focus on schema design and data modeling.• Strong expertise in Azure Data Services (Azure Data Lake, Azure Synapse Analytics, Azure SQL Database, Cosmos DB, Databricks, Azure Data Factory).• Hands-on...


  • Kanpur, Uttar Pradesh, India Technip Energies Full time

    About Technip Energies At Technip Energies, we believe in a better tomorrow and we believe we can make tomorrow better. With approximately 15,000 talented women and men, we are a global and leading engineering and technology company, with a clear vision to accelerate the energy transition. Designing and delivering added value energy solutions is what we...

  • Data Pipeline Lead

    6 days ago


    Kanpur, Uttar Pradesh, India Birlasoft Full time

    **Job Description**Join us at Birlasoft, a leading provider of digital transformation solutions, and become an integral part of our team that empowers businesses to thrive in a rapidly changing world.We take pride in our consultative approach, driving societal progress by enabling customers to run businesses with unmatched efficiency and innovation.About the...

  • Senior Data Engineer

    3 weeks ago


    Kanpur, Uttar Pradesh, India Sigmaways Inc Full time

    As a Sr. Data Engineer you will be a key member of our team, responsible for designing, developing, and maintaining sophisticated software / data platforms in achieving the charter of the engineering group. RESPONSABILITIES:Lead the design and implementation of scalable, efficient, and robust data architectures to meet business needs and analytical...

  • Cloud Data Engineer

    7 days ago


    Kanpur, Uttar Pradesh, India Birlasoft Full time

    **About Us**Birlasoft, a leading provider of digital transformation solutions, empowers businesses to thrive in a rapidly changing world.We take pride in our consultative approach, driving societal progress by enabling customers to run businesses with unmatched efficiency and innovation.Our 12,500+ strong team is committed to upholding the Group's 162-year...


  • Kanpur, Uttar Pradesh, India Infuse Full time

    Company OverviewInfuse is a technology consulting firm that that helps organisations get the most out of their software - we do this by providing our clients with pioneering and cutting-edge Quality Engineering and Agile / DevOps transformation services. We have been established since 2002, are headquartered in London and are fast-growing with great career...


  • Kanpur, Uttar Pradesh, India Navori Labs Full time

    Job DescriptionWe are seeking a skilled Cloud Solutions Architect to join our team. As a key member of our IT department, you will be responsible for designing, implementing, and managing cloud solutions on the Azure platform. Your primary goal will be to ensure optimal cloud performance and cost efficiency while ensuring data security, identity management,...

  • Data Engineer

    23 hours ago


    Kanpur, Uttar Pradesh, India RedDoorz Full time

    We're Hiring a Data Engineer at RedDoorzRedDoorz, Southeast Asia's leading hospitality brand, is looking for a Data Engineer to join our dynamic team If you're passionate about designing scalable data pipelines and driving data infrastructure excellence , we want to hear from you.What You'll Do:Design, build, and maintain scalable ELT pipelines & ...


  • Kanpur, Uttar Pradesh, India Tata Consultancy Services Full time

    Unleash innovation with cloud at TCS Join the Walk-in Drive on 15th Mar-2025 at Hyderabad Synergy Park and Pave your path to value with TCS AI. Cloud Team. Registration time: 9:30 A.M. to 12:30 PM TCS Inviting applications for Power Platform Developer Experience Range- 6 to 10 years Joining location: PAN India Power Apps ( Canvas and Model Driven ) Power...


  • Kanpur, Uttar Pradesh, India RedDoorz Full time

    Our Technology StackWe use a range of technologies to power our data engineering capabilities. These include:* AWS Services: EKS, EC2, S3, RDS, IAM* Data Warehouse Platforms: Snowflake, Databricks, Redshift, BigQuery* Orchestration Tools: Airflow, Prefect, Dagster* DevOps Tools: Jenkins, GitHub Actions, GitLab CIAbout the RoleThis is a fantastic opportunity...

  • Lead Engineer

    2 weeks ago


    Kanpur, Uttar Pradesh, India GE Vernova Full time

    **Job Description Summary**As a member of Engineering Team, you will be responsible for technical excellence in Board design & development and owns complete product design cycle by conceptualization, detail design, simulations, design calculations and prototyping of the electronic boards as products for Steam Power.**Job Description****Roles &...


  • Kanpur, Uttar Pradesh, India CorroHealth Full time

    Hiring AlertWe are looking for highly skilled Site Reliability Engineer (SRE) for our Product Development team based out at Noida LocationOnly Immediate Joiners preferredCandidates who are available for F2F round of interview, can only applyJob DescriptionWe are seeking a highly skilled Site Reliability Engineer (SRE) to join our team. The ideal candidate...


  • Kanpur, Uttar Pradesh, India ServCrust Full time

    Job Description: We are seeking an experienced Senior Cloud-Based Backend Developer who is proficient in developing cloud-based solutions using Spring Boot framework. The successful candidate will be responsible for leading the design, development, testing, deployment, and maintenance of cloud-based applications, microservices, and APIs. Key...


  • Kanpur, Uttar Pradesh, India VeriPark Full time

    We enable financial institutions to become digital leaders.As a professional team of global scale, we work with best clients for great and exciting projects, in an environment where we learn amazing things every day. Each code, each voice, each contribution, each challenge, each success is celebrated here.We welcome candidates who share our values, have the...


  • Kanpur, Uttar Pradesh, India Zoid Technologies Private Limited Full time

    Lead Mechanical EngineerLocation: Noida, Uttar Pradesh (Involves travelling with whole team to various locations across India)Salary CTC : 5 LPA to 8 LPAJob Type: Full Time - On-Site EmployeeApplication Form: You will be considered for the role only if you fill the application form.Company DescriptionZoid is a high-tech R&D startup devoted towards...


  • Kanpur, Uttar Pradesh, India Navori Labs Full time

    ProfileWe are looking for a highly skilled Cloud Solutions Architect to join our team. The ideal candidate will have experience in cloud computing platforms like Microsoft Azure/AWS, knowledge of cloud cost management and optimization, and certifications such as Azure Fundamentals (AZ-900), Azure Administrator (AZ-104), or Azure Solutions Architect (AZ-305)....


  • Kanpur, Uttar Pradesh, India GE Vernova Full time

    **Job Description Summary**Become part of a winning team and help to deliver the Green Energy transition**Job Description**GE Vernova - Grid Systems Integration product line is globally recognized for designing, manufacturing, and delivering customized High Voltage DirectWithin this cross-centers and multicultural organization, the local plant team leader...

  • Lead Engineer

    2 weeks ago


    Kanpur, Uttar Pradesh, India GE Vernova Full time

    **Job Description Summary**The Lead Engineer - Electrical Component (Oil & Gas) will play a key role in supporting the sales and business development by providing technical expertise in the design of electrification solutions for the oil and gas industry. This individual will be responsible for system design, material selection, quantity estimation, various...