Data Engineering Lead

1 month ago

india Kanerika Software Full time
Job Description

Role 

This position will be engaged in design and deploy of large-scale data processing frameworks and pipelines on Data bricks to support enterprise-wide BI and data science initiatives. It will also provide user training and data coaching for effective Data bricks usage. 

 

Locations 

 

We are located at Austin (USA), Singapore, Guru-gram (India), Ahmedabad (India) and Hyderabad (India).  

 

Job Location – Hyderabad, Indore, Ahmadabad, Pune, Gurgaon 

  

 

We are looking for you  

 

Are you passionate about leveraging data to drive business insights and innovation? Do you thrive in a dynamic, collaborative environment where you can contribute your expertise to building scalable data solutions? Data Bricks is seeking a talented Data Engineer to join our team and play a pivotal role in designing and implementing data pipelines and infrastructure for our cloud-based analytics platform. 



Requirements

What you will do 

 

  • Create ML-Ops frameworks for model development, training, deployment, and monitoring. 

  • Optimize Data-bricks clusters for performance, scalability, and resource efficiency. 

  • Monitor cluster health, performance metrics, and resource usage, adjusting cluster sizes as needed. 

  • Design efficient data processing systems and pipelines using Data bricks, APIs, and other cloud services. 

  • Act as the Data bricks expert and uphold platform standards. 

  • Collaborate with Data and Platform architects on defining the product roadmap. 

  • Build and maintain scalable infrastructure on cloud platforms such as AWS, Azure, or Google Cloud to support data processing and analytics workloads. 

  • Implement data governance best practices to ensure data quality, reliability, and security throughout the data lifecycle. 

  • Monitor and troubleshoot data pipelines to identify and resolve performance bottlenecks, data integrity issues, and other operational challenges. 

 

What you’ll bring 

 

  • Bachelor’s degree in computer science or related field. 

  • 7+ years of IT experience, including 5+ years in Data bricks development and administration, Spark, and cloud platforms (preferably Azure). 

  • Proficiency in Databricks admin and development, with expertise in Resource and Performance Management. 

  • Expertise in test automation, continuous integration, Docker, GitHub, Jenkins, and Unix/Linux. 

  • Leadership in large project initiatives, willingness to mentor, and support for team members from different time zones. 

 

You will have advantage if you bring -   

 

  • Experience with Kafka and Spark, and strong in Python and Pyspark. 

  • Hands-on experience in machine learning pipelines and scripting languages. 

  • Familiarity with Java, Python, SQL/NoSQL databases, and Big Data tools. 

  • Passion for new tech, continuous improvement, and collaborative problem-solving. 

  • Proactive, strong work ethic, and team player. 


Benefits

1. Culture: 

  • Open Door Policy: Encourages open communication and accessibility to management. 

  • Open Office Floor Plan: Fosters a collaborative and interactive work environment. 

  • Flexible Working Hours: Allows employees to have flexibility in their work schedules. 

  • Employee Referral Bonus: Rewards employees for referring qualified candidates. 

  • Appraisal Process Twice a Year: Provides regular performance evaluations and feedback. 

2. Inclusivity and Diversity: 

  • Hiring practices that promote diversity: Ensures a diverse and inclusive workforce. 

  • Mandatory POSH training: Promotes a safe and respectful work environment. 

3. Health Insurance and Wellness Benefits: 

  • GMC and Term Insurance: Offers medical coverage and financial protection. 

  • Health Insurance: Provides coverage for medical expenses. 

  • Disability Insurance: Offers financial support in case of disability. 


Requirements
What you will do Create ML-Ops frameworks for model development, training, deployment, and monitoring. Optimize Data-bricks clusters for performance, scalability, and resource efficiency. Monitor cluster health, performance metrics, and resource usage, adjusting cluster sizes as needed. Design efficient data processing systems and pipelines using Data bricks, APIs, and other cloud services. Act as the Data bricks expert and uphold platform standards. Collaborate with Data and Platform architects on defining the product roadmap. Build and maintain scalable infrastructure on cloud platforms such as AWS, Azure, or Google Cloud to support data processing and analytics workloads. Implement data governance best practices to ensure data quality, reliability, and security throughout the data lifecycle. Monitor and troubleshoot data pipelines to identify and resolve performance bottlenecks, data integrity issues, and other operational challenges. What you’ll bring Bachelor’s degree in computer science or related field. 7+ years of IT experience, including 5+ years in Data bricks development and administration, Spark, and cloud platforms (preferably Azure). Proficiency in Databricks admin and development, with expertise in Resource and Performance Management. Expertise in test automation, continuous integration, Docker, GitHub, Jenkins, and Unix/Linux. Leadership in large project initiatives, willingness to mentor, and support for team members from different time zones. You will have advantage if you bring - Experience with Kafka and Spark, and strong in Python and Pyspark. Hands-on experience in machine learning pipelines and scripting languages. Familiarity with Java, Python, SQL/NoSQL databases, and Big Data tools. Passion for new tech, continuous improvement, and collaborative problem-solving. Proactive, strong work ethic, and team player.

  • Data Engineering Lead

    1 week ago

    India Chryselys Full time

    Job Description:We are currently seeking a skilled and experienced individual for the position of Data Engineering Lead. As the Data Engineering Lead, you will have a significant impact on the successful implementation of data projects for our clients. Your proficiency in data engineering and data operations will be crucial in ensuring project completion...

  • Lead Data Engineer

    1 week ago

    India Decision Foundry Full time

    Lead Data Engineer (Salesforce Cloud):Experience: 5-10 years of experience in data engineering and the Salesforce ecosystem.We are looking for a skilled Senior Data Engineer to join our team and have a key role in creating, constructing, and advising on solutions within the Salesforce ecosystem. The perfect candidate should have a solid background in data...

  • Lead Data Engineer

    3 weeks ago

    india Orion Innovation Full time

    As a Lead Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure. You will lead a team of data engineers to ensure the successful delivery of projects and contribute to the overall data strategy. The ideal candidate will have extensive experience in cloud-based data technologies, particularly in Azure,...

  • Lead Data Engineer

    1 week ago

    India Orion Innovation Full time

    As a Lead Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure. You will lead a team of data engineers to ensure the successful delivery of projects and contribute to the overall data strategy. The ideal candidate will have extensive experience in cloud-based data technologies, particularly in Azure,...

  • Lead Data Engineer

    1 month ago

    india Orion Innovation Full time

    As a Lead Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure. You will lead a team of data engineers to ensure the successful delivery of projects and contribute to the overall data strategy. The ideal candidate will have extensive experience in cloud-based data technologies, particularly in Azure,...

  • Data Engineer

    2 months ago

    india dentsu Full time

    The purpose of this role is to maintain, improve, clean and manipulate data in the business’s operational and analytics databases. The Data Engineer works with the business’s software engineers, data analytics teams, data scientists and data warehouse engineers in order to understand and aid in the implementation of database requirements, analyse...

  • Lead data engineer

    4 weeks ago

    india USEReady Full time

    Useready is hiring for Lead Data Engineer(Bangalore/Mohali/Gurugram) Job Description Mentor a team of developers in building and maintaining high-quality, scalable, and secure data pipelines and analytics applications on the Databricks platform Work closely with data engineers, business stakeholders to understand their needs and design and develop...

  • Lead Data Engineer

    4 weeks ago

    india Digitas India Full time

    Role- Lead Data Engineer Experience - 6 +Years Locations- Mumbai/Pune/Ahmedabad Overview: As a Lead Data Engineer, you will play a pivotal role in designing, implementing, and managing complex data pipelines within Google Cloud Platform (GCP) & AWS . Your expertise in integrating SAP, Salesforce and other cloud databases and marketing platforms , will be...

  • Software Test Engineer

    7 hours ago

    India Lead Masters AI Full time

    Company Description LeadMasters.ai is an innovative AI-driven lead generation and ad optimization platform that empowers businesses to maximize their marketing potential through intelligent lead generation and data-driven ad optimization. Our platform leverages advanced artificial intelligence algorithms to analyze top-performing ads across various social...

  • Lead Data Engineer

    3 weeks ago

    india Helius Technologies Full time

    Work Location: Hyderabad Work Mode : 5 days Work from Office Work Timings : 9 am to 6 pm Client: Singapore based Job Type: 1 year renewable contract (Long Term) Kindly revert with below details in addition to updated resume. Full Name as per the Passport or Aadhar Card : Current Salary Exp Salary Notice period (availability) Reason for Change Ok for 1...

  • Snowflake Data Engineer

    4 weeks ago

    india Lapis Data Analytics Full time

    Company Description Lapis Data Analytics is a dynamic and innovative data engineering company founded in 2020 and located in Chennai. Our primary focus is on designing, building, and maintaining robust data pipelines that convert raw data into actionable insights. With a team of seasoned experts and cutting-edge technology, we deliver customized data...

  • Lead Data Engineer

    3 weeks ago

    india Decision Foundry Full time

    Lead Data Engineer (Salesforce Cloud): Experience: 5-10 years of experience in data engineering and the Salesforce ecosystem. We seek an experienced Senior Data Engineer to join our team and play a pivotal role in designing, building, and consulting solutions within the Salesforce ecosystem. The ideal candidate should have a strong background in data...

  • Lead Data Engineer

    4 weeks ago

    india Decision Foundry Full time

    Lead Data Engineer (Salesforce Cloud): Experience: 5-10 years of experience in data engineering and the Salesforce ecosystem. We seek an experienced Senior Data Engineer to join our team and play a pivotal role in designing, building, and consulting solutions within the Salesforce ecosystem. The ideal candidate should have a strong background in data...

  • Azure Lead Data Engineer

    1 month ago

    india IntraEdge Full time

    As Azure Lead Data Engineer, you will have to take over the key activities of designing, coding, and implementing data solutions in the platform. With a focus on leveraging DBT for data transformation and modelling, as well as expertise in MDM tools, you will play a pivotal role in architecting scalable and performant data pipelines and warehouses. You will...

  • Data Engineering Lead

    1 month ago

    india Chryselys Full time

    Job Description: As a Data Engineering Lead, you will play a critical role in driving the delivery for our clients data projects. Leveraging your expertise in data engineering and data operations, you will work closely with clients to ensure that the project delivery on time with quality and meeting the requirements. This role requires a deep understanding...

  • Team Lead- Data Engineering

    4 weeks ago

    india Concentrix Full time

    Experience: 6 Years - 10 Years Job Location: Bengaluru (Hybrid- currently remote) Notice Period: Immediate/ 30 Days (Looking for someone to start from May or Mid June) Shift Timings: 02:00 PM to 11:00 PM (IST)/ 03:00 PM to 12:00 AM (IST) Job Description: We are seeking an experienced and talented Data Engineering Team Lead who can manage a...

  • Team Lead- Data Engineering

    1 week ago

    India Concentrix Full time

    Experience: 6 Years - 10 Years Job Location: Bengaluru (Hybrid- currently remote) Notice Period: Immediate/ 30 Days (Looking for someone to start from May or Mid June) Shift Timings: 02:00 PM to 11:00 PM (IST)/ 03:00 PM to 12:00 AM (IST) Job Description: We are seeking an experienced and talented Data Engineering Team Lead who can manage a high-performing...

  • Lead Engineer

    15 hours ago

    india Padink Engineering Full time

    Job Description Perform piping engineering services such as:a. assist with the preparation of proposals, estimates, budgets and schedules.b. Provide assistance during procurement, construction and commissioning activities. For example, write requisitions, conduct technical bid evaluations.c. Identify issues and problems, develop appropriate solutions.Good...

  • Lead Data Engineer

    1 week ago

    India SeaCross Recruitment Services Full time

    Job Title: Infrastructural Components Developer at LDHAbout the Company: Join a leading company in the tech industry with a focus on infrastructure development.Job Description:Play a key role in developing infrastructural components for the LDH stack with the following responsibilities:Strong coding skills with expertise in Python, Spark, SQL (Postgres), and...

  • Lead Data Engineer

    2 weeks ago

    india UNICO CONNECT PRIVATE LIMITED Full time

    Job Description :Responsibilities for this position include :- Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc) across Fractal and contributes to open source Big Data technologies- Write and tune complex Java, MapReduce, Pig and Hive jobs- Adapt quickly to...