Lead Data Engineer

3 months ago

Bangalore, India KOGNIVERA IT SOLUATION Full time

Functional Title : Lead GCP Engineer

Location : MG Road

Start Date : ASAP

Work mode : 2 days/week in office

Job Description :

About the Role: We are seeking a Lead GCP Data Engineer who can demonstrate a broad range of technical skills across Google Cloud Platform (GCP) and third-party technologies. As a pivotal member of our team, you will be responsible for designing robust data architectures, developing efficient data pipelines, and optimizing data processing workflows on GCP.


Your expertise will be instrumental in driving our data engineering initiatives forward, ensuring scalability, reliability, and performance.

Key Responsibilities :

1. Data Architecture Design :


- Design scalable and efficient data architectures on GCP that meet the organization's data processing and analysis requirements.


- Collaborate closely with data scientists, business analysts, and stakeholders to define effective data models and structures.

2. Data Pipeline Development :


- Develop and implement data pipelines using GCP services such as Google Cloud Storage, BigQuery, Dataflow, and Pub/Sub.


- Ensure data quality, reliability, and governance throughout the data lifecycle.

3. Data Transformation and Integration :


- Utilize technologies like Apache Beam, Apache Spark, and Cloud Dataproc to transform and integrate data from diverse sources.


- Perform data cleansing, aggregation, enrichment, and normalization to support downstream applications and analytics.

4. Performance Optimization :


- Optimize data processing workflows to enhance performance and efficiency.


- Monitor pipelines, identify bottlenecks, and implement optimizations such as improved data partitioning, sharding, and leveraging GCP's autoscaling capabilities.

5. Continuous Improvement :


- Stay abreast of advancements in data engineering and cloud technologies.


- Explore and implement new GCP features and services to enhance data processing capabilities and drive innovation within the team.

6. Research and Innovation :


- Conduct research on emerging data engineering technologies, tools, and best practices.


- Evaluate new methodologies to improve data engineering processes and bring innovative solutions to the organization.

7. Task Automation :


- Automate data engineering tasks using scripting, workflows, or tools like Cloud Composer and Cloud Functions.


- Streamline data ingestion, transformation, monitoring, and other operational processes to improve efficiency and reduce manual effort.

Attributes & Competencies :

Education : BE/BTech, MTech, or MCA.

Experience :


- Minimum 7+ years in development/migration projects, with at least 3 years focused on GCP.


- Experience in GCP-based Big Data deployments (batch/real-time) using BigQuery, BigTable, Google Cloud Storage, Pub/Sub, Data Fusion, Dataflow, Dataproc, and Airflow.

Technical Skills :

- Google Certified Professional Cloud Architect with experience automating and orchestrating workloads on GCP or other public clouds.

- Proficiency in at least one configuration management system (Chef, Puppet, Ansible, Salt, etc.).

- Strong programming skills in Python, Go, with expertise in Git and Git workflows.

- Demonstrated proficiency in CI/CD tools such as Jenkins, TeamCity, or Spinnaker

(ref:hirist.tech)

  • Data Specialist

    1 week ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 287764  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Specialist / Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Data...

  • Data Engineering Lead

    5 days ago

    bangalore, India ofi Full time

    Data Engineering Lead At ofi , we are on a mission to leverage data to drive innovation and business transformation. We are looking for a Data Engineering Lead to spearhead our data engineering team and contribute to our mission of delivering data-driven solutions. As a Data Engineering Lead, you will be responsible for managing the data engineering team...

  • Digital Engineering Lead Engineer

    2 months ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 288920  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Bangalore, Karnātaka (IN-KA), India...

  • Data Engineer

    2 weeks ago

    bangalore, India NTT DATA Full time

    Job Description Data Engineer Grade 07 ResponsibilitiesAssure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements using Azure data factory and Azure databricksDevelop and maintain innovative Azure Azure data factory and Azure databricksSolution design using Microsoft...

  • Data Engineer

    1 week ago

    bangalore, India NTT DATA Full time

    Job Description Data Engineer Grade 07ResponsibilitiesAssure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements using Azure data factory and Azure databricksDevelop and maintain innovative Azure Azure data factory and Azure databricksSolution design using Microsoft Azure...

  • Data Engineering Lead

    7 days ago

    bangalore, India ofi Full time

    Data Engineering LeadAt ofi, we are on a mission to leverage data to drive innovation and business transformation. We are looking for a Data Engineering Lead to spearhead our data engineering team and contribute to our mission of delivering data-driven solutions. As a Data Engineering Lead, you will be responsible for managing the data engineering team and...

  • Data Engineering Lead

    1 week ago

    bangalore, India ofi Full time

    Data Engineering LeadAtofi , we are on a mission to leverage data to drive innovation and business transformation. We are looking for a Data Engineering Lead to spearhead our data engineering team and contribute to our mission of delivering data-driven solutions. As a Data Engineering Lead, you will be responsible for managing the data engineering team and...

  • Lead Data Engineer

    4 days ago

    bangalore, India Papa Johns Global Capability Center Full time

    The Lead Data Analytics Engineer will develop and implement the strategies for the data layer for Papa John’s analytics stack, top to bottom. He/she will work closely with multiple disciplines from data engineering to data analytics and visualization across all stages of data maturity for the purpose of delivering robust Business Intelligence solutions....

  • Data Engineering Lead

    1 week ago

    bangalore, India ofi Full time

    At ofi, we are on a mission to leverage data to drive innovation and business transformation. We are looking for a Data Engineering Lead to spearhead our data engineering team and contribute to our mission of delivering data-driven solutions. As a Data Engineering Lead, you will be responsible for managing the data engineering team and overseeing the design...

  • Digital Engineering Lead Engineer

    4 weeks ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 295706  NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). "Job...

  • Lead Data Engineer

    4 weeks ago

    bangalore, India SMC Squared India Full time

    The Lead Data Analytics Engineer will develop and implement the strategies for the data layer for Papa John’s analytics stack, top to bottom. He/she will work closely with multiple disciplines from data engineering to data analytics and visualization across all stages of data maturity for the purpose of delivering robust Business Intelligence solutions....

  • Lead Data Engineer

    1 week ago

    bangalore, India Papa Johns Global Capability Center Full time

    The Lead Data Analytics Engineer will develop and implement the strategies for the data layer for Papa John’s analytics stack, top to bottom. He/she will work closely with multiple disciplines from data engineering to data analytics and visualization across all stages of data maturity for the purpose of delivering robust Business Intelligence solutions....

  • Lead Data Engineer

    1 week ago

    bangalore, India Papa Johns Global Capability Center Full time

    The Lead Data Analytics Engineer will develop and implement the strategies for the data layer for Papa John’s analytics stack, top to bottom. He/she will work closely with multiple disciplines from data engineering to data analytics and visualization across all stages of data maturity for the purpose of delivering robust Business Intelligence solutions....

  • Lead Data Engineer

    1 week ago

    bangalore, India Intelliserve Konsulting Pvt Ltd Full time

    Organization:- Leading global management consulting firm(One of the Big 3)(Client of Intelliserve) Job type :- Permanent with client Exp_ 7-9 years Job Location :- Bengaluru What You'll Do YOU'RE GOOD AT Leading and Delivering Analysis: Leading data engineering initiatives by providing original analysis and insights to project teams. You will own and...

  • Digital Engineering Lead Engineer

    4 weeks ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 292803  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Bangalore, Karnātaka (IN-KA), India...

  • Data Engineering Lead

    4 weeks ago

    bangalore, India Genpact Full time

    With a startup spirit and 115,000 + curious and courageous minds, we have the expertise to go deep with the world’s biggest brands—and we have fun doing it! We dream in digital, dare in reality, and reinvent the ways companies work to make an impact far bigger than just our bottom line. We’re harnessing the power of technology and humanity to create...

  • Lead Data Engineer

    2 weeks ago

    bangalore, India Intelliserve Konsulting Pvt Ltd Full time

    Organization:- Leading global management consulting firm(One of the Big 3)(Client of Intelliserve)Job type :- Permanent with clientExp_ 7-9 yearsJob Location :- BengaluruWhat You'll DoYOU'RE GOOD ATLeading and Delivering Analysis: Leading data engineering initiatives by providing original analysis and insights to project teams. You will own and oversee...

  • Lead Data Engineer

    19 hours ago

    Bangalore, India Intelliserve Konsulting Pvt Ltd Full time

    Organization:- Leading global management consulting firm(One of the Big 3)(Client of Intelliserve) Job type :- Permanent with client Exp_ 7-9 years Job Location :- Bengaluru What You'll Do YOU'RE GOOD AT Leading and Delivering Analysis: Leading data engineering initiatives by providing original analysis and insights to project teams. You will...

  • Lead Data Engineer

    1 week ago

    bangalore, India Intelliserve Konsulting Pvt Ltd Full time

    Organization:- Leading global management consulting firm(One of the Big 3)(Client of Intelliserve)Job type :- Permanent with client Exp_ 7-9 years Job Location :- BengaluruWhat You'll DoYOU'RE GOOD ATLeading and Delivering Analysis: Leading data engineering initiatives by providing original analysis and insights to project teams. You will own and oversee...

  • Lead Data Engineer

    2 weeks ago

    bangalore, India Intelliserve Konsulting Pvt Ltd Full time

    Organization:- Leading global management consulting firm(One of the Big 3)(Client of Intelliserve) Job type :- Permanent with client Exp_ 7-9 years Job Location :- Bengaluru What You'll Do YOU'RE GOOD AT Leading and Delivering Analysis: Leading data engineering initiatives by providing original analysis and insights to project teams. You will own and...