GCP Data Stream

1 week ago

Bengaluru, India Tekgence Inc Full time

Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query , GCP DataStream.

Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting

ADO, Jenkins, GitHub

Location- Pan India, hybrid mode 3 days in a week

EXP range- 7-12

Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform )

Job Overview:

We are seeking a skilled Senior Cloud Infrastructure Engineer to automate infrastructure creation on Google Cloud Platform (GCP) using Terraform. The candidate will be responsible for provisioning and setting up GCP projects and environments for the entire GCP data engineering stack, including services such as Big Query, Data Flow, Pub/Sub, and GCP DataStream. The role also requires expertise in configuring pipelines using Azure DevOps to automate and manage the infrastructure provisioning process.

Key Responsibilities:

  • Automate GCP Infrastructure: Design and implement Terraform scripts to provision and automate GCP projects, environments, and data engineering services.
  • -Set Up Infrastructure for GCP DataStream: Configure and deploy the infrastructure for GCP DataStream to support real-time data replication and Change Data Capture (CDC) from on-prem databases like Oracle and *DB2* to BigQuery*.
  • -Provision and Manage the GCP Data Engineering Stack: Set up and optimize infrastructure for *BigQuery, *Dataflow*, Pub/Sub, Cloud Storage, and other data engineering tools on GCP.
  • Automate Infrastructure Pipelines with Azure DevOps*: Design, develop, and maintain CI/CD pipelines using Azure DevOps to automate the deployment of Terraform scripts and manage infrastructure changes.
  • -Optimize Cloud Infrastructure*: Ensure high availability, scalability, and security for cloud resources and data pipelines.
  • Security and Compliance*: Implement best practices for security and compliance across GCP environments, including IAM policies, encryption, and network security.
  • Collaboration: Work closely with data engineers and DevOps teams to align infrastructure with data pipelines and operational requirements.


Required Skills and Experience:

  • -*5+ years of experience in GCP infrastructure automation using Terraform to provision cloud resources and environments.
  1. Experience with GCP DataStream*: Proven ability to set up and configure GCP DataStream for real-time data replication and CDC from on-prem databases (Oracle, DB2) to BigQuery.
  2. Strong expertise in GCP data engineering services: Hands-on experience with BigQuery, Dataflow*, Pub/Sub, Cloud Storage, and related GCP services.
  3. Infrastructure Automation: Proficiency in creating reusable Terraform modules and automating cloud infrastructure using Terraform.
  4. Experience with Azure DevOps: Expertise in configuring and maintaining CI/CD pipelines for infrastructure provisioning using Azure DevOps*.
  5. Networking and Security*: Strong understanding of GCP networking, IAM, VPCs, firewall configurations, and security best practices.
  6. Familiarity with on-prem databases such as Oracle and *DB2*.
  7. Scripting and Automation*: Proficiency in scripting (e.g., Bash, Python) for cloud automation and troubleshooting.

  • GCP Data Stream

    1 week ago

    Bengaluru, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query , GCP DataStream.Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scriptingADO, Jenkins, GitHubLocation- Pan India, hybrid mode 3 days in a weekEXP range- 7-12Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform...

  • GCP Data Stream

    1 week ago

    Bengaluru, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query, GCP DataStream.Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting ADO, Jenkins, GitHubLocation- Pan India, hybrid mode 3 days in a weekEXP range- 7-12 Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform)...

  • GCP Data Stream

    1 week ago

    Bengaluru, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query, GCP DataStream.Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting ADO, Jenkins, GitHubLocation- Pan India, hybrid mode 3 days in a weekEXP range- 7-12 Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform)...

  • GCP Data Engineer

    3 weeks ago

    Greater Bengaluru Area, India Tech Mahindra Full time

    Responsibilities:• Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub.• Implement and manage data warehouse solutions using BigQuery, ensuring data is stored securely and efficiently.• Use Pub/Sub for real-time data ingestion and streaming analytics.• Provision and manage GCP infrastructure using...

  • GCP Data Engineer

    3 weeks ago

    Greater Bengaluru Area, India Tech Mahindra Full time

    Responsibilities: • Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub. • Implement and manage data warehouse solutions using BigQuery, ensuring data is stored securely and efficiently. • Use Pub/Sub for real-time data ingestion and streaming analytics. • Provision and manage GCP infrastructure...

  • Idexcel Technologies

    5 days ago

    Bengaluru, India Idexcel Technologies Full time

    Job Description : Data EngineerWe are looking for an experienced Data Engineer with strong expertise in building and managing data pipelines and streaming solutions.Mandatory Skills : Apache Spark : Strong expertise in Spark-based data processing.Python : Proficiency in developing and maintaining data pipelines.Databases & SQL : Hands-on experience in...

  • Gcp Data Science

    7 days ago

    Bengaluru, Karnataka, India NTT DATA Full time

    **Statistical Modeling**: Strong understanding of statistical techniques and experience leveraging BigQuery ML and Vertex AI for model development. **Machine Learning Algorithms**: Experience with GCP's pre-built ML APIs (e.g., Vision AI, Natural Language API) and custom model building using Vertex AI. **Feature Engineering**: Ability to create meaningful...

  • GCP Data Engineer

    2 months ago

    Bengaluru, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello ,Greetings from RIDIK,We have opening for position GCP Data EngineerTechnical Skills· Overall 3-5 years of professional experience· Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components.· Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD process·...

  • GCP Data Engineer

    1 month ago

    Bengaluru, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello ,Greetings from RIDIK,We have opening for position GCP Data EngineerTechnical Skills· Overall 3-5 years of professional experience· Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components.· Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD process·...

  • GCP Data Engineer

    2 months ago

    Bengaluru, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello , Greetings from RIDIK, We have opening for position GCP Data Engineer Technical Skills · Overall 3-5 years of professional experience · Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components. · Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD...

  • GCP Data Engineer

    2 days ago

    Bengaluru, Karnataka, India Tech Mahindra Full time

    Job DescriptionA talented GCP Data Engineer is required to design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub. The ideal candidate will have a strong proficiency in GCP services and hands-on experience with Terraform for provisioning and managing GCP infrastructure.Responsibilities• Design, develop, and...

  • Sr. data engineer-gcp

    3 weeks ago

    Bengaluru, India Avikal Solutions Full time

    Sr. Data Engineer-GCPExp: 4+Location: Bangalore (Hybrid)Notice: Immediate JoinerQualifications:● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools● Experience working with Data warehousing tools, including Dynamo...

  • Senior Big Data Engineer

    7 days ago

    Bengaluru, Karnataka, India IBCScorp Full time

    At IBCScorp, we are looking for a highly skilled Senior Big Data Engineer to lead our event streaming initiatives using Apache Kafka.As a key member of our engineering team, you will be responsible for designing and implementing data pipelines that integrate various data sources and sinks using Kafka. Your expertise in Kafka Streams and Connect will enable...

  • Gcp data lead

    2 weeks ago

    Bengaluru, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data LeadLocation: Bangalore, India (Day 1 Onsite)Job Type: ContractJob Description:Strong GCP experiencePrior experience in building pipelines using dataflow, data fusion, data proc, etcVery strong python coding skillExperience in optimising GCP resources for cost and performanceExperience in containerisation of...

  • GCP Data Lead

    2 weeks ago

    Bengaluru, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data LeadLocation: Bangalore, India (Day 1 Onsite)Job Type: ContractJob Description:Strong GCP experiencePrior experience in building pipelines using dataflow, data fusion, data proc, etcVery strong python coding skillExperience in optimising GCP resources for cost and performanceExperience in containerisation of...

  • Gcp Data Architect

    2 weeks ago

    Bengaluru, Karnataka, India Cygnus Professionals Inc. Full time

    **Title: GCP Data Architect** **Location: Bangalore India (Hybrid Role)** **Employment Type: Full Time** **Job Description for Data Engineering Architect (GCP & Apache Spark)** **Job Summary**: **Key Requirements**: Over 12 years of professional experience in data engineering, specializing in implementing large-scale enterprise Data Engineering projects...

  • GCP Data Lead

    2 weeks ago

    Bengaluru, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data LeadLocation: Bangalore, India (Day 1 Onsite)Job Type: ContractJob Description:Strong GCP experiencePrior experience in building pipelines using dataflow, data fusion, data proc, etcVery strong python coding skillExperience in optimising GCP resources for cost and performanceExperience in containerisation of...

  • Sr. Data Engineer-GCP

    3 weeks ago

    Bengaluru, India Avikal Solutions Full time

    Sr. Data Engineer-GCPExp: 4+Location: Bangalore (Hybrid)Notice: Immediate JoinerQualifications:● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools● Experience working with Data warehousing tools, including DynamoDB,...

  • Big Data and Gcp

    6 months ago

    Bengaluru, Karnataka, India Manage Captive Solutions Full time

    **Who We Are** Manage Captive Solutions (MCS) stands as the ultimate solution for effortless offshore hiring, precise payroll management, streamlined administration, and stringent compliances. We provide businesses distinctive solutions, making it easy to hire and onboard new team members. By removing the hurdles of setting up a local office, we enable...

  • Sr. Data Engineer-GCP

    3 weeks ago

    Bengaluru, India Avikal Solutions Full time

    Sr. Data Engineer-GCPExp: 4+Location: Bangalore (Hybrid)Notice: Immediate JoinerQualifications: ● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms ● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools ● Experience working with Data warehousing tools, including...