GCP Data Stream

2 days ago

Bangalore, India Tekgence Inc Full time

Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query , GCP DataStream.

Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting

ADO, Jenkins, GitHub

Location- Pan India, hybrid mode 3 days in a week

EXP range- 7-12

Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform )

Job Overview:

We are seeking a skilled Senior Cloud Infrastructure Engineer to automate infrastructure creation on Google Cloud Platform (GCP) using Terraform. The candidate will be responsible for provisioning and setting up GCP projects and environments for the entire GCP data engineering stack, including services such as Big Query, Data Flow, Pub/Sub, and GCP DataStream. The role also requires expertise in configuring pipelines using Azure DevOps to automate and manage the infrastructure provisioning process.

Key Responsibilities:

  • Automate GCP Infrastructure: Design and implement Terraform scripts to provision and automate GCP projects, environments, and data engineering services.
  • -Set Up Infrastructure for GCP DataStream: Configure and deploy the infrastructure for GCP DataStream to support real-time data replication and Change Data Capture (CDC) from on-prem databases like Oracle and *DB2* to BigQuery*.
  • -Provision and Manage the GCP Data Engineering Stack: Set up and optimize infrastructure for *BigQuery, *Dataflow*, Pub/Sub, Cloud Storage, and other data engineering tools on GCP.
  • Automate Infrastructure Pipelines with Azure DevOps*: Design, develop, and maintain CI/CD pipelines using Azure DevOps to automate the deployment of Terraform scripts and manage infrastructure changes.
  • -Optimize Cloud Infrastructure*: Ensure high availability, scalability, and security for cloud resources and data pipelines.
  • Security and Compliance*: Implement best practices for security and compliance across GCP environments, including IAM policies, encryption, and network security.
  • Collaboration: Work closely with data engineers and DevOps teams to align infrastructure with data pipelines and operational requirements.

Required Skills and Experience:

  • -*5+ years of experience in GCP infrastructure automation using Terraform to provision cloud resources and environments.
  1. Experience with GCP DataStream*: Proven ability to set up and configure GCP DataStream for real-time data replication and CDC from on-prem databases (Oracle, DB2) to BigQuery.
  2. Strong expertise in GCP data engineering services: Hands-on experience with BigQuery, Dataflow*, Pub/Sub, Cloud Storage, and related GCP services.
  3. Infrastructure Automation: Proficiency in creating reusable Terraform modules and automating cloud infrastructure using Terraform.
  4. Experience with Azure DevOps: Expertise in configuring and maintaining CI/CD pipelines for infrastructure provisioning using Azure DevOps*.
  5. Networking and Security*: Strong understanding of GCP networking, IAM, VPCs, firewall configurations, and security best practices.
  6. Familiarity with on-prem databases such as Oracle and *DB2*.
  7. Scripting and Automation*: Proficiency in scripting (e.g., Bash, Python) for cloud automation and troubleshooting.

  • Gcp data engineer

    2 weeks ago

    Bangalore, India Tech Mahindra Full time

    Responsibilities: • Design, develop, and maintain data pipelines using GCP services such as Dataflow, Big Query, and Pub/Sub. • Implement and manage data warehouse solutions using Big Query, ensuring data is stored securely and efficiently. • Use Pub/Sub for real-time data ingestion and streaming analytics. • Provision and manage GCP...

  • GCP Data Engineer

    2 weeks ago

    bangalore, India Tech Mahindra Full time

    Responsibilities:• Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub.• Implement and manage data warehouse solutions using BigQuery, ensuring data is stored securely and efficiently.• Use Pub/Sub for real-time data ingestion and streaming analytics.• Provision and manage GCP infrastructure using...

  • GCP Data Engineer

    2 weeks ago

    bangalore, India Tech Mahindra Full time

    Responsibilities: • Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub. • Implement and manage data warehouse solutions using BigQuery, ensuring data is stored securely and efficiently. • Use Pub/Sub for real-time data ingestion and streaming analytics. • Provision and manage GCP infrastructure...

  • Tekgence Inc | GCP Data Stream | bangalore

    3 days ago

    bangalore, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query , GCP DataStream. Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting ADO, Jenkins, GitHub Location- Pan India, hybrid mode 3 days in a week EXP range- 7-12 Job Role: Senior Cloud Infrastructure Engineer (GCP & ...

  • Tekgence Inc | GCP Data Stream | bangalore

    2 days ago

    bangalore, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query, GCP DataStream.Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting ADO, Jenkins, GitHubLocation- Pan India, hybrid mode 3 days in a weekEXP range- 7-12 Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform)...

  • GCP Data Engineer

    1 month ago

    Bangalore, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello , Greetings from RIDIK, We have opening for position GCP Data Engineer Technical Skills · Overall 3-5 years of professional experience · Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components. · Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD...

  • GCP Data Engineer

    1 month ago

    bangalore, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello ,Greetings from RIDIK,We have opening for position GCP Data EngineerTechnical Skills· Overall 3-5 years of professional experience· Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components.· Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD process·...

  • Gcp data engineer

    1 month ago

    Bangalore, India RiDiK Full time

    Hello , Greetings from RIDIK, We have opening for position GCP Data Engineer Technical Skills · Overall 3-5 years of professional experience · Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components. · Build and deploy services using Gradle, Maven, Jenkins, Git Hub Actions etc. as part of...

  • GCP Data Engineer

    2 weeks ago

    bangalore, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello , Greetings from RIDIK, We have opening for position GCP Data Engineer Technical Skills · Overall 3-5 years of professional experience · Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components. · Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD...

  • GCP Data Engineer

    2 weeks ago

    Bangalore Metropolitan Area, India Tech Mahindra Full time

    Responsibilities:• Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub.• Implement and manage data warehouse solutions using BigQuery, ensuring data is stored securely and efficiently.• Use Pub/Sub for real-time data ingestion and streaming analytics.• Provision and manage GCP infrastructure using...

  • Gcp Data Engineer

    1 month ago

    Bangalore City, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello ,Greetings from RIDIK,We have opening for position GCP Data EngineerTechnical Skills· Overall 3-5 years of professional experience· Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components.· Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD process·...

  • Sr. Data Engineer-GCP

    2 weeks ago

    bangalore, India Avikal Solutions Full time

    Sr. Data Engineer-GCP Exp: 4+ Location: Bangalore (Hybrid) Notice: Immediate Joiner Qualifications: ● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms ● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools ● Experience working with Data warehousing tools, including...

  • GCP Data Engineer

    1 month ago

    bangalore, India RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS) Full time

    Hello ,Greetings from RIDIK,We have opening for position GCP Data EngineerTechnical Skills· Overall 3-5 years of professional experience· Experience developing Datapipelines using Dataflow. Integrate with Composer/Airflow or any scheduler Components.· Build and deploy services using Gradle, Maven, Jenkins, GitHubActions etc. as part of CI/CD process·...

  • GCP Data Lead

    1 week ago

    bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data Lead Location: Bangalore, India (Day 1 Onsite) Job Type: Contract Job Description: Strong GCP experience Prior experience in building pipelines using dataflow, data fusion, data proc, etc Very strong python coding skill Experience in optimising GCP resources for cost and performance Experience in containerisation of...

  • Sr. data engineer-gcp

    2 weeks ago

    Bangalore, India Avikal Solutions Full time

    Sr. Data Engineer-GCP Exp: 4+ Location: Bangalore (Hybrid) Notice: Immediate Joiner Qualifications: ● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms ● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools ● Experience working with Data warehousing tools,...

  • Gcp data lead

    7 days ago

    Bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data Lead Location: Bangalore, India (Day 1 Onsite) Job Type: Contract Job Description: Strong GCP experience Prior experience in building pipelines using dataflow, data fusion, data proc, etc Very strong python coding skill Experience in optimising GCP resources for cost and performance Experience in...

  • Sr. Data Engineer-GCP

    2 weeks ago

    bangalore, India Avikal Solutions Full time

    Sr. Data Engineer-GCPExp: 4+Location: Bangalore (Hybrid)Notice: Immediate JoinerQualifications:● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools● Experience working with Data warehousing tools, including DynamoDB,...

  • GCP Data Lead

    1 week ago

    bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data LeadLocation: Bangalore, India (Day 1 Onsite)Job Type: ContractJob Description:Strong GCP experiencePrior experience in building pipelines using dataflow, data fusion, data proc, etcVery strong python coding skillExperience in optimising GCP resources for cost and performanceExperience in containerisation of...

  • Sr. Data Engineer-GCP

    1 week ago

    Bangalore, India Avikal Solutions Full time

    Sr. Data Engineer-GCP Exp: 4+ Location: Bangalore (Hybrid) Notice: Immediate Joiner Qualifications: ● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms ● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools ● Experience working with Data warehousing tools,...

  • Sr. Data Engineer-GCP

    2 weeks ago

    Bangalore, India Avikal Solutions Full time

    Sr. Data Engineer-GCP Exp: 4+ Location: Bangalore (Hybrid) Notice: Immediate Joiner Qualifications: ● 4+ years of experience developing scalable Pyspark applications or solutions on distributed platforms ● Experience in Google Cloud Platform (GCP) and good to have other cloud platform tools ● Experience working with Data warehousing tools,...