Cygnus Professionals Inc. | GCP Data Lead | bangalore

2 weeks ago

bangalore, India Cygnus Professionals Inc. Full time

Role: Google Cloud Platform (GCP) Data Lead

Location: Bangalore, India (Day 1 Onsite)

Job Type: Contract


Job Description:


  • Strong GCP experience
  • Prior experience in building pipelines using dataflow, data fusion, data proc, etc
  • Very strong python coding skill
  • Experience in optimising GCP resources for cost and performance
  • Experience in containerisation of applications
  • CI/CD using GIThub actions
  • Exposure to DataPlex, Data Catalog, and metadata management
  • GCP IAM and role management
  • Design and implement end-to-end data architectures leveraging GCP services (e.g., Big Query, Cloud Storage, Dataflow, Pub/Sub, Cloud Composer ) for large-scale data ingestion and processing.
  • Build and optimize large-scale data pipelines using Apache Spark on GCP (via Dataproc or other Spark services). Ensure high performance and scalability in Spark-based data processing workloads.
  • Develop and manage scalable data ingestion pipelines for structured and unstructured data using tools like Cloud Dataflow , Cloud Pub/Sub , and Apache Spark .
  • Provide guidance for designing secure, scalable, and efficient data solutions on the Google Cloud Platform , integrating with on-premise/cloud systems like SAP S/4HANA .
  • Implement both real-time streaming and batch processing pipelines using Apache Spark , Dataflow , and other GCP services to meet business requirements.
  • Implement data governance, access controls, and security best practices to ensure the integrity, confidentiality, and compliance of data across systems.
  • Collaborate with business stakeholders, data scientists, and engineering teams to define data requirements, ensuring the architecture aligns with business goals.
  • Optimize Apache Spark jobs for performance, scalability, and cost-efficiency, ensuring that the architecture can handle growing data volumes.
  • Provide technical leadership to the data engineering team, mentoring junior engineers in data architecture, Apache Spark development, and GCP best practices.


Technical Expertise:


  • Expert-level programming proficiency in Python, Java, and Scala.
  • Extensive hands-on experience with big data technologies, including Spark, Hadoop, Hive, Yarn, MapReduce, Pig, Kafka, and PySpark.
  • Proficient in Google Cloud Platform services, such as BigQuery , Dataflow , Cloud Storage , Dataproc , and Cloud Composer Google Pub/Sub, and Google Cloud Functions.Expertise in Apache Spark for both batch and real-time processing, as well as proficiency in Apache Beam , Hadoop , or other big data frameworks. Experienced in using Cloud SQL, BigQuery, and Looker Studio (Google Data Studio) for cloud-based data solutions.
  • Skilled in orchestration and deployment tools like Cloud Composer, Airflow, and Jenkins for continuous integration and deployment (CI/CD).
  • Expertise in designing and developing integration solutions involving Hadoop/HDFS, real-time systems, data warehouses, and analytics solutions.
  • Experience with DevOps practices, including version control (Git ), CI/CD pipelines, and infrastructure-as-code (e.g., Terraform , Cloud Deployment Manager ). Strong background in working with relational databases, NoSQL databases, and in-memory databases.
  • Experience managing large datasets within Data Lake and Data Fabric architectures.
  • Strong knowledge of security best practices, IAM, encryption mechanisms, and compliance frameworks (GDPR, HIPAA) within GCP environments.
  • Experience in implementing data governance, data lineage, and data quality frameworks.
  • In-depth knowledge of web technologies, application programming languages, OLTP/OLAP technologies, data strategy disciplines, relational databases, data warehouse development, and big data solutions.
  • Led end-to-end processes for the design, development, deployment, and maintenance of data engineering projects.
  • Excellent debugging and problem-solving skills.
  • Retail and e-commerce domain knowledge is a plus.
  • Positive attitude with strong analytical skills and the ability to guide teams effectively.


Preferred Qualifications:


  • GCP Certifications : Such as Professional Data Engineer / Apache Spark and Python certifications.
  • Experience with Data visualization tools like Tableau, Power BI etc.


Nice to have (but not mandatory):


  • Experience in integration with S4 Hana
  • Experience with integration with Salesforce

  • Cygnus Professionals Inc. | GCP Data Lead | bangalore

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data LeadLocation: Bangalore, India (Day 1 Onsite)Job Type: ContractJob Description:Strong GCP experiencePrior experience in building pipelines using dataflow, data fusion, data proc, etcVery strong python coding skillExperience in optimising GCP resources for cost and performanceExperience in containerisation of...

  • Cygnus Professionals Inc. | GCP Data Lead | bangalore

    1 week ago

    bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data Lead Location: Bangalore, India (Day 1 Onsite) Job Type: Contract Job Description: Strong GCP experience Prior experience in building pipelines using dataflow, data fusion, data proc, etc Very strong python coding skill Experience in optimising GCP resources for cost and performance Experience in containerisation of...

  • Cygnus Professionals Inc. | Cloud Architect

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Position: GCP Architect (Apache Spark)Location: BangaloreDuration: Full Time (4 Days WFO) Job Description for Data Engineering Architect (GCP & Apache Spark) Job Summary: We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP), Apache Spark, to architect, design and implement scalable, high-performance...

  • Cygnus Professionals Inc. | Cloud Architect

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Position: GCP Architect (Apache Spark) Location: Bangalore Duration: Full Time (4 Days WFO) Job Description for Data Engineering Architect (GCP & Apache Spark) Job Summary: We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP) , Apache Spark , to architect, design and implement scalable,...

  • GCP Data Lead

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data Lead Location: Bangalore, India (Day 1 Onsite) Job Type: Contract Job Description: Strong GCP experience Prior experience in building pipelines using dataflow, data fusion, data proc, etc Very strong python coding skill Experience in optimising GCP resources for cost and performance Experience in containerisation of...

  • Gcp data lead

    1 week ago

    Bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data Lead Location: Bangalore, India (Day 1 Onsite) Job Type: Contract Job Description: Strong GCP experience Prior experience in building pipelines using dataflow, data fusion, data proc, etc Very strong python coding skill Experience in optimising GCP resources for cost and performance Experience in...

  • GCP Data Lead

    1 week ago

    bangalore, India Cygnus Professionals Inc. Full time

    Role: Google Cloud Platform (GCP) Data LeadLocation: Bangalore, India (Day 1 Onsite)Job Type: ContractJob Description:Strong GCP experiencePrior experience in building pipelines using dataflow, data fusion, data proc, etcVery strong python coding skillExperience in optimising GCP resources for cost and performanceExperience in containerisation of...

  • Cygnus Professionals Inc. | Oracle CPQ | bangalore

    4 days ago

    bangalore, India Cygnus Professionals Inc. Full time

    Position: Oracle CPQ Location: PAN India Duration: Long Term Contract The Oracle CPQ (Configure, Price, Quote) developer will perform hands-on technical implementation, with a focus on delivering functional solutions on the Oracle CPQ platform as well as Provide guidance and best practices around CPQ solutions including product configuration, administration...

  • Cloud Architect

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Position: GCP Architect (Apache Spark) Location: Bangalore Duration: Full Time (4 Days WFO) Job Description for Data Engineering Architect (GCP & Apache Spark) Job Summary: We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP), Apache Spark, to architect, design and implement scalable,...

  • Cloud Architect

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Position: GCP Architect (Apache Spark) Location: Bangalore Duration: Full Time (4 Days WFO) Job Description for Data Engineering Architect (GCP & Apache Spark) Job Summary: We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP) , Apache Spark , to architect, design and implement scalable,...

  • Cloud Architect

    2 weeks ago

    bangalore, India Cygnus Professionals Inc. Full time

    Position: GCP Architect (Apache Spark)Location: BangaloreDuration: Full Time (4 Days WFO)Job Description for Data Engineering Architect (GCP & Apache Spark)Job Summary:We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP) , Apache Spark , to architect, design and implement scalable, high-performance...

  • Tekgence Inc | GCP Data Stream | bangalore

    5 days ago

    bangalore, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query , GCP DataStream. Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting ADO, Jenkins, GitHub Location- Pan India, hybrid mode 3 days in a week EXP range- 7-12 Job Role: Senior Cloud Infrastructure Engineer (GCP & ...

  • Tekgence Inc | GCP Data Stream | bangalore

    5 days ago

    bangalore, India Tekgence Inc Full time

    Mandatory Skills: Azure CI/CD pipeline, GCP Cloud, Terraform, Kubernetes, Dataflow, Big Query, GCP DataStream.Azure Devops (CI/CD pipelines), GCP Cloud, Terraform, Docker, Kubernetes, GitHub actions, scripting ADO, Jenkins, GitHubLocation- Pan India, hybrid mode 3 days in a weekEXP range- 7-12 Job Role: Senior Cloud Infrastructure Engineer (GCP & Terraform)...

  • GCP DevOps Engineer

    2 weeks ago

    bangalore, India Enterprise Minds, Inc Full time

    About us: Enterprise Minds India is a Product Engineering company headquartered in San Francisco, In India Bangalore. Hyderabad and Pune, Tirupati, Puducherry. The company was established in 2017 and has worked extensively in Digital experience, Data engineering, DevOps engineering, AI/ML with Product companies across US and India. We are spread across...

  • Gcp devops engineer

    3 weeks ago

    Bangalore, India Enterprise Minds, Inc Full time

    About us: Enterprise Minds India is a Product Engineering company headquartered in San Francisco, In India Bangalore. Hyderabad and Pune, Tirupati, Puducherry. The company was established in 2017 and has worked extensively in Digital experience, Data engineering, Dev Ops engineering, AI/ML with Product companies across US and India. We are spread across...

  • GCP DevOps Engineer

    3 weeks ago

    bangalore, India Enterprise Minds, Inc Full time

    About us:Enterprise Minds India is a Product Engineering company headquartered in San Francisco, In India Bangalore. Hyderabad and Pune, Tirupati, Puducherry. The company was established in 2017 and has worked extensively in Digital experience, Data engineering, DevOps engineering, AI/ML with Product companies across US and India. We are spread across...

  • GCP DevOps Engineer

    3 weeks ago

    bangalore, India Enterprise Minds, Inc Full time

    About us:Enterprise Minds India is a Product Engineering company headquartered in San Francisco, In India Bangalore. Hyderabad and Pune, Tirupati, Puducherry. The company was established in 2017 and has worked extensively in Digital experience, Data engineering, DevOps engineering, AI/ML with Product companies across US and India. We are spread across...

  • GCP DevOps Engineer

    2 weeks ago

    Bangalore, India Enterprise Minds, Inc Full time

    About us: Enterprise Minds India is a Product Engineering company headquartered in San Francisco, In India Bangalore. Hyderabad and Pune, Tirupati, Puducherry. The company was established in 2017 and has worked extensively in Digital experience, Data engineering, DevOps engineering, AI/ML with Product companies across US and India. We are spread across...

  • Lead Engineer

    1 day ago

    Bangalore, India Live Connections Full time

    Lead Engineer - GCP Data Protection Experience: 12 to 16 Years Location: Bangalore / Hyderabad Notice Period: Maximum 30 Days Organization: One of the leading GCCs CTC: 60LPA or 30% on current CTC, whichever is lower. We are looking for a Lead Engineer - GCP Data Protection to spearhead cutting-edge cloud transformation initiatives...

  • GCP DevOps Engineer

    3 weeks ago

    Bangalore, India Enterprise Minds, Inc Full time

    About us: Enterprise Minds India is a Product Engineering company headquartered in San Francisco, In India Bangalore. Hyderabad and Pune, Tirupati, Puducherry. The company was established in 2017 and has worked extensively in Digital experience, Data engineering, DevOps engineering, AI/ML with Product companies across US and India. We are spread across...