Data Engineering Architect

3 weeks ago

Bangalore, India CygnusPro Software Solutions Pvt. Ltd Full time

Role : Data Engineering Architect (GCP & Apache Spark).

Job Summary :

We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP), Apache Spark, to architect, design and implement scalable, high-performance data lake solutions.

The ideal candidate will have extensive experience in building data ingestion pipelines, managing big data processing using Apache Spark,.

Key Requirements :

- Over 12 years of professional experience in data engineering, specializing in implementing large-scale enterprise Data Engineering projects with the latest technologies.

- Over 5 years of hands-on experience in GCP technologies and over 3 years of architect experience.

- Design and implement end-to-end data architectures leveraging GCP services (e., Big Query, Cloud Storage, Dataflow, Pub/Sub, Cloud Composer) for large-scale data ingestion and processing.

- Build and optimize large-scale data pipelines using Apache Spark on GCP (via Dataproc or other Spark services).

- Ensure high performance and scalability in Spark-based data processing workloads.

- Lead the integration of SAP S/4HANA data with GCP for real-time and batch data processing.

- Manage data extraction, transformation, and loading (ETL) processes from SAP S/4HANA into cloud storage and data lakes.

- Develop and manage scalable data ingestion pipelines for structured and unstructured data using tools like Cloud Dataflow, Cloud Pub/Sub, and Apache Spark.

- Provide architectural guidance for designing secure, scalable, and efficient data solutions on the Google Cloud Platform, integrating with on-premise/cloud systems like SAP S/4HANA.

- Implement both real-time streaming and batch processing pipelines using Apache Spark, Dataflow, and other GCP services to meet business requirements.

- Implement data governance, access controls, and security best practices to ensure the integrity, confidentiality, and compliance of data across systems.

- Collaborate with business stakeholders, data scientists, and engineering teams to define data requirements, ensuring the architecture aligns with business goals.

- Optimize Apache Spark jobs for performance, scalability, and cost-efficiency, ensuring that the architecture can handle growing data volumes.

- Provide technical leadership to the data engineering team, mentoring junior engineers in data architecture, Apache Spark development, and GCP best practices.

Technical Expertise :

- Expert-level programming proficiency in Python, Java, and Scala.

- Extensive hands-on experience with big data technologies, including Spark, Hadoop, Hive, Yarn, MapReduce, Pig, Kafka, and PySpark.

- Proficient in Google Cloud Platform services, such as BigQuery, Dataflow, Cloud Storage, Dataproc, and Cloud Composer Google Pub/Sub, and Google Cloud Functions.

- Expertise in Apache Spark for both batch and real-time processing, as well as proficiency in Apache Beam, Hadoop, or other big data frameworks.

- Experienced in using Cloud SQL, BigQuery, and Looker Studio (Google Data Studio) for cloud-based data solutions.

- Skilled in orchestration and deployment tools like Cloud Composer, Airflow, and Jenkins for continuous integration and deployment (CI/CD).

- Expertise in designing and developing integration solutions involving Hadoop/HDFS, real-time systems, data warehouses, and analytics solutions.

- Experience with DevOps practices, including version control (Git), CI/CD pipelines, and infrastructure-as-code (e., Terraform, Cloud Deployment Manager).

- Strong background in working with relational databases, NoSQL databases, and in-memory databases.

- Experience managing large datasets within Data Lake and Data Fabric architectures.

- Strong knowledge of security best practices, IAM, encryption mechanisms, and compliance frameworks (GDPR, HIPAA) within GCP environments.

- Experience in implementing data governance, data lineage, and data quality frameworks.

- In-depth knowledge of web technologies, application programming languages, OLTP/OLAP technologies, data strategy disciplines, relational databases, data warehouse development, and big data solutions.

- Led end-to-end processes for the design, development, deployment, and maintenance of data engineering projects.

- Excellent debugging and problem-solving skills.

- Retail and e-commerce domain knowledge is a plus.

- Positive attitude with strong analytical skills and the ability to guide teams effectively.

Preferred Qualifications :

- GCP Certifications: Such as Professional Data Engineer or Professional Cloud Architect.

- Apache Spark and Python certifications.

- Experience with Data visualization tools like Tableau, Power BI etc.

(ref:hirist.tech)

  • Data Inc | Data Architect

    4 weeks ago

    bangalore, India Data Inc Full time

    Job Title: Data ArchitectLocation: Pune/Hyderabad/BangaloreVacancy Type: Fixed Term contract (6 Months initial)Pay rate:- -Role purpose (overall high-level summary of the role)Data & Analytics Office (DAO) promotes a data-driven culture across bank through our strategy of:Fix The Back Book: To protect our data we need robust policies for data management and...

  • Data engineer

    3 months ago

    Bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost...

  • Data Engineer

    3 months ago

    Bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and...

  • Data Engineer

    2 months ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices.Deliver assigned projects within budgeted time, cost and agreed...

  • Data Engineer

    3 months ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and agreed...

  • Data Engineer

    3 months ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and...

  • Data Labs

    7 days ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.- Lead the architecture, design, and delivery of enterprise grade microservices.- Deliver assigned projects within budgeted time, cost and agreed...

  • Data architect

    4 weeks ago

    Bangalore, India MPACT TRAINING AND DEVELOPMENT Full time

    We are seeking experienced, motivated andenthusiastic Data Architect to join our team. As part of the team, you will be involved in the implementation of the ongoing and new initiatives for our company and contribute to our mission of delivering high-quality data-driven solutions to our clients. Primary Skills: Snowflake, AWS/Azure data services, Data...

  • Data Labs

    6 days ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and...

  • Architect - data

    2 days ago

    Bangalore, India Quantiphi Full time

    JOB ROLE - Architect/Associate Architect - Data JOB LOCATION: Mumbai/Bangalore EXPERIENCE LEVEL: 6-14yrs We are looking for an experienced data personnel as a key member of our technology consulting team. You will play a vital role in designing, planning and implementing data solutions on AWS for our clients. This role requires expertise in...

  • Architect - data

    2 days ago

    Bangalore, India Quantiphi Full time

    JOB ROLE - Architect/Associate Architect - Data (GCP) JOB LOCATION: Mumbai/Bangalore EXPERIENCE LEVEL: 6-14yrs We are looking for an experienced data personnel as a key member of our technology consulting team. You will play a vital role in designing, planning and implementing data solutions on AWS for our clients. This role requires expertise in...

  • Senior data architect

    2 weeks ago

    Bangalore, India Trianz Full time

    Responsibilities - Architectural Design: Architect secure, scalable, highly performant data engineering and management solutions, including data warehouses, data lake, ELT / ETL, and real-time data engineering/pipeline solutions. Support Principal Data Architect in defining and maintaining Practice reference data engineering and data management...

  • Data Architect

    4 weeks ago

    Bangalore, India MPACT TRAINING AND DEVELOPMENT Full time

    We are seeking experienced, motivated andenthusiastic Data Architect to join our team. As part of the team, you will be involved in the implementation of the ongoing and new initiatives for our company and contribute to our mission of delivering high-quality data-driven solutions to our clients. Primary Skills: Snowflake, AWS/Azure data services, Data...

  • Data Architect

    4 weeks ago

    Bangalore, India MPACT TRAINING AND DEVELOPMENT Full time

    We are seeking experienced, motivated andenthusiastic Data Architect to join our team. As part of the team, you will be involved in the implementation of the ongoing and new initiatives for our company and contribute to our mission of delivering high-quality data-driven solutions to our clients. Primary Skills: Snowflake, AWS/Azure data services, Data...

  • Capgemini Engineering | Azure IOT Architect | 12-20 Years | PAN India

    6 days ago

    bangalore, India Capgemini Engineering Full time

    Position :- Azure IOT Architect (6+ yrs as Architect exp. is Mandatory)Experience : - 12 to 20 yearsWork location:- PAN IndiaQualification :- BE/B.Tech , ME/M.Tech Computer Science, Engineering, or a related field.Job Responsibilities :-- Substantial experience in Azure IoT Hub,- Design and implement scalable IoT solutions utilizing Azure IoT Hub, Azure IoT...

  • Capgemini Engineering | Azure IOT Architect | 12-20 Years | PAN India

    2 weeks ago

    bangalore, India Capgemini Engineering Full time

    Position :- Azure IOT Architect (6+ yrs as Architect exp. is Mandatory)Experience : - 12 to 20 yearsWork location:- PAN IndiaQualification :- BE/B.Tech , ME/M.Tech Computer Science, Engineering, or a related field.Job Responsibilities :-Substantial experience in Azure IoT Hub,Design and implement scalable IoT solutions utilizing Azure IoT Hub, Azure IoT...

  • Architect - Data

    2 days ago

    Bangalore, India Quantiphi Full time

    JOB ROLE - Architect/Associate Architect - Data JOB LOCATION: Mumbai/Bangalore EXPERIENCE LEVEL: 6-14yrs We are looking for an experienced data personnel as a key member of our technology consulting team. You will play a vital role in designing, planning and implementing data solutions on AWS for our clients. This role requires expertise in...

  • Architect - Data

    2 days ago

    Bangalore, India Quantiphi Full time

    JOB ROLE - Architect/Associate Architect - Data (GCP) JOB LOCATION: Mumbai/Bangalore EXPERIENCE LEVEL: 6-14yrs We are looking for an experienced data personnel as a key member of our technology consulting team. You will play a vital role in designing, planning and implementing data solutions on AWS for our clients. This role requires expertise in...

  • Data Inc | Data Architect

    4 weeks ago

    bangalore, India Data Inc Full time

    Key accountabilities include:Subject matter expert on the Data Value Stream Enabler which contains data architecture current state, future state, and roadmap for the transition.Collaborate with stakeholders across various value streams to understand their requirements, data architecture plans and ensure alignment with Data Value StreamLead and own the...

  • Capgemini Engineering | Information Architect | bangalore

    1 month ago

    bangalore, India Capgemini Engineering Full time

    Hi All, we are looking out for IT/OT Architect Exp 8 to 14 Years Notice period- 30 days the JD is as follows: . Strong knowledge of IoT platforms and technologies, including sensor devices, gateways, communication protocols, and IoT frameworks. II. Expertise in designing and architecting scalable and secure IoT solutions. III. Proficiency in...