Data Engineering Architect

1 month ago

Bengaluru, India CygnusPro Software Solutions Pvt. Ltd Full time

Role : Data Engineering Architect (GCP & Apache Spark).

Job Summary :

We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP), Apache Spark, to architect, design and implement scalable, high-performance data lake solutions.

The ideal candidate will have extensive experience in building data ingestion pipelines, managing big data processing using Apache Spark,.

Key Requirements :

- Over 12 years of professional experience in data engineering, specializing in implementing large-scale enterprise Data Engineering projects with the latest technologies.

- Over 5 years of hands-on experience in GCP technologies and over 3 years of architect experience.

- Design and implement end-to-end data architectures leveraging GCP services (e., Big Query, Cloud Storage, Dataflow, Pub/Sub, Cloud Composer) for large-scale data ingestion and processing.

- Build and optimize large-scale data pipelines using Apache Spark on GCP (via Dataproc or other Spark services).

- Ensure high performance and scalability in Spark-based data processing workloads.

- Lead the integration of SAP S/4HANA data with GCP for real-time and batch data processing.

- Manage data extraction, transformation, and loading (ETL) processes from SAP S/4HANA into cloud storage and data lakes.

- Develop and manage scalable data ingestion pipelines for structured and unstructured data using tools like Cloud Dataflow, Cloud Pub/Sub, and Apache Spark.

- Provide architectural guidance for designing secure, scalable, and efficient data solutions on the Google Cloud Platform, integrating with on-premise/cloud systems like SAP S/4HANA.

- Implement both real-time streaming and batch processing pipelines using Apache Spark, Dataflow, and other GCP services to meet business requirements.

- Implement data governance, access controls, and security best practices to ensure the integrity, confidentiality, and compliance of data across systems.

- Collaborate with business stakeholders, data scientists, and engineering teams to define data requirements, ensuring the architecture aligns with business goals.

- Optimize Apache Spark jobs for performance, scalability, and cost-efficiency, ensuring that the architecture can handle growing data volumes.

- Provide technical leadership to the data engineering team, mentoring junior engineers in data architecture, Apache Spark development, and GCP best practices.

Technical Expertise :

- Expert-level programming proficiency in Python, Java, and Scala.

- Extensive hands-on experience with big data technologies, including Spark, Hadoop, Hive, Yarn, MapReduce, Pig, Kafka, and PySpark.

- Proficient in Google Cloud Platform services, such as BigQuery, Dataflow, Cloud Storage, Dataproc, and Cloud Composer Google Pub/Sub, and Google Cloud Functions.

- Expertise in Apache Spark for both batch and real-time processing, as well as proficiency in Apache Beam, Hadoop, or other big data frameworks.

- Experienced in using Cloud SQL, BigQuery, and Looker Studio (Google Data Studio) for cloud-based data solutions.

- Skilled in orchestration and deployment tools like Cloud Composer, Airflow, and Jenkins for continuous integration and deployment (CI/CD).

- Expertise in designing and developing integration solutions involving Hadoop/HDFS, real-time systems, data warehouses, and analytics solutions.

- Experience with DevOps practices, including version control (Git), CI/CD pipelines, and infrastructure-as-code (e., Terraform, Cloud Deployment Manager).

- Strong background in working with relational databases, NoSQL databases, and in-memory databases.

- Experience managing large datasets within Data Lake and Data Fabric architectures.

- Strong knowledge of security best practices, IAM, encryption mechanisms, and compliance frameworks (GDPR, HIPAA) within GCP environments.

- Experience in implementing data governance, data lineage, and data quality frameworks.

- In-depth knowledge of web technologies, application programming languages, OLTP/OLAP technologies, data strategy disciplines, relational databases, data warehouse development, and big data solutions.

- Led end-to-end processes for the design, development, deployment, and maintenance of data engineering projects.

- Excellent debugging and problem-solving skills.

- Retail and e-commerce domain knowledge is a plus.

- Positive attitude with strong analytical skills and the ability to guide teams effectively.

Preferred Qualifications :

- GCP Certifications: Such as Professional Data Engineer or Professional Cloud Architect.

- Apache Spark and Python certifications.

- Experience with Data visualization tools like Tableau, Power BI etc.

(ref:hirist.tech)

  • Senior Data Architect

    3 weeks ago

    Bengaluru, Karnataka, India Altair Engineering Full time

    Job Title: Senior Data ArchitectWe are seeking a highly skilled Data Architect to join our team at Altair Engineering. As a Senior Data Architect, you will be responsible for designing and implementing data solutions that drive business value.About the Role:This is a challenging and rewarding role that requires strong technical skills, excellent...

  • Snowflake Data Engineer and Architect

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    About the RoleWe are seeking a highly skilled Snowflake Data Engineer and Architect to join our team in Bangalore, India. This is a senior-level position that requires expertise in designing, developing, and maintaining large-scale data warehousing solutions using Snowflake.Key ResponsibilitiesDesign and implement efficient data models and architectures for...

  • Lead Data Architect

    5 days ago

    Bengaluru, Karnataka, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data ArchitectLocation: Yemalur, BangaloreAt Data Labs - LANDMARK GROUP, we are seeking a skilled Lead Data Architect to spearhead our engineering team in successfully designing and deploying cutting-edge analytics products.Main Responsibilities:Lead the architecture, design, and delivery of enterprise-grade microservices.Partner with Product...

  • Ai Data Architect

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    **Req ID**: 303825 We are currently seeking a AI Data Architect to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Job Duties: Experience: - 10+ years of experience in Solution Architecting of Data & Analytics, AI/ML & Gen AI Solutions as Presales Consultant/ Business Development Manager/ Solution Architect **Responsibilities**: - Solid...

  • Data Solutions Architect

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    We are currently seeking a highly skilled Data Solutions Architect to join our team in Bangalore, India.About the Role:As a Data Solutions Architect, you will design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.Main...

  • Senior Cloud Data Architect

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    NTT DATA is a global technology services company with a presence in over 50 countries.We have a wide range of job opportunities available for talented professionals like you.About the Job:Job DescriptionWe are seeking a Senior Cloud Data Architect to join our team in Bangalore, India. The successful candidate will be responsible for designing and...

  • Senior Data Architect

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    Job OverviewWe are seeking an experienced Senior Data Architect - Cloud Engineer to join our team in Remote India. The successful candidate will play a critical role in supporting and enhancing our Azure Data Lakehouse environment.About the RoleThis is an exciting opportunity to leverage your expertise in designing and implementing efficient data models,...

  • Senior AI Data Engineer and Solutions Architect

    3 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    About NTT DATANTT DATA is a global innovator of business and technology services, serving 75% of the Fortune Global 100. Our diverse experts in over 50 countries collaborate to drive long-term success for clients.Job OverviewWe are seeking an experienced Senior AI Data Engineer and Solutions Architect to join our team in Bangalore, India. This role requires...

  • Strategic Data Architect Leader

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    About the RoleWe are seeking a highly experienced and dynamic Strategic Data Architect Leader to join our Data, Analytics and AI team.Job SummaryThe ideal candidate will have a strong background in Data, Big Data and related technology and management, with a proven track record of leading innovation and architecture initiatives.Key ResponsibilitiesLead the...

  • Senior Data Architect

    2 weeks ago

    Bengaluru, Karnataka, India Ness Digital Engineering Full time

    We are seeking an experienced Technical Data Engineer / ELT Developer to manage data ingestion and transformation, working with an offshore team to deliver efficient and scalable data solutions that align with business requirements and performance standards.Job OverviewNess Digital Engineering is looking for a skilled professional to design, build, and...

  • Aws Cloud Architect

    5 months ago

    Bengaluru, India NTT DATA Full time

    **Req ID**: 248981 We are currently seeking a AWS Cloud Architect to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Background As an Amazon Web Services (AWS) cloud architect, you will be responsible to architect and manage the cloud environment for a Digital Banking Asset. Duties include architecting and implementing cloud solutions,...

  • Data Scientist Architect Lead

    5 days ago

    Bengaluru, Karnataka, India Capgemini Engineering Full time

    About Capgemini EngineeringWe are committed to creating a diverse and inclusive work environment where everyone has equal opportunities to grow and succeed. All applications welcome and will be considered based on merit against the job and/or experience for the position.Overview of the Role:Data Scientist Developer/Architect is a senior-level position that...

  • Data Engineer

    4 weeks ago

    Bengaluru, India NTT DATA Full time

    Job Description Req ID: 301314 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN).Job Duties: As a data...

  • Principal Data Architect

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    Job Title: Principal Data ArchitectWe are seeking a highly experienced Innovation and Technology Lead to join our team at NTT DATA Services. This role requires strategic thinking, excellent leadership skills, and the ability to drive technology adoption and efficiency improvements across the organization.

  • Senior Cloud Data Architect

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    About the RoleWe are seeking an experienced Senior Cloud Data Architect to join our team at NTT DATA Services. As a key member of our data engineering team, you will be responsible for designing and implementing cloud-based data solutions on Azure Data platform.ResponsibilitiesDesign and develop big data processing notebooks using Azure DatabricksScripting...

  • Senior Data Architect

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Services Full time

    About the RoleNTT DATA Services is seeking a highly skilled Senior Data Architect to join our team in Bangalore, India.The Ideal CandidateThis is an exceptional opportunity for an experienced data professional to work with cutting-edge technologies and contribute to the design and development of innovative solutions. To be successful in this role, you will...

  • Aiml Architect

    4 months ago

    Bengaluru, Karnataka, India NTT DATA Full time

    **About NTT DATA** NTT DATA Services currently seeks AIML Architect to join our team in Bangalore Key Responsibilities: - Design and architect scalable AI platforms to develop, deploy AI solutions leveraging ML techniques and Deep Learning Techniques. - Drive Joint Architecture Design to collaborating with business stakeholders, data scientists,...

  • Data Engineering Architect

    2 weeks ago

    Bengaluru, Karnataka, India Cloudera Full time

    Job Title: Data Engineering ArchitectWe are seeking a highly skilled Data Engineering Architect to join our team at Cloudera. As a Senior Software Engineer, you will play a key role in designing and building scalable data systems using Hadoop, Spark, Kafka, Hive, and HBase.About the Role:The successful candidate will be responsible for conceptualizing,...

  • AI Solutions Architect

    1 week ago

    Bengaluru, Karnataka, India Capgemini Engineering Full time

    Job DescriptionCapgemini Engineering is seeking a seasoned AI Solutions Architect to lead our data science initiatives across India. As a key member of our team, you will be responsible for developing innovative AI and machine learning solutions that drive business growth and improvement.About the RoleThe ideal candidate will have at least 10-15 years of...

  • Solutions Architect

    6 months ago

    Bengaluru, Karnataka, India NTT DATA Full time

    **Req ID**: 268772 We are currently seeking a Solutions Architect to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Software Engineering architect to support Single Service Experience and capabilities (Identity, UAM, Launch Pad). Require skills include engineering leadership, technical skills (Node, React, AWS, Git, automated testing, CI/CD...