Data Architect

2 weeks ago

Bangalore, India The Metromax Group Full time

Job Summary

At The Metromax Group, we are seeking a skilled Data Architect to design and develop data pipelines using AWS Glue, EMR, Spark Scala, and S3 to support both batch and real-time data processing needs. This role will involve implementing ETL processes to extract, transform, and load data from various sources (structured and unstructured) into the data lake. You will leverage Apache Spark on EMR for big data processing and transformations, using Spark Scala, and manage and optimize data storage on S3, ensuring proper data partitioning, file formats (Parquet, ORC, Avro), and lifecycle policies for cost-effective storage solutions. Additionally, you will monitor, troubleshoot, and optimize EMR clusters for performance, scalability, and cost efficiency. If you have a strong background in data engineering and experience with AWS services, we encourage you to apply for this challenging role.

Responsibilities

- Design and develop data pipelines using AWS Glue, EMR, Spark Scala, and S3 to support both batch and real-time data processing needs.

- Implement ETL processes to extract, transform, and load data from various sources (structured and unstructured) into the data lake.

- Leverage Apache Spark on EMR for big data processing and transformations, using Spark Scala.

- Manage and optimize data storage on S3, ensuring proper data partitioning, file formats (Parquet, ORC, Avro), and lifecycle policies for cost-effective storage solutions.

- Monitor, troubleshoot, and optimize EMR clusters for performance, scalability, and cost efficiency.

Requirements

- Bachelor's or master's degree in computer science, data engineering, or a related technical field.

- 3-5 years of experience in data engineering, particularly using AWS services (EMR, Glue, S3, Lambda).

- Strong expertise in Apache Spark for distributed data processing, with hands-on coding experience in Scala and Python.

- Experience with building ETL pipelines and working with big data in a cloud-based Lakehouse environment.

- Deep understanding of data formats (Parquet, Avro, ORC) and file optimization techniques.

- Familiarity with data modeling principles, including partitioning, bucketing, and schema management in AWS Glue Data Catalog.

- Strong knowledge of SQL and query optimization for working with large datasets.

- Experience with AWS security services such as IAM, KMS (Key Management Service), and encryption best practices.

- Proficiency in troubleshooting and performance tuning of Spark and EMR clusters for large-scale data processing.

- Familiarity with CI/CD pipelines and infrastructure-as-code (Terraform, CloudFormation) for managing AWS environments.

Preferred Qualifications

- AWS Certified Data Analytics, Developer, or Solutions Architect certification.

- Experience with streaming data technologies such as Kinesis or Kafka for real-time data ingestion.

- Knowledge of serverless computing and experience with AWS Lambda, Step Functions, and DynamoDB.

- Familiarity with DevOps and automation tools (e.g., Jenkins, Git, Docker).

Soft Skills

- Strong problem-solving and analytical thinking skills.

- Ability to work collaboratively in a fast-paced, cross-functional environment.

- Excellent communication skills to explain complex technical issues to both technical and non-technical stakeholders.

  • Data Solutions Architect

    3 weeks ago

    Bangalore, India HEVO DATA Full time

    Job Title: Data Solutions ArchitectJob Summary:We are seeking a skilled Data Solutions Architect to join our team at Hevo Data. As a Data Solutions Architect, you will be responsible for working closely with our sales team to understand customer needs and demonstrate how Hevo's platform can solve their real-time data replication needs.Key...

  • Data Architect

    3 weeks ago

    Bangalore, India Ample Success Hr Solutions Full time

    Job Title: Data ArchitectJob Summary:We are seeking a highly skilled Data Architect to join our team at Ample Success Hr Solutions. As a Data Architect, you will be responsible for designing, implementing, and developing data pipelines to collect and process large amounts of data from various sources.Key Responsibilities:Design and implement data storage...

  • Data Architect

    2 weeks ago

    Bangalore, India manogna it services private limited Full time

    Job Description : - Utilize advanced knowledge of data structures and algorithms to design and implement efficient solutions. - Develop detailed Low-Level Designs (LLD) and High-Level Designs (HLD) for complex systems. - Demonstrate proficiency in real-time coding to ensure optimal system performance and responsiveness. - Architect scalable and...

  • Data Architect

    1 month ago

    Bangalore, Karnataka, India manogna it services private limited Full time

    Job Description :- Utilize advanced knowledge of data structures and algorithms to design and implement efficient solutions. - Develop detailed Low-Level Designs (LLD) and High-Level Designs (HLD) for complex systems. - Demonstrate proficiency in real-time coding to ensure optimal system performance and responsiveness. - Architect scalable and resilient...

  • Data Architect

    3 weeks ago

    Bangalore, India D Square Consulting Service Full time

    Job Title: Data ArchitectWe are seeking a highly skilled Data Architect to join our team at D Square Consulting Service. As a Data Architect, you will play a critical role in designing and implementing data architectures that support business growth and innovation.Key Responsibilities:Design and implement data architectures that meet business requirements...

  • Data Architect

    3 weeks ago

    Bangalore, India manogna it services private limited Full time

    Job Title: Data ArchitectAt Manogna IT Services Private Limited, we are seeking a highly skilled Data Architect to join our team. The successful candidate will be responsible for designing and implementing efficient data structures and algorithms to drive business growth.Key Responsibilities:Design and implement data architecture vision, strategy, and...

  • Data Solutions Architect

    2 weeks ago

    Bangalore, India HEVO DATA Full time

    Overview:We're seeking a talented Solutions Engineer to join our Sales team at Hevo Data. As a Solutions Engineer, you'll work closely with the sales team to understand customers' business and technical needs and demonstrate how our platform can solve their real-time data replication needs.This role requires a deep understanding of our architecture and a...

  • Data Architect

    3 weeks ago

    Bangalore, India Winning Edge Full time

    Job Title: Data ArchitectJob Summary:We are seeking a highly skilled Data Architect to join our team at Winning Edge. As a Data Architect, you will be responsible for designing and implementing end-to-end data architectures using Databricks, ensuring they meet scalability, reliability, and security requirements.Key Responsibilities:Design and develop data...

  • Data Solution Architect/Data Architect/Big Data Architect

    2 weeks ago

    Bangalore/Hyderabad, India Coders Brain Technology & Private Limited Full time

    SALARY : 20LPA - 75LPA Data Solution Architect/Data Architect /Big Data ArchitectExperience : 11 to 22 YearsLocation : Hyderabad onlyNotice Period : ImmediateRole : Data Solution ArchitectShift Timings : RegularWork Mode : Hybrid(3 days a week is mandatory)Job Role : FTEEducation Qualification : BE/BTech/MCA onlyJob Description :- Minimum of 12 years of...

  • Data Solution Architect/Data Architect/Big Data Architect

    1 month ago

    Bangalore/Hyderabad, Karnataka, India Coders Brain Technology & Private Limited Full time

    SALARY : 20LPA - 75LPAData Solution Architect/Data Architect /Big Data ArchitectExperience : 11 to 22 YearsLocation : Hyderabad onlyNotice Period : ImmediateRole : Data Solution ArchitectShift Timings : RegularWork Mode : Hybrid(3 days a week is mandatory)Job Role : FTEEducation Qualification : BE/BTech/MCA onlyJob Description :- Minimum of 12 years of...

  • Data Architect

    3 weeks ago

    Bangalore, India Winning Edge Full time

    Job Title: Data ArchitectWe are seeking a highly skilled Data Architect to join our team at Winning Edge. As a key member of our data team, you will be responsible for designing and implementing large-scale data models for functional, operational, and analytical environments.Key Responsibilities:Design and develop conceptual, logical, physical, and...

  • Data Architect

    3 weeks ago

    Bangalore, India Twenty20 Systems Full time

    Job Title: Data ArchitectWe are seeking a highly skilled and experienced Data Architect to join our team at Twenty20 Systems. As a Data Architect, you will play a key role in designing and implementing modern data solutions that drive business impact.Key Responsibilities:Design and implement data architecture that reduces time to insight and informs business...

  • Data Architect

    2 weeks ago

    Bangalore, India ThinkMove Solutions Full time

    Job Title : Data ArchitectLocation : Not SpecifiedExperience : 8+ yearsNotice Period : ImmediateWe are seeking a Data Architect with 8+ years of overall experience and a minimum of 5 years of expertise in Data & Analytics initiatives. This role is ideal for a professional who can design and implement robust data architecture solutions to support data-driven...

  • Data Architect

    3 weeks ago

    bangalore, India Bottomline Full time

    DescriptionBottomline is at the forefront of digital transformation. We are a growing global market leader uniquely equipped to address the changing needs of how businesses pay and get paid. Our culture of Working with and for each other enables us to delight our customers. We empower our teams to think like owners driving customer satisfaction, helping them...

  • Data architect

    3 weeks ago

    Bangalore, India Bottomline Full time

    Description Bottomline is at the forefront of digital transformation. We are a growing global market leader uniquely equipped to address the changing needs of how businesses pay and get paid. Our culture of Working with and for each other enables us to delight our customers . We empower our teams to think like owners driving customer...

  • Data Architect

    3 weeks ago

    Bangalore, India Bottomline Full time

    Description Bottomline is at the forefront of digital transformation. We are a growing global market leader uniquely equipped to address the changing needs of how businesses pay and get paid. Our culture of Working with and for each other enables us to delight our customers . We empower our teams to think like owners driving customer satisfaction,...

  • Data Architect

    3 weeks ago

    Bangalore, India D Square Consulting Service Full time

    About the RoleWe are seeking a highly skilled Data Architect to join our team at D Square Consulting Service. As a key member of our data platforms and products team, you will be responsible for designing and implementing data architecture and modeling solutions that drive business growth.Key ResponsibilitiesProvide expertise on data architecture and...

  • Data Architect

    3 weeks ago

    Bangalore, India Winning Edge Full time

    Job Title: Data ArchitectAt Winning Edge, we are seeking a highly skilled Data Architect to design and implement large-scale data models for our functional, operational, and analytical environments.Key Responsibilities:Design and develop conceptual, logical, physical, and dimensional data models for complex data domains.Develop and maintain relational schema...

  • Data Architect

    3 weeks ago

    Bangalore/Hyderabad, India Coders Brain Technology & Private Limited Full time

    Job Title: Data Solution Architect/Data Architect/Big Data ArchitectSalary: ₹20LPA - ₹75LPAExperience: 11 to 22 YearsLocation: Hyderabad onlyNotice Period: ImmediateRole: Data Solution ArchitectShift Timings: RegularWork Mode: Hybrid (3 days a week is mandatory)Job Role: FTEEducation Qualification: BE/BTech/MCA onlyJob Description:Key...

  • Data Architect

    2 days ago

    bangalore, India Manuh Technologies Full time

    Position - Data Architect Location - Bangalore Experience - 9+ We are seeking a talented and experienced Data Architect to join our dynamic team. The ideal candidate will have extensive experience in designing and implementing data architectures within a banking platform, with a strong emphasis on cloud technologies (preferably Google Cloud Platform), SQL,...