Real-Time Data Solutions Architect

18 hours ago

Chennai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 22,00,000 - ₹ 28,00,000
Real-Time Data Engineer

This role will be instrumental in building and maintaining robust, scalable, and reliable data pipelines using Confluent Kafka, ksqlDB, Kafka Connect, and Apache Flink.

The ideal candidate will have a strong understanding of data streaming concepts, experience with real-time data processing, and a passion for building high-performance data solutions.

This role requires excellent analytical skills, attention to detail, and the ability to work collaboratively in a fast-paced environment.

Key Responsibilities:

  • Design & develop data pipelines for real-time and batch data ingestion and processing using Confluent Kafka, ksqlDB, Kafka Connect, and Apache Flink.
  • Build and configure Kafka Connectors to ingest data from various sources (databases, APIs, message queues, etc.) into Kafka.
  • Develop Flink applications for complex event processing, stream enrichment, and real-time analytics.
  • Develop and optimize ksqlDB queries for real-time data transformations, aggregations, and filtering.
  • Implement data quality checks and monitoring to ensure data accuracy and reliability throughout the pipeline.
  • Monitor and troubleshoot data pipeline performance, identify bottlenecks, and implement optimizations.
  • Automate data pipeline deployment, monitoring, and maintenance tasks.
  • Stay up-to-date with the latest advancements in data streaming technologies and best practices.
  • Contribute to the development of data engineering standards and best practices within the organization.
  • Participate in code reviews and contribute to a collaborative and supportive team environment.
  • Work closely with other architects and tech leads in India & US and create POCs and MVPs.
  • Provide regular updates on the tasks, status and risks to project manager.

Requirements:

  • Bachelors degree or higher from a reputed university.
  • 8 to 10 years total experience with majority of that experience related to ETL/ELT, big data, Kafka etc.
  • Proficiency in developing Flink applications for stream processing and real-time analytics.
  • Strong understanding of data streaming concepts and architectures.
  • Extensive experience with Confluent Kafka, including Kafka Brokers, Producers, Consumers, and Schema Registry.
  • Hands-on experience with ksqlDB for real-time data transformations and stream processing.
  • Experience with Kafka Connect and building custom connectors.
  • Extensive experience in implementing large scale data ingestion and curation solutions.
  • Good hands on experience in big data technology stack with any cloud platform.
  • Excellent problem-solving, analytical, and communication skills.
  • Ability to work independently and as part of a team.

Preferred Qualifications:

  • Experience in Google Cloud.
  • Healthcare industry experience.
  • Experience in Agile.

  • Data Architect

    3 days ago

    Chennai, Tamil Nadu, India OptiSol Business Solutions Full time

    Position – Data Architect Experience: 8+ years Position Overview We are seeking a Data Architect who can understand enterprise data from the ground up and translate it into scalable, reliable, and dynamic designs. The ideal candidate will not only build robust data foundations but also leverage Generative AI (GenAI) and advanced data insights frameworks...

  • Senior Real-Time Data Engineer Wanted

    17 hours ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,00,00,000

    About UsWe are a global healthcare partner delivering strategic innovation and IT expertise to organizations in the healthcare sector.As a leading provider of next-generation solutions, we bring unique insights into healthcare operations, helping our partners deliver superior consumer experiences.We are committed to integrity, diversity, and inclusion,...

  • Data Solutions Architect

    17 hours ago

    Chennai, Tamil Nadu, India beBeeDataSolutions Full time ₹ 1,50,00,000 - ₹ 2,25,00,000

    Job Title: Data Solutions Architect">About the Role:", "In this key position, you will be responsible for architecting and delivering scalable, high-performance data solutions using the Databricks Unified Data Analytics platform. You will work closely with clients, delivery teams, technology partners, and internal stakeholders to define and deliver solutions...

  • Data Architect

    5 days ago

    Chennai, Tamil Nadu, India Capgemini Engineering Full time

    Data Architect / Data EngineerLocation: ChennaiExperience: 4–6 yearsYour RoleAs a Data Architect / Data Engineer, you will be responsible for creating and maintaining digital twin models and integrating real-time data streams to support simulation and analytics in maritime systems. You will work with advanced tools and platforms to build scalable,...

  • Technology Solutions Architect

    3 days ago

    Chennai, Tamil Nadu, India beBeeEnterprise Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OpportunityWe are seeking an experienced Enterprise Architect to lead solutioning and architectural direction for mission-critical technology programs.The ideal candidate will have a deep technical background in integration architecture, web-scale application development, and data architecture.Key responsibilities include defining target architectures,...

  • Cloud Data Solutions Architect

    1 week ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Cloud Data Solutions ArchitectWe are seeking a skilled professional to design and develop innovative cloud-based data solutions. This role will play a critical part in building and maintaining our data infrastructure.Key Responsibilities:Data Systems Engineering: Develop and maintain cloud-based data lakes, warehouses, and ETL/ELT pipelines that ingest,...

  • Enterprise Data Solutions Architect

    3 days ago

    Chennai, Tamil Nadu, India beBeeData Full time US$ 1,20,000 - US$ 1,80,000

    Job Title: Enterprise Data Solutions Architect About the Role:The successful candidate will be responsible for designing and building large-scale real-time stream and batch data processing systems, developing and running services and analytics SaaS solutions, and writing unit and end-to-end tests for any code developed. Responsibilities:Collaborate closely...

  • Data Architect

    3 days ago

    Chennai, Tamil Nadu, India Capgemini Engineering Full time

    Data Architect / Data Engineer Location: Chennai Experience: 4–6 years Your Role As a Data Architect / Data Engineer, you will be responsible for creating and maintaining digital twin models and integrating real-time data streams to support simulation and analytics in maritime systems. You will work with advanced tools and platforms to build scalable,...

  • Data Architect

    8 hours ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Role: Data ArchitectWe are seeking an experienced Data Architect to lead the development and maintenance of our data pipelines, ETL workflows, and big data solutions across cloud environments.Design and develop robust ETL/ELT pipelines to process structured and unstructured data.Build and optimize real-time and batch data processing systems using Spark,...

  • Data Solutions Architect

    10 hours ago

    Chennai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Solutions ArchitectWe are seeking a skilled and experienced data engineer to take on the role of Data Solutions Architect.About the Role:As a Data Solutions Architect, you will be responsible for designing and implementing efficient data models and transformations using DBT in conjunction with Snowflake.Key Responsibilities:Translate...