Building Scalable Data Pipelines

3 days ago

Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

As a Data Integration Specialist, you will have the opportunity to design and build robust data pipelines for processing channel activity data from multiple sources. This role requires strong knowledge of AWS, Python, and experience in handling data from various sources such as Hadoop and Terradata.

Key Responsibilities:
  • Design, build, and maintain scalable data pipelines for processing channel activity data, focusing on digital channels and integrating multiple sources of data such as Hadoop and Terradata.
  • Work on AWS cloud infrastructure and orchestration tools like Airflow to ensure high availability, scalability, and performance.
  • Leverage Python for building automated data engineering solutions and data transformations.
  • Collaborate with cross-functional teams to ensure data from various digital channels is effectively ingested and processed.
  • Contribute to architecture decisions related to data lakes, data warehouses, and big data ecosystems.
  • Ensure data quality, governance, and integrity across all data systems.

Required Skills and Qualifications:

  • AWS: Strong hands-on experience with AWS technologies such as PySpark, Glue, EMR, Lambda, Redshift, Athena, EC2, and S3.
  • Python: Proficiency in Python for data processing, automation, and pipeline development.
  • Big Data Platforms: Experience with big data technologies like Hadoop and Terradata.
  • Strong Analytical Skills: Ability to analyze and interpret data from different digital channels.
  • Collaboration: Ability to work closely with cross-functional teams, including business analysts, data scientists, and stakeholders.
  • Problem Solving: Strong problem-solving skills and ability to handle complex data-related challenges.
  • Communication: Excellent communication skills, both written and verbal, for documentation and collaboration.

Desirable Qualifications:

  • A degree in Computer Science, Engineering, or a related field.
  • Certifications or experience in AWS Cloud or relevant data engineering tools would be beneficial.
  • Experience in working on Snowflake would be good to have.
  • Knowledge of ingesting real-time data (Apache Kafka) would be an advantage.

  • Building Scalable Data Pipelines

    6 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Data EngineerAs a data engineer, you will play a crucial role in building and optimizing data pipelines, ensuring data is clean, secure, and accessible for analytics and operations.Key Responsibilities:Design and create scalable data pipelines and architectures.Work with Azure Cloud, SQL Server, ClickHouse, and MongoDB for data modeling, ETL, and...

  • Building Scalable Data Pipelines

    6 hours ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Engineer">">Data Architect Job DescriptionDear candidate, we are looking for a skilled data engineer to join our team. We are seeking an experienced individual with expertise in building scalable data pipelines using Apache Spark, PySpark, SQL, and Python.Key Responsibilities:Design and develop robust ETL/ELT pipelines on AWS using Spark and...

  • Building Scalable Data Infrastructure

    1 week ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,10,00,000

    Job Title:As a senior data engineer, you will design and build scalable data pipelines to support real-time clinical workflows.Key Responsibilities:Design event-driven data pipelines that integrate with EHR/EMR systems using standards like FHIR and HL7.Develop reliable systems to ingest, track, and update clinical events in real-time.Ensure high-quality,...

  • Data Pipeline Architect

    3 days ago

    Kottayam, Kerala, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job Title: Data Engineering Professional">">Job Description">The role of a Data Engineer is to design, build, and maintain large-scale data systems that support business operations. This includes developing and deploying data pipelines, implementing data warehouses and lakes, and ensuring data quality and integrity.">As a Data Engineer, you will work closely...

  • Building Scalable Systems

    6 days ago

    Kottayam, Kerala, India beBeeBackend Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Senior Backend EngineerWe are seeking a highly skilled Senior Backend Engineer to join our high-impact engineering team. As a key member of the team, you will be responsible for designing, developing, and maintaining scalable backend systems and microservices.Key Responsibilities:Design and develop robust backend systems that meet the highest standards of...

  • Senior Technical Architect

    7 days ago

    Kottayam, Kerala, India beBeeTechnical Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job DescriptionWe are seeking an accomplished technical professional to spearhead the development of scalable data pipelines and enhance our platform architecture.Design and implement a shared library/SDK for ingestion, parsing, validation, enrichment, and publishing of data.Create standardized configuration, packaging, versioning, CI/CD, and documentation...

  • Building Scalable Big Data Platforms

    7 days ago

    Kottayam, Kerala, India beBeeTechnical Full time ₹ 20,00,000 - ₹ 30,00,000

    Big Data Platform EngineerSigmoid empowers enterprises to make data-driven decisions by integrating advanced data engineering with AI consulting. We collaborate with leading organizations across sectors such as CPG-retail, BFSI, life sciences, manufacturing, and more to solve complex business challenges.As a Technical Lead, you will build a highly scalable...

  • Senior Data Pipeline Architect

    2 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,10,00,000

    About the RoleWe are seeking a seasoned data engineer to lead our analytics team and leverage expertise in log-level data handling and modern BI tools.The ideal candidate will have a strong track record of building scalable data pipelines, developing analytics strategies, and mentoring analysts to ensure quality delivery and technical upskilling.Design and...

  • Data Pipeline Architect

    2 days ago

    Kottayam, Kerala, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

    About the JobElevate your career with a challenging data-driven role at a global leading manufacturer.Design and develop scalable pipelines to efficiently process transform data using SparkDevelop a robust framework for generating PDF reports using Python SparkUtilize Snowflake Spark SQL to perform aggregations on high volumes of dataKey...

  • Building Scalable Financial Infrastructure

    2 weeks ago

    Kottayam, Kerala, India beBeeBackend Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Senior Backend EngineerBuild the backbone of our infrastructure by designing scalable, secure systems that handle mission-critical financial flows. This role involves architecting and implementing core payment processing engines, developing reliable and scalable financial systems for global operations, building integration layers with banking partners and an...