Data Engineer at Scale

3 days ago

hubballi, India beBeeData Full time

Big Data Engineer PositionWe are seeking a talented Big Data Engineer to join our team and drive the design and implementation of real-time data ingestion pipelines using Apache NiFi.The ideal candidate will have experience with structured and unstructured datasets, as well as proficiency in integrating ingestion and processing pipelines with the GCP ecosystem, including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer.Design and Implement Real-Time Data Ingestion Pipelines: Using Apache NiFi, develop efficient and scalable data pipelines to handle high-throughput data movement across ingestion, processing, and storage layers.Integrate with GCP Ecosystem: Collaborate with cross-functional teams to integrate ingestion and processing pipelines with various GCP services, ensuring seamless data flow and minimizing latency.Develop SQL Queries: Create optimized SQL queries for BigQuery and relational databases to support analytics and reporting requirements.Data Transformation and Automation: Build Python scripts for data transformation, automation, and developing custom NiFi processors to enhance data quality and integrity.Data Quality Checks: Implement robust data quality checks, validation rules, deduplication, late-arriving data handling, and error-handling frameworks to ensure data accuracy and consistency.Collaboration and Communication: Work closely with data analysts, data scientists, and platform engineering teams to deliver analytics-ready datasets and provide timely feedback.Security and Compliance: Apply data security best practices, GCP IAM policies, VPC Service Controls, and encryption standards to maintain compliance with industry regulations.Pipeline Monitoring and Optimization: Continuously monitor, troubleshoot, and optimize pipeline performance to ensure stability, reliability, and optimal resource utilization.Documentation and Knowledge Sharing: Document data flows, architecture diagrams, transformation logic, and operational best practices to facilitate knowledge sharing and onboarding.

  • Chief Architect

    6 days ago

    hubballi, India beBeeData Full time

    Senior Data Systems DeveloperThe Role:We're looking for an accomplished senior data systems developer to spearhead the creation of large-scale, high-performance data platforms. This includes designing and implementing infrastructure from the ground up, ensuring seamless data flow, and guaranteeing scalability.Main Responsibilities:Design, build, and maintain...

  • Expert Large-Scale Data Migration Specialist

    4 days ago

    hubballi, India beBeeDataMigration Full time

    About the Role: As a Senior Data Migration Consultant, you will play a pivotal role in delivering end-to-end data migration solutions to clients. Your primary responsibility will be to provide frontline support for SAP Data Migration Projects, addressing client requirements and participating in feasibility studies. You will develop strategies for data...

  • Software Engineer

    7 days ago

    hubballi, India beBeeBackend Full time

    Job DescriptionFintech companies seek backend engineers who are skilled in designing systems that handle large transactions. These engineers play a crucial role in creating seamless payment and banking features for millions of users.Backend engineers at fintech companies have the opportunity to adapt to various programming languages, including Java, Python,...

  • Chief Data Engineer

    4 days ago

    hubballi, India beBeeData Full time

    Enterprise Data Architect RoleJob Description:Design and implement a comprehensive data architecture strategy to align with business goals and objectives.Develop scalable and efficient data platforms using cloud-native technologies such as Snowflake, Databricks, and Azure Data Factory.Lead the modernization of existing data systems to improve performance,...

  • Senior Data Engineer

    4 days ago

    hubballi, India beBeeDataEngineering Full time

    Job OverviewWe are seeking a highly skilled Data Engineering Specialist to join our team. As a key member of the data engineering team, you will be responsible for designing and implementing scalable data pipelines, developing workflows using Apache Airflow and Spark, and leveraging AWS services to manage and optimize large-scale data infrastructure.The...

  • Big Data Engineer

    6 days ago

    hubballi, India beBeeDataEngineer Full time

    We are seeking a seasoned data engineer to spearhead the development of robust data acquisition frameworks.The ideal candidate will design, develop, and maintain scalable data pipelines that support large-scale analytics and operational data systems.You will work with real-time data streaming and message queuing using Kafka, build ETL/ELT pipelines using...

  • Chief Data Solutions Engineer

    7 days ago

    hubballi, India beBeeData Full time

    Job Title:Data ArchitectWe are seeking a highly skilled Data Architect to lead our data engineering efforts and design large-scale database systems.About the Role:Lead the design, development, and implementation of complex database systems, including data warehouses and data pipelines.Collaborate with cross-functional teams to understand business...

  • Principal Data Engineer

    2 days ago

    hubballi, India beBeeData Full time

    Senior Data Architect OpportunityWe are seeking a highly experienced Senior Data Architect to lead our technical design, architecture, and implementation of complex, scalable data solutions.Design and implement large-scale data platforms using SnowflakeDevelop ETL processes and large data migration solutionsLead technical teams in data engineering and data...

  • Cloud Data Engineer

    4 days ago

    hubballi, India beBeeDataEngineer Full time

    Senior Data Engineer Job DescriptionWe are building the future of healthcare analytics to design, build and scale robust data pipelines powering nationwide analytics and supporting our machine learning systems.About this role:This senior position involves designing, building and maintaining scalable ETL pipelines using Python (Pandas, PySpark) and SQL,...

  • Web Data Engineering Lead

    6 days ago

    hubballi, India beBeeDataEngineering Full time

    Job Title: Web Data Engineering LeadOverview:The role of Web Data Engineering Lead involves overseeing the development and maintenance of large-scale data engineering systems, including web crawling infrastructure.Design and implement scalable data pipelines with expertise in Python and SQL.Collaborate with cross-functional teams to translate data...