Data Pipeline Architect

7 days ago

vadodara, India beBeeCloudDataEngineer Full time

Job DescriptionWe're seeking an experienced Cloud Data Engineer to spearhead the design, implementation, and maintenance of robust data pipelines and scalable data lakes.As a key member of our team, you'll work closely with data scientists, analysts, and stakeholders to deliver tailored solutions. Your expertise in cloud-native ETL tools like AWS DMS, AWS Glue, Kafka, Azure Data Factory, GCP Dataflow, and more will be crucial in ensuring seamless data integration and management.Key ResponsibilitiesDesigning and developing scalable ETL pipelines using cloud-native toolsArchitecting and implementing data lakes and data warehouses on cloud platformsDeveloping and optimizing data ingestion, transformation, and loading processes using Databricks, Snowflake, Redshift, and BigQueryImplementing ETL processes using tools like Informatica and SAP Data IntelligenceDeveloping and optimizing data processing jobs using Spark ScalaYou will also:Integrate various data sources, including relational databases, APIs, unstructured data, and ERP systems into the data lakeEnsure data quality and integrity through rigorous testing and validationPerform data extraction from SAP or ERP systems when necessaryMonitor and optimize the performance of data pipelines and ETL processesImplement best practices for data management, including data governance, security, and complianceA suitable candidate should possess:Required Skills and Qualifications7+ years of experience as a Data Engineer or in a similar roleProven experience with cloud platforms: AWS, Azure, and GCPHands-on experience with cloud-native ETL tools such as AWS DMS, AWS Glue, Kafka, Azure Data Factory, GCP Dataflow, etc.Experience with other ETL tools like Informatica, SAP Data Intelligence, etc.Experience in building and managing data lakes and data warehousesProficiency with data platforms like Redshift, Snowflake, BigQuery, Databricks, and Azure SynapseStrong experience with Spark and Scala for data processingBenefits100% remote workFlexibilityTrainingState-of-the-art techTeam

  • Data Architect

    1 week ago

    vadodara, India beBeeBackend Full time

    Software ArchitectWe are seeking an experienced Software Architect to design the backbone of our AI System.Key Responsibilities:Standardize Data Models: Develop and maintain canonical data structures that normalize conflicting metrics across disparate sources.Build Deep Pipelines: Construct data pipelines to ingest high-fidelity signals directly from the...

  • Data Pipeline Tester

    1 week ago

    vadodara, India beBeeDataPipeline Full time

    Job Title: Data Pipeline TesterWe are seeking a skilled Data Pipeline Tester to validate, analyze and ensure the accuracy, performance and reliability of data pipelines.Strong experience in data validation, SQL, ETL process testing and working with modern data platforms.Familiarity with Agile/Scrum methodology and defect management tools.Understanding of...

  • Web Crawler Pipeline Architect

    1 week ago

    vadodara, India beBeeDataEngineering Full time

    Data Engineering Manager – Web CrawlingWe are seeking a seasoned Data Engineering Manager to spearhead our web crawling and pipeline architecture efforts.Lead and mentor data engineering and web crawling teams in driving scalable data pipelines using Airflow or CeleryCollaborate with cross-functional teams to design, build, and optimize cloud-based data...

  • Technical Lead – Web Crawling Systems, Data Pipelines

    1 week ago

    Vadodara, India AIMLEAP Full time

    Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate What We Are Looking For Proven experience leading data engineering teams with strong ownership of web crawling...

  • Technical Lead – Web Crawling Systems, Data Pipelines

    1 week ago

    Vadodara, India AIMLEAP Full time

    Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate What We Are Looking For Proven experience leading data engineering teams with strong ownership of web crawling...

  • Data Platform Architect

    7 days ago

    vadodara, India beBeeSolutions Full time

    Senior Solutions Architect Role OverviewWe are seeking a highly skilled Senior Solutions Architect to lead solution design and delivery execution for data, analytics, and AI programs across Clinical, Commercial, and Medical Affairs domains.This individual will provide architectural leadership, guide engineering teams, and ensure high-quality, compliant...

  • Data Pipeline and API Tester

    7 days ago

    vadodara, India beBeeData Full time

    Job DescriptionWe are seeking a highly skilled Data Pipeline and API Testing professional to join our team. The ideal candidate will have strong hands-on experience in ETL testing, data pipeline testing, and API testing.Key ResponsibilitiesETL / Data Pipeline Testing: Perform end-to-end testing of ETL pipelines across cloud/on-prem environments.API Testing:...

  • Strategic Data Architect

    2 weeks ago

    vadodara, India beBeeDataEngineer Full time

    Job OverviewAs a seasoned data engineer, you will be responsible for architecting and implementing large-scale data architectures using Azure Data Services. Manage and configure Apache Spark clusters on Linux platforms.Design, optimize, and deploy ETL/ELT pipelines using the DataFrame API in PySpark.Deploy PySpark jobs using spark-submit in client/cluster...

  • Real-Time Data Pipeline Developer

    5 days ago

    vadodara, India beBeeData Full time

    Cloud Data EngineerThe ideal candidate will design and build real-time data ingestion pipelines using Apache NiFi for both structured and unstructured datasets, integrating them with the Google Cloud Platform (GCP) ecosystem.Data Ingestion Expertise: Design, implement, and optimize real-time data ingestion pipelines in Apache NiFi.Integration Specialist:...

  • Cloud-Native Data Architect

    6 days ago

    vadodara, India beBeeDataEngineer Full time

    AWS Data Engineer RoleWe're seeking a seasoned professional to design and maintain scalable, efficient data architectures on AWS. Your technical expertise in AWS services, data integration, and real-time/batch data processing will be essential in architecting and maintaining complex queries, stored procedures, indexing strategies, and partitioning for large...