Data Engineering Pipeline Architect

6 days ago


vellore, India beBeeData Full time

Web Crawler Architecture LeadWe are seeking a skilled Web Crawler Architecture Lead to oversee our web crawling and pipeline architecture teams.Oversee web engineering and crawling teams, ensuring high-quality delivery and adherence to best practices.Design, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Develop and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.Define and enforce data quality, validation, and security measures across all data flows and pipelines.The ideal candidate will have:A Bachelor's or Master's degree in computer science, information technology, or a related field.7-12 years of relevant experience in web engineering, pipeline design, or large-scale web crawling systems.Strong expertise in Python, SQL, and modern data processing practices.Experience working with Airflow, Celery, or similar workflow automation tools.Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture.Hands-on experience with cloud data platforms (AWS/GCP/Azure).Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).



  • vellore, India beBeeData Full time

    Lead Data EngineerUnlock your potential as a talented data engineer, contributing to our mission of harnessing the power of data.As a lead data engineer, you will play a pivotal role in designing and implementing scalable data pipelines and ETL processes that drive business success.You will collaborate with cross-functional teams to understand business...


  • vellore, India beBeeDataEngineer Full time

    Key Roles and Responsibilities:We are seeking a highly skilled Senior Data Engineer to design and implement robust data pipelines that power nationwide analytics and support machine learning systems.This is an exciting opportunity to build scalable ETL pipelines using Python and SQL, orchestrated with Airflow. The ideal candidate will have a strong...


  • Vellore, India Quantiphi Full time

    Company ProfileQuantiphi is an award-winning AI-first digital transformation engineering company driven by the desire to solve transformational problems at the heart of the business. We are passionate about our customers and obsessed with problem-solving. Quantiphi solves our clients’ toughest and most complex business problems by combining deep industry...


  • vellore, India beBeeDataEngineer Full time

    Our organization is seeking a highly skilled Senior Data Engineer to fill a key role in our data management team. In this position, you will be responsible for designing and architecting large database systems, particularly on Microsoft platforms such as SQL Server and Azure Database.The ideal candidate will have a minimum of 7 years of experience in data...


  • vellore, India beBeeDataEngineering Full time

    Data Engineering SpecialistThis role involves leading the development, optimization, and maintenance of data pipelines and back-end services within a microservices environment.Key Responsibilities:Design and implement scalable data pipelines for ingestion, processing, and storage of data from various sources.Craft robust data solutions and APIs that power...


  • vellore, India beBeeAzure Full time

    Job Title: Azure Data ArchitectThe ideal candidate should have a deep understanding of data warehousing concepts, medallion architecture, and data pipeline engineering.This role requires strong leadership to manage the successful delivery of multiple data projects and provide technical guidance to data engineering teams. The successful candidate will be...

  • Chief Data Architect

    10 hours ago


    vellore, India beBeeData Full time

    Job OpportunityData Engineering ExcellenceWe are seeking an experienced Senior Data Engineer with in-depth knowledge of Oracle Data Integrator (ODI) and a strong understanding of Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) frameworks. This role involves designing and developing robust data pipelines across on-premises and cloud Oracle...


  • vellore, India beBeeDataLead Full time

    Azure Data Architect RoleWe are seeking a seasoned Data Engineer to lead our Azure data initiative. The ideal candidate will have at least 6 years of experience in a data engineering environment and expertise in CI-CD, PySpark, SQL, ADF, ADLS, ADB, and Azure Services.The successful candidate will also have experience with GIT, Jenkins, and Azure DevOps. They...


  • Vellore, India Whatjobs IN C2 Full time

    Looking for a Freelance Data Architect to join a team of rockstar developers. The candidate should have a minimum of 12+ yrs. of experience. There are multiple openings. If you're looking for freelance/ part time opportunity (along with your day job) & a chance to work with the top 0.1% of developers in the industry, this one is for you! You will report into...

  • Big Data Architect

    6 days ago


    vellore, India beBeeDataEngineer Full time

    Job DescriptionAn Azure Data Engineer will build, implement and maintain scalable big data architectures using Microsoft Azure Databricks, Synapse Analytics, ADLS Gen 2 and other related technologies. This role requires a strong background in data engineering, with experience in ETL, ELT, data warehousing, data governance and compliance.Required Skills and...