Technical Lead – Web Crawling Systems, Data Pipelines

1 week ago

Vadodara, India AIMLEAP Full time

Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate What We Are Looking For Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture. Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery. Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage. Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations. Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling). Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows.. Responsibilities Lead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling Qualifications Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS/GCP/Azure). Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.

  • Web Crawling Specialist

    3 days ago

    vadodara, India beBeeCrawler Full time

    Job Opportunity: Web Crawling SpecialistWe are seeking a highly skilled Web Crawling Specialist to join our team. As a key member, you will be responsible for designing and implementing web crawlers to extract valuable insights from the internet.The ideal candidate will have a strong background in Python programming and experience with web scraping...

  • Web Crawler Pipeline Architect

    7 days ago

    vadodara, India beBeeDataEngineering Full time

    Data Engineering Manager – Web CrawlingWe are seeking a seasoned Data Engineering Manager to spearhead our web crawling and pipeline architecture efforts.Lead and mentor data engineering and web crawling teams in driving scalable data pipelines using Airflow or CeleryCollaborate with cross-functional teams to design, build, and optimize cloud-based data...

  • Senior Data Crawling Specialist

    7 days ago

    vadodara, India beBeeData Full time

    Web Crawler DevelopmentOur organization is seeking a skilled web crawling engineer to design, build, and maintain efficient web crawlers.Maintaining and enhancing existing web scraping and data crawling projects is a core responsibility of the role.The ideal candidate will have expertise in developing and refining crawlers using Python-based tools and...

  • Senior Data Engineer and Technical Architect

    7 days ago

    vadodara, India beBeeDataEngineering Full time

    Job DescriptionWe are seeking a skilled Technical Architect with hands on coding to design and implement data engineering solutions using graph databases and data lake technologies.This role involves architecting and implementing ETL processes with cloud native technologies, utilizing Large Language Models (LLMs) for data extraction, transformation, and...

  • Data Pipeline Tester

    7 days ago

    vadodara, India beBeeDataPipeline Full time

    Job Title: Data Pipeline TesterWe are seeking a skilled Data Pipeline Tester to validate, analyze and ensure the accuracy, performance and reliability of data pipelines.Strong experience in data validation, SQL, ETL process testing and working with modern data platforms.Familiarity with Agile/Scrum methodology and defect management tools.Understanding of...

  • Data Pipeline Architect

    5 days ago

    vadodara, India beBeeCloudDataEngineer Full time

    Job DescriptionWe're seeking an experienced Cloud Data Engineer to spearhead the design, implementation, and maintenance of robust data pipelines and scalable data lakes.As a key member of our team, you'll work closely with data scientists, analysts, and stakeholders to deliver tailored solutions. Your expertise in cloud-native ETL tools like AWS DMS, AWS...

  • Data System Development Lead

    7 days ago

    vadodara, India beBeeDataDeveloper Full time

    Lead Data Systems DeveloperWe are seeking a seasoned professional with a proven track record of designing and building scalable data solutions and pipelines.Design, develop, and optimize large-scale data pipelines and workflows to support business growth and efficiencyCreate efficient and reusable code for data processing tasks using Python programming...

  • Data Systems Engineer

    5 days ago

    vadodara, India beBeeIntegration Full time

    Workday Integration Specialist Job OverviewWe are seeking an experienced Workday Integration Specialist to join our team. As a key member of our integration team, you will play a pivotal role in designing, building, and maintaining complex Workday integrations while leading the integration team to success.This position requires deep expertise in Workday...

  • Cloud-Native Data Engineering Specialist

    14 hours ago

    vadodara, India beBeeDataEngineer Full time

    About the RoleAs a Technical Architect, you will be responsible for designing and implementing cloud-native ETL processes. This role requires expertise in data engineering, including modelling and managing multi-dimensional entities and graphs. Knowledge of graph databases and data lake technologies is also essential.Experience with data/content crawling...

  • Web Scrapping(Internship)

    1 week ago

    Vadodara, India Career Skills Education & Research Foundation Full time

    **Job Description Position**: Web Scraping(Internship) **Company**: CareerNaksha Location: Vadodara, Gujarat **Work Mode**: Work from office / Work from home **Day schedule**:Day shift **About us**:Career Naksha is an Ed-tech startup providing modern personalised Career counselling & development to school and college students, graduates and...