Senior Web Crawling Architect

2 weeks ago

tiruchirappalli, India beBeeDataEngineering Full time

Job OverviewWe are seeking an experienced Data Engineering Manager to lead our web crawling efforts.This role involves architecting, implementing, and optimizing scalable data pipelines to support high-volume data ingestion, transformation, and storage.Key ResponsibilitiesLead and mentor data engineering and web crawling teams to ensure high-quality delivery and adherence to best practices.Design and implement robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.RequirementsBachelor's or master's degree in a relevant field.7–12 years of experience in data engineering, pipeline design, or large-scale web crawling systems.Strong expertise in Python, SQL, and modern data processing practices.Experience working with Airflow, Celery, or similar workflow automation tools.Desirable SkillsSolid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture.Hands-on experience with cloud data platforms (AWS/GCP/Azure).Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).A competitive compensation package will be offered to the successful candidate.

  • Senior Technology Professional

    1 week ago

    tiruchirappalli, India beBeeTechnology Full time

    Job Opportunity:As a senior technology professional, you will develop innovative data crawling solutions to extract relevant information from public internet sources. You will apply techniques such as bot detection avoidance and VPN utilization to ensure efficient and effective data extraction.You will design and manage complex datasets and graphs using Data...

  • Data Extraction Expert

    1 week ago

    tiruchirappalli, India beBeeCrawler Full time

    Key Roles and ResponsibilitiesWe are seeking a Web Crawler Specialist who will be responsible for developing and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.Develop and refine crawlers using Python-based tools and frameworks to handle dynamic content.Collaborate with cross-functional teams to improve data...

  • Data Engineering Leader

    1 week ago

    tiruchirappalli, India beBeeTechnical Full time

    Job DescriptionAs a Technical Architect, you will be responsible for designing and implementing large-scale data engineering systems. This includes experience with data crawling from public internet sources, avoiding bot detection techniques, and utilizing VPNs.You will have expertise in data engineering, including modeling and managing multi-dimensional...

  • Data Architecture Expert

    2 weeks ago

    tiruchirappalli, India beBeeData Full time

    Job Title: Technical Architect with hands on codingAbout the RoleWe are seeking a skilled Technical Architect to join our team. As a key member of our architecture team, you will play a pivotal role in designing and developing scalable and efficient solutions.Key ResponsibilitiesDesign and develop data architectures for crawling public internet...

  • Senior Web Engineer

    2 weeks ago

    tiruchirappalli, India beBeeSoftware Full time

    Senior Web EngineerWe are seeking a highly skilled and experienced Front-End Software Professional with strong proficiency in programming languages to join our core development team.This is a full-time position offering the opportunity to contribute directly to the design and development of cutting-edge web applications, trading interfaces, and real-time...

  • Senior Architect

    4 weeks ago

    Tiruchirappalli, India APICES Studio Full time

    About APICES Studio Pvt. Ltd.:APICES Studio Pvt. Ltd. is a Mumbai based award winning Architectural Design Organization with team of 125+ . We are involved in the professional practice of Architecture, Planning, Consultancy and Engineering Services.To know more about us kindly visit https://www.apices.in/Join our pioneering, multi-disciplinary team, where...

  • Senior Architect

    3 weeks ago

    Tiruchirappalli, India APICES Studio Full time

    About APICES Studio Pvt. Ltd.:APICES Studio Pvt. Ltd. is a Mumbai based award winning Architectural Design Organization with team of 125+ . We are involved in the professional practice of Architecture, Planning, Consultancy and Engineering Services.To know more about us kindly visit https://www.apices.in/Join our pioneering, multi-disciplinary team, where...

  • Senior Data Architect Position

    4 days ago

    tiruchirappalli, India beBeeData Full time

    Senior Data Architect Position OverviewThe role of a Senior Data Architect involves designing, implementing and managing large-scale data platforms to meet business needs. Key responsibilities include leading solution design and delivery execution for data analytics programs across various domains.Design end-to-end data platform architectures and build...

  • Chief Software Architect

    2 weeks ago

    tiruchirappalli, India beBeeBackend Full time

    Senior Backend EngineerAbout the roleWe are seeking a seasoned Senior Backend Engineer to join our team. You will be responsible for developing and maintaining highly efficient and scalable software systems, architecting secured backend infrastructure, and designing, developing, and maintaining robust backend services.Key Responsibilities:Design and develop...

  • Senior Frontend Architect

    4 days ago

    tiruchirappalli, India beBeeFrontend Full time

    Job Title: Senior Frontend ArchitectThis role involves designing and building large-scale frontend applications, as well as collaborating with backend teams to ensure seamless application behavior.The ideal candidate will have a strong proficiency in JavaScript, TypeScript, and modern frontend frameworks (React preferred; Angular/Vue acceptable if strong...