Freelance Deep Web Crawler Engineer

4 days ago

Guwahati, India Whatjobs IN C2 Full time

About the Project We’re developing a next-generation intelligent web crawling system capable of exploring deep and dynamic web data sources — including sites behind authentication, infinite scrolls, and JavaScript-heavy pages. The crawler will be integrated with an AI-driven pipeline for automated data understanding, classification, and transformation. We’re looking for a highly experienced engineer who has previously built large-scale, distributed crawling frameworks and integrated AI or NLP/LLM-based components for contextual data extraction. Key Responsibilities Design, develop, and deploy scalable deep web crawlers capable of bypassing common anti-bot mechanisms. Implement AI-integrated pipelines for data processing, entity extraction, and semantic categorization. Develop dynamic scraping systems for sites that rely on JavaScript, infinite scrolling, or APIs. Integrate with vector databases , LLM-based data labeling, or automated content enrichment modules. Optimize crawling logic for speed, reliability, and stealth across distributed environments. Collaborate on data pipeline orchestration using tools like Airflow, Prefect, or custom async architectures. Required Expertise Proven experience building deep or dark web crawlers (Playwright, Scrapy, Puppeteer, or custom async frameworks). Strong understanding of browser automation, session management, and anti-detection mechanisms . Experience integrating AI/ML/NLP pipelines — e.G., text classification, entity recognition, or embedding-based similarity. Skilled in asynchronous Python (asyncio, aiohttp, Playwright async API). Familiar with database and pipeline systems — PostgreSQL, MongoDB, Elasticsearch, or similar. Ability to design robust data flows that connect crawling → AI inference → storage/visualization. Nice to Have Knowledge of LLMs (OpenAI, Hugging Face, LangChain, or custom fine-tuned models) . Experience with data cleaning, deduplication, and normalization pipelines . Familiarity with distributed crawling frameworks (Ray, Celery, Kafka) . Prior experience integrating real-time analytics dashboards or monitoring tools. What We Offer Competitive freelance pay based on expertise and delivery. Flexible, async-first remote collaboration. Opportunity to shape an AI-first data platform from the ground up. Potential for long-term partnership if the collaboration is successful.

  • Freelance Content Writer

    2 days ago

    Guwahati, India North East Educational Trust Full time

    **Urgently required a passionate, smart & dedicated Freelance Content Writer for Web development of our Organization (Based in Guwahati, preferred) with proven intelligence & minimum 1 year experience** * **Skills**: - Need to manage Creative content in English. - Collect and organize material and complete writing assignments according to set standards...

  • SEO Content Writer

    3 days ago

    Guwahati, India Web Stars Full time

    Hey **Writers** its golden opprtunity for learning and grow your career in the Content writing Industry **Implementing various elements of a client's content marketing strategy as instructed by Project Managers**. Optimizing current copy on a client's website (page titles, meta descriptions, h1s, alts, anchor text, etc.) Writing new SEO-friendly copy for...

  • Web Developer

    2 weeks ago

    Guwahati, Assam, India Tebixa Technologies Full time ₹ 2,50,000 - ₹ 7,50,000 per year

    Knowledge in Website Development Using HTML, CSS, PHP. Should be able to develop websites as per the requirements and using best technology. Should have a basic understanding of Fronted and backend of website development.Role :Full Stack DeveloperIndustry Type: IT Services & ConsultingDepartment Engineering - Software & QAEmployment Type:Full Time,...

  • Search Engine Optimization Specialist

    7 days ago

    Guwahati, Assam, India Alterera Networks Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Company DescriptionAt Alterera, we help businesses turn bold ideas into scalable digital products. We offer robust SaaS and custom software development, white-label IT solutions, and flexible team extensions for scaling. We specialize in web and mobile app development, as well as DevOps and cloud services, serving over 10 industries with a proven track...

  • Sr. Front End Engineer

    2 weeks ago

    Guwahati, India InRhythm Full time

    At InRhythm, You Will Design, develop, and test web applications as well as work with fellow Software Engineers to navigate the discussion and strategy in re-architecting existing web platforms Identifying pitfalls ahead of time, raising concerns, and managing expectations with all stakeholders Define and break out epics/features for others, while...

  • Software Development Engineer-II

    2 weeks ago

    Guwahati, India Srujanee Full time

    Job Title: Software Development Engineer-IILocation: Remote Company: Srujanee Experience Required: 4-6 years Department: Product EngineeringBrief:Senior Full Stack Developerexperienced with Next.Js, Node.Js, MongoDb and Express.Js Responsibility: 1.⁠ ⁠Architecture & Development2.⁠ ⁠Design, develop, and maintain scalable applications using Next.js,...

  • Senior Civil Engineer

    1 week ago

    Guwahati, Assam, India PG Construction(PGC) Full time

    Company DescriptionPG Construction is a prominent construction company in Assam, established in 2004. Over the years, PGC has expanded from being a residential developer to engaging in commercial constructions, PWD roads, buildings, bridges, water resources, embankments, canals, and other real estate verticals. With innovation, quality, and sustainability at...

  • (Part Time) Content Writer

    2 days ago

    Guwahati, India Telstra Full time

    Data Entry Jobs in Guwahati, Assam Data Entry, Computer Operator, Back Office, Typist For more details or immediate response WhatssApp your name on this number **8 O5 2 8 4 94 44** After-sending messages, with in 2 minutes you will get complete information on your WhatssApp debate decade decide decision deep defense degree Democrat democratic describe...

  • Digital Collections-Product Engineering Lead

    4 days ago

    Guwahati, India Whatjobs IN C2 Full time

    Job Description: Digital Collections-Product Engineering Lead Company: Firstsource Solutions Limited Location: Bangalore, India (Preferred) Experience: 15+ Years Position Type: Full-Time About Firstsource Firstsource Solutions Limited, an RP-Sanjiv Goenka Group company (NSE: FSL, BSE: , Reuters: FISO.BO, Bloomberg: FSOL:IN), is a specialized global business...

  • Ai Engineer Intern

    1 week ago

    Guwahati, India Whatjobs IN C2 Full time

    Experience: 0–3 yearsType: Full-time, On-site (Bengaluru)About Mandrake Bioworks Mandrake Bioworks is an AI-first biotechnology company reimagining how we design and engineer life itself. Our mission is to build intelligent systems that make biology programmable . We’re unlocking a new generation of gene editing technologies that will power breakthroughs...