Data Engineering Manager – Web Crawling

3 days ago

Aligarh, India AIMLEAP Full time

Data Engineering Manager – Web Crawling & Pipeline Architecture Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate  What We Are Looking For Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture. Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery. Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage. Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations. Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling). Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..  Responsibilities Lead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage. Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies. Define and enforce data quality, validation, and security measures across all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure. Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling  Qualifications Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems. Strong expertise in Python, SQL, and modern data processing practices. Experience working with Airflow, Celery, or similar workflow automation tools. Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture. Hands-on experience with cloud data platforms (AWS/GCP/Azure). Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills. 

  • Ai web scraping engineer

    3 weeks ago

    Aligarh, India S2T AI - AI-Powered Investigations Full time

    We're seeking a forward-thinking Web Scraping Engineer who leverages AI tools to accelerate development and streamline data extraction processes. Join our India team and work at the intersection of traditional scraping expertise and cutting-edge AI-powered development. The Role: Design and implement scalable data extraction solutions using AI to rapidly...

  • Python Web Scraping Engineer – Automation

    3 days ago

    Aligarh, India AIMLEAP Full time

    Python Web Scraping Engineer – Advanced Automation (WFH) Experience: 3–10 Years Location: Remote (Work from Home) Mode of Engagement: Full-time No of Positions: 8 Educational Qualification: Bachelor’s degree in Computer Science, IT, or related field Industry: IT / Software Services / Data & AI Notice Period: Immediate Joiners Preferred What We...

  • Data Engineer

    3 weeks ago

    Aligarh, India IntraEdge Full time

    We are seeking a highly skilled Data Engineer with strong experience in Python, PySpark, Snowflake, and AWS Glue to join our growing data team. You will be responsible for building scalable and reliable data pipelines that drive business insights and operational efficiency. This role requires a deep understanding of data modeling, ETL frameworks, and...

  • Lead Data Engineer

    19 hours ago

    Aligarh, India Dr. Martens plc Full time

    SO, WHAT'S THE STORY? As a Lead Data Engineer, you will play a key role in the development and maintenance of the organization's data infrastructure. Working within a multi-disciplined team led by the Senior Manager - Data Engineer, you will focus on building and optimizing scalable data pipelines and supporting the delivery of high-quality, reliable data...

  • Data engineer

    2 weeks ago

    Aligarh, India Tata Consultancy Services Full time

    TCS has been a great pioneer in feeding the fire of young techies like you. We are global leaders in the technology arena and there's nothing that can stop us from growing together. TCS Hiring for skill " Digital: AWS Data Engineer". Role: AWS Data Engineer – Databricks Required Technical Skill Set: Data Lake architecture, AWS Services – S3, EMR,...

  • AI & Enterprise Application Architect

    4 weeks ago

    Aligarh, India RMT Engineering Full time

    Job Title: AI & Enterprise Application Architect About the Role We are looking for a highly skilled Architect who can lead the design and delivery of both AI-powered systems (including Agentic AI and GenAI applications) and enterprise Line-of- Business (LoB) applications. This role requires a visionary leader who combines deep technical expertise with...

  • Senior data engineer

    4 weeks ago

    Aligarh, India Quantiphi Full time

    Company Profile: Quantiphi is an award-winning Data Science and Machine Learning Software and Services Company focused on helping organizations translate the big promise of Machine Learning technologies into quantifiable business impact. We were founded on the belief that machine learning and artificial intelligence are transformative technologies that will...

  • Data Engineer

    6 days ago

    Aligarh, India KPI Partners Full time

    KPI Partners is seeking a talented Data Engineer with expertise in STIBO (STEP) development to join our dynamic team. The ideal candidate will be responsible for designing, developing, and implementing data solutions that enhance our data management practices and streamline processes.Experience - 3 Years to 6 YearsResponsibilitiesDevelop, implement, and...

  • Data Engineer

    6 days ago

    Aligarh, India KPI Partners Full time

    KPI Partners is seeking a talented Data Engineer with expertise in STIBO (STEP) development to join our dynamic team. The ideal candidate will be responsible for designing, developing, and implementing data solutions that enhance our data management practices and streamline processes.Experience - 3 Years to 6 YearsResponsibilitiesDevelop, implement, and...

  • Highly Skilled Data Engineering Professional Wanted

    2 weeks ago

    aligarh, India beBeeDataEngineer Full time

    Job OverviewThe role of a Data Engineer involves designing, developing, and maintaining large-scale data systems to meet business needs.Key Responsibilities:Design scalable data pipelines using PySpark, SQL, and Azure Data Factory.Implement Continuous Integration and Continuous Deployment processes using Jenkins and Azure DevOps.Collaborate with...