Senior Web Crawling Data Engineer

7 days ago


rajahmundry, India beBeeDataEngineering Full time

Job OverviewWe are seeking an experienced data engineering leader to spearhead our web crawling and pipeline architecture initiatives.The successful candidate will be responsible for leading and mentoring data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices.They will design, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.A strong understanding of modern web crawling frameworks and IP rotation techniques is essential.Experience with distributed processing technologies such as Airflow or Celery is required.The ideal candidate will also have expertise in defining and enforcing data quality, validation, and security measures across all data flows and pipelines.Collaboration with cross-functional teams to translate data requirements into technical solutions is a key aspect of this role.Additionally, the candidate should have hands-on experience with cloud-based deployments on AWS/GCP/Azure and AI/LLM frameworks.RequirementsBachelor's or master's degree in engineering, computer science, or related field.At least 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems.Strong expertise in Python, SQL, and modern data processing practices.Proficiency in proxy systems, anti-bot techniques, and scalable crawler architecture is necessary.Hands-on experience with cloud data platforms and AI/LLM frameworks is required.BenefitsOpportunity to work with cutting-edge technology and innovative projects.Collaborative and dynamic work environment.Professional growth and development opportunities.



  • rajahmundry, India beBeeEngineer Full time

    Web Crawling SpecialistWe are seeking a skilled web crawling specialist to join our team. In this role, you will be responsible for designing and implementing web crawlers, extracting valuable insights from the web, and ensuring data quality.


  • rajahmundry, India beBeeCrawling Full time

    Job TitleWe are seeking a seasoned professional to take up the role of Web Crawler Specialist. The ideal candidate will be responsible for designing, developing and maintaining web crawlers that can effectively extract valuable insights from the web.Maintain and enhance existing web scraping projects.Develop and refine crawlers using Python-based tools and...


  • rajahmundry, India beBeeDataEngineering Full time

    Job Title: Data Engineering SpecialistWe are seeking a highly skilled data engineering specialist to join our team. The ideal candidate will have hands-on experience with data/content crawling, bot detection avoidance techniques, and the use of VPNs.Main Responsibilities:Designing and implementing efficient ETL processes using cloud-native...


  • rajahmundry, India beBeeAutomation Full time

    Job OpportunityWe are looking for an Associate Automation Engineer to fill this role. As an entry-level position, you will gain expertise in Appium Pro and enjoy building web UIs.Write automation code for mobile applicationsCapture data and complete analysis on mobile applicationsCreate a dashboard with the captured data and provide it to customersAssist...


  • rajahmundry, India beBeeData Full time

    Job OpportunityWe seek an experienced Senior Data Engineer to lead the design, development, and implementation of robust data solutions on Microsoft Azure.Key Responsibilities:Data Engineering Expertise: Proven experience with SQL databases (e.g., Azure SQL/SQL Server), including query optimization and performance tuning.Azure Services: Advanced expertise in...


  • rajahmundry, India beBeeDataEngineer Full time

    Job DescriptionThe ideal candidate should have hands-on coding experience and expertise in data engineering, including modeling and managing multi-dimensional entities and graphs.They must also have knowledge of graph databases and data lake technologies, along with experience in architecting and implementing ETL processes using cloud-native...


  • rajahmundry, India beBeeEngineer Full time

    Job OverviewThe Director of Frontend Engineering leads a high-performing team focused on building scalable and efficient web applications.Key Responsibilities:Own and evolve the frontend architecture and component strategy to drive business growth.Manage a team of 10+ frontend engineers and managers to deliver high-quality React/Next.js applications.Develop...


  • rajahmundry, India beBeeFrontend Full time

    We are seeking a visionary leader to spearhead our Web Fabric initiative. As Principal Frontend Engineer, you will define the architecture, scale teams, and ensure low-latency, high-quality user experience across all products.This leader will manage a team of 10+ engineers and managers, aligning with Data Engineering and ML/API teams to deliver end-to-end...


  • rajahmundry, India beBeeDataEngineer Full time

    The following job description is for a Senior Data Engineer with expertise in Databricks. About the RoleWe are seeking an experienced Data Engineer to support our enterprise-wide Sustainability initiative. The ideal candidate will have extensive knowledge of ETL/ELT pipelines on Databricks using PySpark and SQL, as well as experience working with data...

  • Senior Data Engineer

    3 weeks ago


    Rajahmundry, India Live Connections Full time

    📌 Hiring: Data Engineer | 9-12 Years Experience | Remote (India) 💰 Salary: Up to 26 LPA 🛠️ Skills: Python, SQL, Big Data, Cloud, SnowflakeJob Description: We are looking for an experienced Data Engineer with strong hands-on expertise in designing and building scalable data pipelines and cloud-based data solutions. The ideal candidate will have...