Large-Scale Data Engineering Specialist

2 weeks ago

kurnool, India beBeeDataEngineering Full time

Data Engineering Manager - Web CrawlingYou will be responsible for leading and mentoring teams in designing, building, and optimizing large-scale data pipelines using workflow orchestration tools like Airflow or Celery.We are looking for a highly skilled professional with hands-on expertise in Python and SQL for data extraction, transformation, processing, and storage, as well as experience working with cloud platforms for data infrastructure, deployments, and pipeline operations.The ideal candidate will have strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows.Lead and mentor data engineering and web crawling teams to ensure high-quality delivery and adherence to best practices.Design, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.Define and enforce data quality, validation, and security measures across all data flows and pipelines.Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions.Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure.Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling.Benefits: We offer a competitive compensation package, comprehensive benefits, and opportunities for growth and development.Required Skills and Qualifications: Proven experience in data engineering, web crawling, and AI-driven automation. Strong expertise in Python, SQL, and cloud platforms. Excellent leadership and communication skills.

  • Cybersecurity Developer

    2 weeks ago

    kurnool, India beBeeCybersecurity Full time

    Job OverviewXcitium is rapidly scaling its engineering team, seeking high-caliber developers to help build the future of a cybersecurity platform.This opportunity allows you to work on complex, large-scale, high-availability distributed systems while shaping the next generation of product ecosystem—trusted globally for endpoint protection, ZeroDwell...

  • Enterprise-Scale Integration Solutions Specialist

    2 weeks ago

    kurnool, India beBeeIntegration Full time

    Workato Integration Implementation SpecialistDeliver high-quality enterprise-scale integration solutions by designing, developing, and implementing Workato-based integrations for large-scale enterprise systems.Key Skills and QualificationsAt least 2+ years of hands-on experience implementing integrations using Workato.Proven experience in end-to-end...

  • Data Engineering Specialist

    1 week ago

    kurnool, India beBeeDataEngineering Full time

    Job Title:Data Engineering SpecialistAbout the Role:We are seeking a highly skilled Data Engineering Specialist to join our team. As a key member of our data engineering team, you will be responsible for designing and maintaining scalable, efficient, and well-partitioned schemas in MSSQL, Redshift, and Snowflake.You will work closely with software engineers,...

  • Cultivating Expertise in Large-Scale Generative AI Systems

    2 weeks ago

    kurnool, India beBeeartificial Full time

    Artificial Intelligence ArchitectWe are seeking a highly skilled Artificial Intelligence Architect to lead the design and development of large-scale generative AI systems. As a key member of our team, you will be responsible for architecting and implementing intelligent agents and automation workflows.AI System Design & Development: Develop...

  • Sap Data Quality

    4 weeks ago

    Kurnool, India Whatjobs IN C2 Full time

    Job Title: SAP Data Quality & Master Data Specialist (MRO/MDO/Content Management) Job Summary We are seeking an experienced(4Y+) SAP Data Quality & Master Data Specialist to manage, govern, and optimize data across SAP systems with a focus on MRO (Maintenance, Repair & Operations) data, MDO (Master Data Organization), and Content Management. The ideal...

  • Senior Data Engineering Specialist

    5 days ago

    kurnool, India beBeeDataEngineer Full time

    Key ResponsibilitiesData engineering involves designing and implementing end-to-end data pipelines following architecture patterns.Data transformations using SQL for large-scale data processing are key to our operations.Developing dimensional data models, aggregations, and KPI calculations in analytics consumption is crucial.We need someone to implement data...

  • Senior Google Cloud Data Engineering Specialist

    2 weeks ago

    kurnool, India beBeeDataEngineer Full time

    Job DescriptionWe are seeking a skilled GCP Data Engineer to design, develop, and maintain large-scale data processing systems on Google Cloud Platform (GCP).The ideal candidate will have expertise in designing and implementing data pipelines using GCP services such as Dataflow, BigQuery, and Storage Classes.

  • Data Center Specialist

    2 weeks ago

    kurnool, India beBeeDataCenter Full time

    Network Engineer SpecialistThis role focuses on delivering high-quality technical solutions, resolving complex issues, and driving technology adoption across large-scale environments.Key Responsibilities:• Deliver customer engagements and provide support for large-scale solutions and complex technical issues through deep domain expertise.• Drive customer...

  • Data Analytics Specialist

    2 weeks ago

    kurnool, India beBeeDataAnalytics Full time

    Job Title: Data Analytics SpecialistWe are seeking a skilled Data Analytics Specialist to join our team. This role offers an exceptional opportunity for hands-on experience with data analysis, statistical modeling, and trading strategies in digital asset markets.Key Responsibilities:Conduct comprehensive research on cryptocurrency markets, analyzing price...

  • Chief Data Systems Specialist

    1 week ago

    kurnool, India beBeeData Full time

    Key Data Engineer RoleWe are seeking a seasoned Data Engineer with hands-on experience in designing and implementing scalable data pipelines and cloud-based solutions.Create large-scale data architectures and ETL workflows for high-volume data handlingDevelop Python scripts for efficient data processing and analyticsDesign complex SQL queries for data...