Senior Data Engineering Manager

6 days ago

Gulbarga, Karnataka, India beBeeDataEngineering Full time ₹ 10,00,000 - ₹ 15,00,000

Job Title: Senior Data Engineering Manager

We are seeking an experienced and skilled Senior Data Engineering Manager to oversee the development and implementation of large-scale data scraping solutions.

This role requires expertise in advanced scraping techniques, reverse engineering, automation, and leadership skills to drive a team towards success.

Key Responsibilities:

  • Design and develop scalable data scraping solutions using tools like Scrapy and Python libraries.
  • Mentor and lead a team of 5+ developers, managing project timelines and deliverables.
  • Implement advanced blocking and captcha-solving techniques to bypass scraping restrictions.
  • Conduct source code reverse engineering and automate web and app interactions.
  • Manage proxies, IP rotation, and SSL unpinning to ensure effective scraping.
  • Maintain and improve API integrations and data pipelines.
  • Ensure code quality through effective version control, error handling, and documentation.
  • Collaborate with cross-functional teams for project planning and execution.
  • Monitor performance and provide solutions under high-pressure environments.

Required Skills and Experience:

  • Data Scraping: Minimum 4 years in IT scraping industry.
  • Leadership: Minimum 2 years leading a team of 5+ developers.
  • Scraping Tools: Scrapy, Threading, requests, web automation.
  • Technical Proficiency:
    • Advanced Python programming skills.
    • Captcha solving and blocking handling techniques.
    • Source reverse engineering expertise.
    • Proxy management & IP rotation knowledge.
    • App automation, SSL Unpin, Frida proficiency.
    • API Management, Version Control Systems.
    • Error Handling, SQL, MongoDB, Pandas.
  • Leadership Skills:
    • Basic project management skills.
    • Moderate documentation skills.
    • Team handling abilities.
    • Pressure management skills.
    • Flexibility and adaptability.
    • High accountability.

Preferred (Good to Have):

  • Experience with Linux operating system.
  • Knowledge of Appium, Fiddler, Burp Suite tools.

  • Senior Data Engineering Specialist

    5 days ago

    Gulbarga, Karnataka, India beBeeDataEngineering Full time US$ 60,000 - US$ 80,000

    Senior Data Engineering SpecialistWe are seeking a seasoned Senior Data Engineering Specialist to join our global engineering team.This key role will involve leveraging expertise in data engineering on a greenfield multi-year project initiative, utilizing strong experience with Palantir Foundry, Workshop, PySpark, and Typescript.Palantir Foundry / Workshop:...

  • Senior Data Architect

    1 week ago

    Gulbarga, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Key Senior Data Engineering PositionWe are seeking a senior data engineer to join our global intelligence practice, working with clients to design, develop and maintain large-scale data engineering solutions.Design and implement analytical reporting, compliance and data governance solutions that meet or exceed client expectations.Collaborate closely with...

  • Senior Cloud-Native Data Engineer

    6 days ago

    Gulbarga, Karnataka, India beBeeDataEngineering Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Senior Data Engineering SpecialistJob Description:We are seeking a seasoned data engineering specialist to join our team. The ideal candidate will have a strong background in cloud-native data engineering, with expertise in designing and implementing scalable data pipelines using Azure.This role will play a pivotal part in developing and optimizing data...

  • High-Performance Data Engineer

    6 days ago

    Gulbarga, Karnataka, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior Data Engineering LeadAbout the Position:We are seeking a seasoned data engineering expert to lead our team of talented professionals in designing and delivering high-performance, scalable data pipelines and analytics platforms on Microsoft Azure.Key Responsibilities:Mentor and grow a team of data engineers working on large-scale distributed data...

  • Senior Data Engineer

    4 days ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    As a seasoned data professional, you will be responsible for developing and maintaining large-scale data pipelines using Python and AWS.About the RoleThis position involves designing and implementing efficient data solutions to drive business growth through data analysis and interpretation.You will collaborate with cross-functional teams to ensure data...

  • Data Engineer Role

    4 days ago

    Gulbarga, Karnataka, India beBeeDataArchitect Full time ₹ 18,00,000 - ₹ 24,00,000

    Senior Data Architect PositionWe are seeking a highly skilled Senior Data Architect to join our team. The ideal candidate will have expertise in designing and implementing scalable data architectures, developing and optimizing ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics.This role involves performing data ingestion,...

  • Senior Associate, Data Management

    2 weeks ago

    Gulbarga, Karnataka, India KMK Consulting Inc. Full time

    KMK is a global data analytics and technology consulting company empowering leaders across the Life Sciences industries to make better data-driven decisions. Our data analytics and software platforms support data science, commercial operations, real-world evidence, and cloud information management. We help to optimize business strategy and operations by...

  • High-Level Data Engineer

    3 days ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    About the RoleWe are seeking a seasoned Senior Data Engineer to join our elite team of professionals. In this position, you will be responsible for designing, developing, and maintaining large-scale data processing systems that enable advanced analytics and business intelligence.

  • Enterprise Data Engineer

    4 days ago

    Gulbarga, Karnataka, India beBeeDataScientist Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineer PositionWe are seeking a skilled professional to fill this critical role.Minimum of 4 years of experience with Azure Databricks, Py-spark, SQL, Python, AutomationDatabase Administration: Hands-on expertise in SQL and NoSQL databasesDevOps & Cloud Computing: Familiarity with AWS, GCP, or Azure, and containerization tools like Docker/KubernetesC...

  • Principal Data Engineering Specialist

    5 days ago

    Gulbarga, Karnataka, India beBeeDataEngineering Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Senior Data Engineer Role OverviewWe are seeking a highly skilled data engineer to join our team. This is an exciting opportunity for an individual who is passionate about working on complex data engineering projects.The ideal candidate will have a strong background in ETL development and data engineering, with expertise in Snowflake and data pipeline tools...