Data Pipelines Engineering Manager – Python

Data Systems Engineering Manager – Python

Experience: 7 to 12 Years

Location: Remote (Work from Home) / Bangalore / India

Mode of Engagement: Full-time

No of Positions: 2

Educational Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT

Industry: IT / Data / AI / LegalTech / Enterprise Solutions

Notice Period: Immediate

What We Are Looking For:

• 7–12 years of experience in managing end-to-end data systems, including data collection, API-driven data delivery, and AI model integrations.
• Proven expertise in Python-based data engineering, data scraping, and pipeline orchestration for large-scale analytics and automation projects.
• Strong background in building scalable APIs, microservices, and data pipelines using Python, Docker, and Kubernetes.
• Experience in customizing open-source language models (LLMs) and integrating them into enterprise solutions.
• Proficiency in AWS cloud services, PostgreSQL/MySQL, CI/CD, and DevOps tools for continuous delivery and monitoring.
• Ability to lead distributed engineering teams, manage delivery timelines, and drive innovation through automation and AI-driven workflows.

Responsibilities:

• Lead and mentor engineering teams responsible for designing, implementing, and maintaining data ingestion, transformation, and delivery pipelines.
• Architect data systems combine scraping, big data processing, and AI-based enrichment to support analytics and product intelligence.
• Oversee API architecture and system scalability, ensuring high performance and secure data flow across services.
• Manage public data scraping and data integration projects that involve large, unstructured datasets and complex relational mappings.
• Drive AI model customization using open-source frameworks (LangChain, LlamaIndex, or custom LLM integrations) to improve automation and search intelligence.
• Collaborate with product managers, AI teams, and data analysts to translate requirements into actionable data engineering tasks.
• Implement and monitor data quality, governance, and compliance standards throughout the data lifecycle.
• Optimize deployment processes using Docker, Kubernetes, AWS Lambda, and CI/CD pipelines.
• Conduct regular technical reviews, mentor developers, and establish best practices for Python, cloud deployment, and automation.

Qualifications:

• Bachelor's or master's degree in computer science, Engineering, or related field.
• Hands-on expertise in Python, Django, FastAPI, SQL, and API-driven system design.
• Strong understanding of data pipelines, scraping frameworks (BeautifulSoup, Scrapy), and distributed data systems.
• Experience with AI/LLM integration, automation frameworks, and data-driven architecture design.
• Proficiency in AWS cloud ecosystem, Docker, Kubernetes, and Git-based CI/CD workflows.
• Demonstrated leadership in managing multi-member engineering teams and delivering enterprise-grade products.
• Excellent analytical, communication, and problem-solving abilities.
• Familiarity with Agile methodologies, Jira/ClickUp, and project management best practices.

Job Types: Full-time, Permanent

Pay: ₹60, ₹90,000.03 per month

Benefits:

• Health insurance
• Paid sick time
• Provident Fund
• Work from home

Application Question(s):

• How many end-to-end data pipeline or automation projects have you delivered successfully?
• What tools or frameworks have you used to build or orchestrate data pipelines (e.g., Airflow, Prefect, Luigi, or custom-built using Python)
• How many total years of experience do you have in Python-based data engineering (including data pipelines, data scraping, or API automation)?
• How many end-to-end data engineering or AI-integrated projects have you delivered or led successfully?
• How many years of experience do you have working with AWS or other cloud platforms (for deployment or data management)?
• How many large-scale data scraping projects have you handled (mention approximate count)?
• How many projects have involved AI or LLM integration (e.g., LangChain, LlamaIndex, or custom AI models)?
• Notice period (days)
• Current CTC (monthly)

Work Location: Remote