Data Pipeline Automation Engineer
4 weeks ago
CategoryDetailsRoleSite Reliability Engineer (SRE) III – Data EngineeringLocationHyderabad- Employment TypeFull TimeExperience7–12 years in site reliability, cloud-based data infrastructure, data pipeline observability, automation, and high-availability engineering within EdTech platforms (2U)Primary Skills (Must-Have)AWS, CI/CD, Jenkins, IAAC, Terraform, KubernetesSecondary Skills (Good-to-Have)AWS systems; Dataiku data, Platform updates and patchingTools & PlatformsData Warehousing & Processing: Snowflake, Redshift, Apache Airflow, dbtCI/CD & Deployment: Jenkins, GitHub Actions, AWS CodePipeline, TerraformCloud & Event Processing: AWS Lambda, API Gateway, SNS/SQS, Kafka, Step FunctionsMonitoring & Logging: DataDog, AWS CloudWatch, Prometheus, SplunkIncident Management: PagerDuty, Opsgenie, AWS Health DashboardCollaboration & Code Review: GitHub, Jira, ConfluenceKey ResponsibilitiesData Pipeline Reliability & Observability:- Maintain and optimize highly available, fault-tolerant infrastructure for data pipelines, ETL jobs, and real-time data processing- Implement end-to-end monitoring of Airflow DAGs, Snowflake queries, and AWS-based data workflows- Automate data pipeline health checks, error handling, and auto-remediation strategies Infrastructure & Cloud Automation:- Deploy and manage AWS-based data infrastructure using Terraform and CloudFormation- Optimize Kubernetes (EKS) clusters for processing large-scale datasets and real-time analytics- Ensure high availability and cost-efficient scaling for Redshift, Snowflake, and data storage solutions Performance, Monitoring & Incident Response:- Implement real-time monitoring, logging, and alerting using DataDog, AWS CloudWatch, and Prometheus- Define and track SLOs, SLIs, and error budgets to improve data reliability and uptime- Conduct Root Cause Analysis (RCA), security audits, and post-mortems for incidents Security & Compliance:- Ensure GDPR, CCPA, and SOC 2 compliance for data storage, access controls, and retention policies- Implement AWS security best practices (IAM, KMS, Shield, WAF) to secure data access and encryption- Secure API gateways, authentication mechanisms, and data lake permissions to prevent unauthorized access Collaboration & Leadership:- Work closely with data engineers, analytics teams, and DevOps engineers to enhance data platform reliability- Participate in incident response drills, disaster recovery planning, and security compliance reviews- Advocate for best practices in automation, cost optimization, and cloud-native data solutions
-
IT Data Engineer
4 weeks ago
Hyderabad, India Tungsten Automation Full timeJob Description The IT Data Engineer plays a critical role in enabling data-driven decision-making across the organization by designing, building, and maintaining robust data infrastructure. This position is responsible for developing scalable data pipelines, managing databases and data warehouses, and ensuring the integrity, accessibility, and quality of...
-
Data Pipeline Engineer
4 weeks ago
Hyderabad, India Tanla Platforms Limited Full timeWe are seeking a skilled Data Engineer with 5–10 years of experience to join our dynamic DataEngineering team. The ideal candidate will design, build, and optimize large-scale data pipelines and warehouses that enable real-time analytics and business intelligence across Tanla’s CPaaS ecosystem.You’ll work on diverse datasets, including Call Detail...
-
Senior Automation Engineer
1 day ago
Hyderabad, Telangana, India IC Data Full time ₹ 12,00,000 - ₹ 36,00,000 per yearJob DescriptionPosition: Senior Software Development Engineer in TestReports To: Director, Software Development Engineer in TestLocation: IndiaJob OverviewAre you a talented, motivated engineer with a strong interest in testing? Join an innovative and fast-paced team-building software and infrastructure that allows litigation professionals to filter...
-
Lead Data Pipeline Engineer
3 weeks ago
Hyderabad, India Straive Full timeJob Summary We're looking for a Senior Data Engineer with 5-8 years of experience to build and maintain scalable, production-grade data pipelines. The ideal candidate is a strong software engineer with hands-on experience in Spark (3.X), Scala, SQL, and Python. You'll be responsible for designing and implementing ETL/ELT solutions, collaborating with teams...
-
Data Pipeline Operations Specialist
4 weeks ago
Hyderabad, India Tanla Platforms Limited Full timeJob Scope:Responsible for orchestrating, monitoring, and maintaining reliable and scalable data pipelines across Tanla’s CPaaS ecosystem.Ensure observability, reliability, and data quality across all data platforms through monitoring, alerting, and automated health checks.Participate in incident detection, troubleshooting, and resolution to minimize...
-
Senior Data Pipeline Developer
3 weeks ago
Hyderabad, India Sonata Software Full timeWe are seeking a highly skilled Senior Talend Developer to design, build, and optimize scalable ETL pipelines and data integration solutions. The role requires expertise in Talend, SQL, and cloud-based data platforms, along with the ability to lead technical design, mentor team members, and deliver high-quality data solutions aligned with business...
-
Data Pipeline Engineer
4 weeks ago
Hyderabad, India Optum Full timePrimary Responsibilities:Support the full data engineering lifecycle including research, proof of concepts, design, development, testing, deployment, and maintenance of data management solutionsUtilize knowledge of various data management technologies to drive data engineering projectsLead data acquisition efforts to gather data from various structured or...
-
Data Pipeline Architect
4 weeks ago
Hyderabad, India People Tech Group Inc Full timeWe are Hiring - Data Engineer Hyderabad !Job Title: Data EngineerLocation: HyderabadExperience: 5+YearsNotice : Immediate JoinersJob Summary:We are looking for a skilled and experienced Data Engineer with over 5 years of experience in data engineering and data migration projects. The ideal candidate should possess strong expertise in SQL, Python, data...
-
Azure Data Engineer
1 day ago
Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 25,00,000 per yearEnable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...
-
Data Pipeline Architect
4 weeks ago
Hyderabad, India Tanla Platforms Limited Full timeRole OverviewWe are seeking a skilled Data Engineer with 5–10 years of experience to join our dynamic Data Engineering team. The ideal candidate will design, build, and optimize large-scale data pipelines and warehouses that enable real-time analytics and business intelligence across Tanla’s CPaaS ecosystem. You’ll work on diverse datasets, including...