Senior Data Engineer

3 weeks ago

GTB Nager, India Markovate Full time

Job description

We are looking for an experienced Senior Data Engineer to lead the development of scalable AWS-native data lake pipelines, with a strong focus on time series forecasting, upsert-ready architectures, and enterprise-grade data governance. This role demands end-to-end ownership of the data lifecycle from ingestion to partitioning, versioning, QA, lineage tracking, and BI delivery.

The ideal candidate will be highly proficient in AWS data services, PySpark, and versioned storage formats such as Apache Hudi or Iceberg. A strong understanding of data quality, observability, governance, and metadata management in large-scale analytical systems is critical.

Roles & Responsibilities

  • Design and implement data lake zoning (Raw Clean Modeled) using Amazon S3, AWS Glue, and Athena.
  • Ingest structured and unstructured datasets including POS, USDA, Circana, and internal sales data.
  • Build versioned and upsert-ready ETL pipelines using Apache Hudi or Iceberg.
  • Create forecast-ready datasets with lagged, rolling, and trend features for revenue and occupancy modeling.
  • Optimize Athena datasets with partitioning, CTAS queries, and S3 metadata tagging.
  • Implement S3 lifecycle policies, intelligent file partitioning, and audit logging for performance and compliance.
  • Build reusable transformation logic using dbt-core or PySpark to support KPIs and time series outputs.
  • Integrate data quality frameworks such as Great Expectations, custom logs, and AWS CloudWatch for field-level validation and anomaly detection.
  • Apply data governance practices using tools like OpenMetadata or Atlan, enabling lineage tracking, data cataloging, and impact analysis.
  • Establish QA automation frameworks for pipeline validation, data regression testing, and UAT handoff.
  • Collaborate with BI, QA, and business teams to finalize schema design and deliverables for dashboard consumption.
  • Ensure compliance with enterprise data governance policies and enable discovery and collaboration through metadata platforms.

Preferred Candidate Profile

  • 9-12 years of experience in data engineering.
  • Deep hands-on experience with AWS Glue, Athena, S3, Step Functions, and Glue, Data Catalog.
  • Strong command over PySpark, dbt-core, CTAS query optimization, and advanced partition strategies.
  • Proven experience with versioned ingestion using Apache Hudi, Iceberg, or Delta Lake.
  • Experience in data lineage, metadata tagging, and governance tooling using OpenMetadata, Atlan, or similar platforms.
  • Proficiency in feature engineering for time series forecasting (lags, rolling windows, trends).
  • Expertise in Git-based workflows, CI/CD, and deployment automation (Bitbucket or similar).
  • Strong understanding of time series KPIs: revenue forecasts, occupancy trends, demand volatility, etc.
  • Knowledge of statistical forecasting frameworks (e.g., Prophet, GluonTS, Scikit-learn).
  • Experience with Superset or Streamlit for QA visualization and UAT testing.
  • Experience building data QA frameworks and embedding data validation checks at each stage of the ETL lifecycle.
  • Independent thinker capable of designing systems that scale with evolving business logic and compliance requirements.
  • Excellent communication skills for collaboration with BI, QA, data governance, and business stakeholders.
  • High attention to detail, especially around data accuracy, documentation, traceability, and auditability.

  • Data Engineer

    3 days ago

    GTB Nager, India Marktine Technology Solutions Pvt Ltd Full time

    Position: Senior Data EngineerExp: 12-15 YearsLocation: RemoteWe are seeking a Senior Data Engineer. The ideal candidate will be based in India and work remotely. This role requires a blend of design and implementation expertise.Key Responsibilities:Design and bootstrap data solutions, including setting up Snowflake for a large, complex conglomerate.Utilize...

  • Data Engineer

    4 weeks ago

    GTB Nager, India EnableMining Full time

    Job Title: Data EngineerCompany: EnableminingLocation: RemoteEmployment Type: Full-timeSeniority Level: Mid-LevelExperience: Minimum 2 yearsEducation: BE/BTech or MCAAbout UsEnablemining is a global mining consultancy headquartered in Australia. We specialize in strategy, mine planning, and technical evaluations for coal and metalliferous mines. Our work is...

  • Data Engineer

    2 weeks ago

    GTB Nager, India EnableMining Full time

    Job Title: Data Engineer Company: Enablemining Location: Remote Employment Type: Full-time Seniority Level: Mid-Level Experience: Minimum 2 years Education: BE/BTech or MCA About Us Enablemining is a global mining consultancy headquartered in Australia. We specialize in strategy, mine planning, and technical evaluations for coal and metalliferous...

  • Lead Azure Data Engineer

    4 weeks ago

    GTB Nager, India Celebal Technologies Full time

    Job Title: Lead Azure Data Engineer Experience Level: Mid - Senior Level Location: Delhi Duration: Fulltime Experience Required: 6-8+ Years Job Summary: We are looking for a Tech Lead – Data Engineering with 6+ years of hands-on experience in designing and building robust data pipelines and architectures on the Azure cloud platform. The ideal candidate...

  • Lead Azure Data Engineer

    4 weeks ago

    GTB Nager, India Celebal Technologies Full time

    Job Title: Lead Azure Data EngineerExperience Level: Mid - Senior LevelLocation: DelhiDuration: FulltimeExperience Required: 6-8+ YearsJob Summary:We are looking for a Tech Lead – Data Engineering with 6+ years of hands-on experience in designing and building robust data pipelines and architectures on the Azure cloud platform. The ideal candidate should...

  • Lead Azure Data Engineer

    3 weeks ago

    GTB Nager, India Celebal Technologies Full time

    Job Title: Lead Data Engineer Experience Level: Mid - Senior Level Location: DelhiDuration: FulltimeExperience Required: 6-8+ Years Description: We are seeking a highly skilled and experienced Lead Azure Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with a focus on working with Databricks, PySpark,...

  • Data Engineer

    4 weeks ago

    GTB Nager, India R Systems Full time

    We are looking for an experienced Data Engineer with strong expertise in Databricks and Azure Data Factory (ADF) to design, build, and manage scalable data pipelines and integration solutions. The ideal candidate will have a solid background in big data technologies, cloud platforms, and data processing frameworks to support enterprise-level data...

  • Data Engineer

    2 weeks ago

    GTB Nager, India R Systems Full time

    We are looking for an experienced Data Engineer with strong expertise in Databricks and Azure Data Factory (ADF) to design, build, and manage scalable data pipelines and integration solutions. The ideal candidate will have a solid background in big data technologies, cloud platforms, and data processing frameworks to support enterprise-level data...

  • Data Engineer

    2 weeks ago

    GTB Nager, India SMC Global Securities Ltd. Full time

    JD for Data Engineer : Hiring : Mid-Level Data Engineer Location : Delhi Experience Required : 4 -10 years Job Type : Full-Time Key Skills Required : - Strong proficiency in SQL and Python - Hands-on experience with AWS services, including : Athena, Redshift, EC2, EMR, Lambda, S3, Glue - Proven experience in building and maintaining batch & real-time data...

  • Senior Sales Engineer

    5 days ago

    GTB Nager, India ColorTokens Inc. Full time

    Senior Sales Engineer – APJC Location: India, Delhi (Remote/Field-Based) About ColorTokens ColorTokens, based in Silicon Valley, is a leader in enterprise and cloud cybersecurity. Our award-winning Zero Trust platform simplifies security deployment at scale by automating protection across endpoints, workloads, applications, and users in hybrid...