Senior Data Engineer

4 weeks ago

Bhavnagar, Gujarat, India Markovate Full time

Job description

We are looking for an experienced Senior Data Engineer to lead the development of scalable AWS-native data lake pipelines, with a strong focus on time series forecasting, upsert-ready architectures, and enterprise-grade data governance. This role demands end-to-end ownership of the data lifecycle from ingestion to partitioning, versioning, QA, lineage tracking, and BI delivery.

The ideal candidate will be highly proficient in AWS data services, PySpark, and versioned storage formats such as Apache Hudi or Iceberg. A strong understanding of data quality, observability, governance, and metadata management in large-scale analytical systems is critical.

Roles & Responsibilities

  • Design and implement data lake zoning (Raw Clean Modeled) using Amazon S3, AWS Glue, and Athena.
  • Ingest structured and unstructured datasets including POS, USDA, Circana, and internal sales data.
  • Build versioned and upsert-ready ETL pipelines using Apache Hudi or Iceberg.
  • Create forecast-ready datasets with lagged, rolling, and trend features for revenue and occupancy modeling.
  • Optimize Athena datasets with partitioning, CTAS queries, and S3 metadata tagging.
  • Implement S3 lifecycle policies, intelligent file partitioning, and audit logging for performance and compliance.
  • Build reusable transformation logic using dbt-core or PySpark to support KPIs and time series outputs.
  • Integrate data quality frameworks such as Great Expectations, custom logs, and AWS CloudWatch for field-level validation and anomaly detection.
  • Apply data governance practices using tools like OpenMetadata or Atlan, enabling lineage tracking, data cataloging, and impact analysis.
  • Establish QA automation frameworks for pipeline validation, data regression testing, and UAT handoff.
  • Collaborate with BI, QA, and business teams to finalize schema design and deliverables for dashboard consumption.
  • Ensure compliance with enterprise data governance policies and enable discovery and collaboration through metadata platforms.

Preferred Candidate Profile

  • 9-12 years of experience in data engineering.
  • Deep hands-on experience with AWS Glue, Athena, S3, Step Functions, and Glue, Data Catalog.
  • Strong command over PySpark, dbt-core, CTAS query optimization, and advanced partition strategies.
  • Proven experience with versioned ingestion using Apache Hudi, Iceberg, or Delta Lake.
  • Experience in data lineage, metadata tagging, and governance tooling using OpenMetadata, Atlan, or similar platforms.
  • Proficiency in feature engineering for time series forecasting (lags, rolling windows, trends).
  • Expertise in Git-based workflows, CI/CD, and deployment automation (Bitbucket or similar).
  • Strong understanding of time series KPIs: revenue forecasts, occupancy trends, demand volatility, etc.
  • Knowledge of statistical forecasting frameworks (e.g., Prophet, GluonTS, Scikit-learn).
  • Experience with Superset or Streamlit for QA visualization and UAT testing.
  • Experience building data QA frameworks and embedding data validation checks at each stage of the ETL lifecycle.
  • Independent thinker capable of designing systems that scale with evolving business logic and compliance requirements.
  • Excellent communication skills for collaboration with BI, QA, data governance, and business stakeholders.
  • High attention to detail, especially around data accuracy, documentation, traceability, and auditability.

  • Senior Data Engineer

    2 days ago

    Bhavnagar, Gujarat, India The Mastermind Group Full time

    Senior Data Engineer Location: Remote (India) – Occasional travel to Riyadh, KSAContract Type: 12-Month ContractWork Week: Sunday–Thursday preferred (9 AM – 5 PM IST)Start Date: As soon as possibleAbout The Mastermind Group (TMG)TMG is a global leader in IT Spend Optimisation, helping enterprises take control of their tech investments across software,...

  • Senior Data Engineer

    1 day ago

    Bhavnagar, Gujarat, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Engineering Leadership RoleThis position involves leading a team of data engineers to drive the development and maintenance of our data warehouse infrastructure.Prior experience as a principal or senior data engineer is essential for this role.A strong background in cloud native data engineering, particularly with Amazon Web Services, is...

  • Senior Data Engineering Lead

    3 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior ConsultantWe are seeking a seasoned data engineering professional to lead the design and development of end-to-end data integration solutions.Main Responsibilities:Develop, implement and maintain efficient ETL/ELT workflows using Informatica IICS.Integrate IICS solutions with cloud-based platforms like Google Cloud Platform (GCP) and Google...

  • Transformative Senior Data Professional

    7 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team and play a key role in helping clients optimize their operations and achieve strategic goals through data-driven insights.Main Responsibilities:Develop and implement innovative solutions to enhance business performance and competitive advantage using GCP Data...

  • Cloud Data Engineering Leader

    4 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Opportunity: Senior Data EngineerWe are seeking a seasoned and proficient data engineer with substantial experience in cloud technologies to join our dynamic team.The ideal candidate will possess a strong command of Python, data engineering principles, and a proven track record of successful implementation of scalable solutions in cloud...

  • Data Science and Engineering Leader

    6 days ago

    Bhavnagar, Gujarat, India beBeeMachineLearning Full time ₹ 15,00,000 - ₹ 27,00,000

    Senior Machine Learning Engineer Job SummaryWe are seeking a highly skilled Senior Machine Learning Engineer to join our team. The successful candidate will be responsible for developing and deploying predictive models using machine learning techniques.Develop and train advanced predictive models using supervised and unsupervised learning...

  • Senior Data Engineering Professional

    3 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,30,00,000

    Big Data Engineer LeadWe are seeking a highly skilled and experienced Big Data Engineer Lead to join our team. As a Big Data Engineer Lead, you will be responsible for leading a team of big data engineers and working closely with clients and team members to understand their requirements and develop architectures that meet their needs.

  • Senior Data Engineer Role

    5 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Engineering ProfessionalWe are seeking a highly skilled Data Engineering professional to join our team. As a key member of the data engineering team, you will be responsible for developing and maintaining large-scale data processing systems that support business analytics initiatives.

  • Senior Cloud Data Engineer

    1 day ago

    Bhavnagar, Gujarat, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Cloud Data Engineering RoleWe are seeking a skilled data engineer to design and implement scalable data solutions using SQL and Azure services. The ideal candidate will have hands-on experience with T-SQL, SSIS, and Azure Data Factory.Key Responsibilities:Develop and optimize complex T-SQL queries, stored procedures, views, and functions for data processing...

  • Senior Data Platform Architect

    6 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineering Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Imagine building large-scale data platforms on Google Cloud. We're seeking a senior data engineer to create enterprise-grade data solutions using modern tools and practices.Job DescriptionOur ideal candidate will design and build robust data pipelines for ingestion, transformation, and analytics. They'll work with Big Data tools like Hadoop, Spark, Kafka,...