Lead Data Engineer

2 weeks ago

Ahmedabad, Gujarat, India Advantmed Full time ₹ 1,20,00,000 - ₹ 2,50,00,000 per year

We are seeking a talented and experienced Senior Data Engineer to join our team and contribute to building a robust data platform on Azure Cloud. The ideal candidate will have hands-on experience designing and managing data pipelines, ensuring data quality, and leveraging cloud technologies for scalable and efficient data processing. The Data Engineer will design, develop, and maintain scalable data pipelines and systems to support the ingestion, transformation, and analysis of large datasets. The role requires a deep understanding of data workflows, cloud platforms (Azure), and strong problem-solving skills to ensure efficient and reliable data delivery.

Key Responsibilities

  1. Data Ingestion and Integration:

  2. Develop and maintain data ingestion pipelines using tools like Azure Data Factory, Databricks, and Azure Event Hubs.

  3. Integrate data from various sources, including APIs, databases, file systems, and streaming data.

  4. ETL/ELT Development:

  5. Design and implement ETL/ELT workflows to transform and prepare data for analysis and storage in the data lake or data warehouse.

  6. Automate and optimize data processing workflows for performance and scalability.

  7. Data Modeling and Storage:

  8. Design data models for efficient storage and retrieval in Azure Data Lake Storage and Azure Synapse Analytics.

  9. Implement best practices for partitioning, indexing, and versioning in data lakes and warehouses.

  10. Quality Assurance:

  11. Implement data validation, monitoring, and reconciliation processes to ensure data accuracy and consistency.

  12. Troubleshoot and resolve issues in data pipelines to ensure seamless operation.

  13. Collaboration and Documentation:

  14. Work closely with data architects, analysts, and other stakeholders to understand requirements and translate them into technical solutions.

  15. Document processes, workflows, and system configurations for maintenance and onboarding purposes.

  16. Cloud Services and Infrastructure:

  17. Leverage Azure services like Azure Data Factory, Databricks, Azure Functions, and Logic Apps to create scalable and cost-effective solutions.

  18. Monitor and optimize Azure resources for performance and cost management.

  19. Security and Governance:

  20. Ensure data pipelines comply with organizational security and governance policies.

  21. Implement security protocols using Azure IAM, encryption, and Azure Key Vault.

  22. Continuous Improvement:

  23. Monitor existing pipelines and suggest improvements for better efficiency, reliability, and scalability.

  24. Stay updated on emerging technologies and recommend enhancements to the data platform.

Skills

  • Strong experience with Azure Data Factory, Databricks, and Azure Synapse Analytics.
  • Proficiency in Python, SQL, and Spark.
  • Hands-on experience with ETL/ELT processes and frameworks.
  • Knowledge of data modeling, data warehousing, and data lake architectures.
  • Familiarity with REST APIs, streaming data (Kafka, Event Hubs), and batch processing.

Good To Have:

  • Experience with tools like Azure Purview, Delta Lake, or similar governance frameworks.
  • Understanding of CI/CD pipelines and DevOps tools like Azure DevOps or Terraform.
  • Familiarity with data visualization tools like Power BI.

Competency

  • Analytical Thinking
  • Clear and effective communication
  • Time Management
  • Team Collaboration
  • Technical Proficiency
  • Supervising Others
  • Problem Solving
  • Risk Management
  • Organizing & Task Management
  • Creativity/innovation
  • Honesty/Integrity

Education:

  • Bachelors degree in Computer Science, Data Science, or a related field.
  • 8+ years of experience in a data engineering or similar role.

  • Senior Software Engineer/Technical Lead

    2 days ago

    Ahmedabad, Gujarat, India Crest Data Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Company Overview:Crest Data is a leading provider of data center solutions and engineering/marketing services in the areas of Networking/SDN, Storage, Security, Virtualization, Cloud Computing, and Big Data / Data Analytics. The team has extensive experience in building and deploying various Data Center products from Cisco, VMware, NetApp, Amazon AWS, EMC,...

  • Lead Data Engineer

    2 weeks ago

    Ahmedabad, Gujarat, India Circle K Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job DescriptionAlimentation Couche-Tard Inc., (ACT) is a global Fortune 200 company. A leader in the convenience store and fuel space with over 16,700 stores in 31 countries, serving more than 9 million customers each day. At Circle K, we are building a best-in-class global data engineering practice to support intelligent business decision-making and drive...

  • Lead Data Engineer

    2 weeks ago

    Ahmedabad, Gujarat, India Inferenz Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    About the CompanyWe are seeking a Lead Data Engineer with deep expertise in Snowflake, dbt, and Apache Airflow to design, implement, and optimize scalable data solutions. This role involves working on complex datasets, building robust data pipelines, ensuring data quality, and collaborating closely with analytics and business teams to deliver actionable...

  • Lead Data Engineer

    3 days ago

    Ahmedabad, Gujarat, India Tecblic Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Description : Data EngineerLocation : AhmedabadExperience : 6+ yearsEmployment Type : Full-Time We are looking for a highly motivated and experienced Data Engineer to join our team. As a Data Engineer, you will play a critical role in designing, building, and optimizing data pipelines that ensure the availability, reliability, and performance of our...

  • Lead Data Engineer

    2 weeks ago

    Ahmedabad, Gujarat, India Advantmed Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    We are seeking a talented and experienced Lead Data Engineer to join our team and contribute to building a robust data platform on Azure Cloud. The ideal candidate will have hands-on experience designing and managing data pipelines, ensuring data quality, and leveraging cloud technologies for scalable and efficient data processing. The Data Engineer will...

  • Senior Software Engineer

    2 days ago

    Ahmedabad, Gujarat, India Crest Data Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Company Overview:Crest Data is a leading provider of data center solutions and engineering/marketing services in the areas of Networking/SDN, Storage, Security, Virtualization, Cloud Computing, and Big Data / Data Analytics. The team has extensive experience in building and deploying various Data Center products fromCisco, VMware, NetApp, Amazon AWS, EMC,...

  • Senior Administration Executive

    2 weeks ago

    Ahmedabad, Gujarat, India Technobeat Engineer Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    We are seeking a dedicated Senior Admin Executive. The role is to manage and preparation of reports, ensure accuracy, and maintain monthly documentation. Responsible for collecting data from engineers, guiding team in report creation, and submission.

  • Lead Data Architect

    2 weeks ago

    Ahmedabad, Gujarat, India RK HR Management Private Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Key Responsibilities: Define and lead enterprise data strategy & governance Oversee modern data platforms (Snowflake, Databricks, AWS, Kafka) Enable AI/ML for personalization, forecasting & analytics Partner cross-functionally with Marketing, Supply Chain, Finance & R&D Lead and mentor a high-performing data engineering teamWhat We're Looking For: 18+ years...

  • Lead Data Scientist

    2 weeks ago

    Ahmedabad, Gujarat, India eInfochips (An Arrow Company) Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Role: Lead AI/ML EngineerLocation: Ahmedabad, Pune, IndoreYears of Experience: 8 to 12 yearsNotice Period – 0-45 Days MaxWe are looking for a highly skilled and experienced Lead AI/ML Engineer to lead the development and deployment of machine learning solutions across the organization. The ideal candidate will possess deep technical expertise in ML...

  • Lead Data Architect

    2 weeks ago

    Ahmedabad, Gujarat, India adani capital pvt ltd Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Lead – Data Architect ResponsibilitiesLead the design and implementation of data architecture and data modeling solutions for airport operations, ensuring scalability, performance, and reliability. Collaborate with cross-functional teams to gather and understand data requirements and translate them into effective database designs and data management...