Lead Azure Big Data Engineer

2 days ago

Pune, Maharashtra, India VidPro Consultancy Services Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Senior Azure Data Engineer

Location: Hyderabad, Bangalore and Pune

Experience: 3-7 Years

Work Mode: Hybrid

Mandatory Skills: Python, Spark, SQL, Hadoop, Hive, Spark, Azure Databricks, Azure DataFactory, Azure Synapse, DLTs, Unity Catalog, Workflows, Infrastructure as Code (IaC), Orchestration-Airflow, ETL & Data Pipelines.

Role Overview:

We are looking for a Senior Data Engineer with a strong background in Python, Spark, Databricks, and Azure to join our growing team. The ideal candidate will have extensive experience in data processing, working with cloud platforms (specifically Azure), and be able to handle complex data engineering tasks. 

Key Responsibilities:

  • Design, build, and optimize scalable data pipelines for large-scale data processing using Python and Apache Spark.
  • Leverage Azure Databricks to perform big data processing, including working with PySpark to process and analyze large datasets.
  • Utilize Azure Data Factory for seamless data integration and orchestration across cloud environments.
  • Implement and maintain solutions using Azure Data Lake Storage for big data analytics and transformation.
  • Write efficient and reusable Python code, utilizing libraries such as Pandas and NumPy, to process and manipulate data.
  • Work with core Azure data services like Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, and Azure Blob Storage.
  • Should have worked with popular tools for Infrastructure as Code (IaC) like Terraform or Biceps in conjunction with CI/CD pipelines for the purpose of provisioning new Azure Infra resources.
  • Architectural experience is plus
  • Design and maintain complex SQL queries for managing and analyzing relational databases.
  • Ensure high-quality data management, integrity, and security across all stages of data processing & analysis.
  • Collaborate with cross-functional teams to understand data requirements and build solutions to meet business needs.
  • Provide mentorship and guidance to junior team members.

Key Skills & Qualifications:

  • Programming: Proficiency in Python, including knowledge of Pythonic principles, data structures, and algorithms. Hands-on experience with libraries such as Pandas and NumPy.
  • Cloud Technologies: In-depth knowledge of Azure, including Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, Azure Data Factory, Azure Blob Storage, and Azure Data Lake Storage.
  • Data Processing: Strong experience with Apache Spark using Azure Databricks and PySpark for large-scale data processing.
  • SQL: Strong skills in SQL for querying and managing relational databases. Ability to write complex and optimized SQL queries.
  • Communication: Excellent communication skills, with the ability to collaborate effectively across teams and present technical solutions to non-technical stakeholders.
  • Problem-Solving: Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data challenges

Education:  B.Tech in any branch or specialization.

  • Lead Data Engineer –Azure

    2 days ago

    Pune, Maharashtra, India Admin Looks Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Urgently looking for highly experienced Lead Data Engineer with a proven track record in Azure data technologies and team leadership. The ideal candidate will have 12+ years of experience, including 7+ years of expertise in Azure Data Factory, Azure Databricks, Data Modeling, PySpark, SQL Queries, and Unity Catalog. This is a remote role requiring excellent...

  • Big Data Engineer

    1 week ago

    Pune, Maharashtra, India FUNIC TECH PRIVATE LIMITED Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Description :Job Opening : Big Data Engineer (7-10 yrs)Location : Aundh, Pune (Work From Office)Job Type : Full-Time | General Shift | Monday FridayKey Responsibilities : - Build and manage highly scalable Big Data pipelines for large structured datasets. - Collaborate with Product Management & Engineering leadership to design and implement optimal...

  • Big Data Engineer

    1 week ago

    Pune, Maharashtra, India Funic Tech Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Job Opening: Big Data EngineerLocation: Aundh, Pune (Work From Office)Job Type: Full-Time | General Shift | Monday – FridayKey ResponsibilitiesBuild and manage highly scalable Big Data pipelines for large structured datasets.Collaborate with Product Management & Engineering leadership to design and implement optimal solutions.Participate in design...

  • Big Data Engineer

    2 days ago

    Pune, Maharashtra, India Information Technology Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Big Data EngineerKey Responsibilities : - Build and manage highly scalable Big Data pipelines for large structured datasets. - Collaborate with Product Management & Engineering leadership to design and implement optimal solutions. - Participate in design discussions and contribute to selecting, integrating, and maintaining Big Data frameworks. -...

  • Big Data Lead

    2 days ago

    Pune, Maharashtra, India Populace World Solutions Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Designation: Big Data LeadLocation: PuneExperience: 8-10 yearsReports To – Product Engineering HeadJob OverviewWe are looking to hire a talented big data engineer to develop and manage our company's Big Data solutions. In this role, you will be required to design and implement Big Data tools and frameworks, implement ELT processes, collaborate with...

  • Big Data Engineer

    2 days ago

    Pune, Maharashtra, India Information Technology Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key ResponsibilitiesBuild and manage highly scalable Big Data pipelines for large structured datasets.Collaborate with Product Management & Engineering leadership to design and implement optimal solutions.Participate in design discussions and contribute to selecting, integrating, and maintaining Big Data frameworks.Develop distributed data processing systems...

  • Big Data Lead

    2 weeks ago

    Pune, Maharashtra, India Populace World Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Designation: Big Data Lead/ArchitectLocation: PuneExperience: 8-10 yearsNP - immediate joiner/15-30 days' noticeJob OverviewWe are looking to hire a talented big data engineer to develop and manage our company's Big Data solutions. In this role, you will be required to design and implement Big Data tools and frameworks, implement ELT processes, collaborate...

  • Lead Big Data Engineer

    7 days ago

    Pune, Maharashtra, India Devlats Pvt Ltd Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    We are seeking a talented Lead Big Data Engineer to deliver roadmap features which would help customers to Measure, Communicate and Eliminate Cyber Risks. Working with a team of engineers and architects, you will be responsible for prototyping, designing, developing and supporting a highly scalable, distributed SaaS based Security Risk Prioritization...

  • Azure Data Engineer

    3 days ago

    Pune, Maharashtra, India TEKIT SOFTWARE SOLUTIONS PRIVATE LIMITED Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Job Title : Azure Data EngineerLocation : Pune, India (On-site only)Experience Required : 7-8 YearsJob Description : We are looking for an experienced Azure Data Engineer with strong proficiency in Azure Data Ingestion Services and Data Engineering to join our dynamic team. As a key member of our data engineering team, you will play a crucial role in...

  • Azure Data Engineer

    2 days ago

    Pune, Maharashtra, India BSR & Co Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    DescriptionAbout KPMG in IndiaKPMG entities in India are professional services firm(s). These Indian member firms are affiliated with KPMG International Limited. KPMG was established in India in August 1993. Our professionals leverage the global network of firms, and are conversant with local laws, regulations, markets and competition. KPMG has offices...