Lead Azure Big Data Engineer

19 hours ago

Pune, Maharashtra, India VidPro Consultancy Services Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Senior Azure Data Engineer

Location: Hyderabad, Bangalore and Pune

Experience: 3-7 Years

Work Mode: Hybrid

Mandatory Skills: Python, Spark, SQL, Hadoop, Hive, Spark, Azure Databricks, Azure DataFactory, Azure Synapse, DLTs, Unity Catalog, Workflows, Infrastructure as Code (IaC), Orchestration-Airflow, ETL & Data Pipelines.

Role Overview:

We are looking for a Senior Data Engineer with a strong background in Python, Spark, Databricks, and Azure to join our growing team. The ideal candidate will have extensive experience in data processing, working with cloud platforms (specifically Azure), and be able to handle complex data engineering tasks. 

Key Responsibilities:

  • Design, build, and optimize scalable data pipelines for large-scale data processing using Python and Apache Spark.
  • Leverage Azure Databricks to perform big data processing, including working with PySpark to process and analyze large datasets.
  • Utilize Azure Data Factory for seamless data integration and orchestration across cloud environments.
  • Implement and maintain solutions using Azure Data Lake Storage for big data analytics and transformation.
  • Write efficient and reusable Python code, utilizing libraries such as Pandas and NumPy, to process and manipulate data.
  • Work with core Azure data services like Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, and Azure Blob Storage.
  • Should have worked with popular tools for Infrastructure as Code (IaC) like Terraform or Biceps in conjunction with CI/CD pipelines for the purpose of provisioning new Azure Infra resources.
  • Architectural experience is plus
  • Design and maintain complex SQL queries for managing and analyzing relational databases.
  • Ensure high-quality data management, integrity, and security across all stages of data processing & analysis.
  • Collaborate with cross-functional teams to understand data requirements and build solutions to meet business needs.
  • Provide mentorship and guidance to junior team members.

Key Skills & Qualifications:

  • Programming: Proficiency in Python, including knowledge of Pythonic principles, data structures, and algorithms. Hands-on experience with libraries such as Pandas and NumPy.
  • Cloud Technologies: In-depth knowledge of Azure, including Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, Azure Data Factory, Azure Blob Storage, and Azure Data Lake Storage.
  • Data Processing: Strong experience with Apache Spark using Azure Databricks and PySpark for large-scale data processing.
  • SQL: Strong skills in SQL for querying and managing relational databases. Ability to write complex and optimized SQL queries.
  • Communication: Excellent communication skills, with the ability to collaborate effectively across teams and present technical solutions to non-technical stakeholders.
  • Problem-Solving: Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data challenges

Education:  B.Tech in any branch or specialization.

  • Lead Data Engineer –Azure

    15 hours ago

    Pune, Maharashtra, India Admin Looks Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Urgently looking for highly experienced Lead Data Engineer with a proven track record in Azure data technologies and team leadership. The ideal candidate will have 12+ years of experience, including 7+ years of expertise in Azure Data Factory, Azure Databricks, Data Modeling, PySpark, SQL Queries, and Unity Catalog. This is a remote role requiring excellent...

  • Big Data Engineer

    7 days ago

    Pune, Maharashtra, India Funic Tech Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Job Opening: Big Data EngineerLocation: Aundh, Pune (Work From Office)Job Type: Full-Time | General Shift | Monday – FridayKey ResponsibilitiesBuild and manage highly scalable Big Data pipelines for large structured datasets.Collaborate with Product Management & Engineering leadership to design and implement optimal solutions.Participate in design...

  • Big Data Lead

    2 weeks ago

    Pune, Maharashtra, India Populace World Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Designation: Big Data Lead/ArchitectLocation: PuneExperience: 8-10 yearsNP - immediate joiner/15-30 days' noticeJob OverviewWe are looking to hire a talented big data engineer to develop and manage our company's Big Data solutions. In this role, you will be required to design and implement Big Data tools and frameworks, implement ELT processes, collaborate...

  • Big Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Vodafone Full time ₹ 1,00,00,000 - ₹ 2,00,00,000 per year

    Design, develop, and maintain scalable data pipelines for processing structured and unstructured data. Build and optimise data storage solutions including distributed file systems, NoSQL databases, and data warehouses. Utilise big data frameworks such as Apache Hadoop, Spark, Flink, and Kafka to develop data processing workflows. Ensure high performance and...

  • Azure Data Engineer

    16 hours ago

    Pune, Maharashtra, India BSR & Co Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    DescriptionAbout KPMG in IndiaKPMG entities in India are professional services firm(s). These Indian member firms are affiliated with KPMG International Limited. KPMG was established in India in August 1993. Our professionals leverage the global network of firms, and are conversant with local laws, regulations, markets and competition. KPMG has offices...

  • Senior Java Big Data Engineer

    2 weeks ago

    Pune, Maharashtra, India 6b7ed692-3bb5-4e80-abde-f1d5ee352806 Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    We are seeking a Senior Java Big Data Engineer to design, develop, and optimize large-scale, high-performance data processing and backend systems. The ideal candidate will have a strong foundation in java, Soring Boot, RESTful APIs, and big data streaming platforms (especially Kafka),with additional experience in Azure Cloud developments and CI /CD...

  • Azure Data Engineer

    14 hours ago

    Pune, Maharashtra, India BSR & Co Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionAbout KPMG in IndiaKPMG entities in India are professional services firm(s). These Indian member firms are affiliated with KPMG International Limited. KPMG was established in India in August 1993. Our professionals leverage the global network of firms, and are conversant with local laws, regulations, markets and competition. KPMG has offices...

  • Azure Data Lead Engineer

    2 weeks ago

    Pune, Maharashtra, India Persistent Systems Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About Position:We are looking for Lead Azure Data engineer who have experience in Azure data bricks, Data factory, to join our development team.Role: Azure Data Lead EngineerLocation: Pune/ BangaloreExperience: 8 to 10 yearsJob Type: Full Time EmploymentWhat You'll Do:Design and implement complex, scalable data pipelines for ingestion, processing, and...

  • Big Data

    7 days ago

    Pune, Maharashtra, India Atgeir Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Minimum Experience: 10 YearsWe are looking for a Big Data / Cloud Architect to become part of Atgeir's Advanced Data Analytics team. The desired candidate is a Professional with proven track record of working on Big Data and Cloud Platforms.Required Skills:Work closely with customers, understand customer requirements and render those as architectural...

  • Big Data Lead

    23 hours ago

    Pune, Maharashtra, India DemandMatrix Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Company Description At DemandMatrix, our vision is to disrupt the $100 billion sales and marketing intelligence industry by using domain knowledge, machine learning and AI. Fortune 100 companies like Microsoft, Google, Adobe, Amazon, IBM trust us to identify their next customer. Job Description What will you do?To help us go to the next level we are looking...