Senior Big Data Engineer

6 days ago

Malappuram, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Azure Databricks Engineer Job Description

Design and develop data ingestion pipelines to integrate multiple sources into Databricks.

The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support scalable and efficient ingestion workflows.

Key Responsibilities:

  • Data Ingestion Design and Development: Design, develop, and optimize data ingestion pipelines for integrating various data sources into Databricks.
  • CI/CD Pipeline Implementation: Implement and maintain CI/CD pipelines for data workflows.
  • Kubernetes Management: Deploy and manage containerized applications using Kubernetes.
  • Data Integrity and Security: Ensure data integrity, security, and compliance throughout the ingestion process.
  • Collaboration and Troubleshooting: Collaborate with Data Engineers and other stakeholders to streamline data ingestion strategies and troubleshoot and optimize ingestion pipelines for performance and scalability.

Required Skills & Qualifications:

  • Proven experience in data ingestion and pipeline development.
  • Hands-on experience with CI/CD tools (e.g., GitHub Actions, Jenkins, Azure DevOps).
  • Strong knowledge of Kubernetes and container orchestration.
  • Experience with Databricks, Spark, and data lake architectures.
  • Proficiency in Python, Scala, or SQL for data processing.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP.
  • Strong problem-solving and analytical skills.

Preferred Qualifications:

  • Experience with Infrastructure as Code (Terraform, Helm, etc.).
  • Background in streaming data ingestion (Kafka, Kinesis, etc.).
  • Knowledge of data governance and security best practices.

  • Big Data Engineer

    3 days ago

    Malappuram, Kerala, India beBeeDataScientist Full time ₹ 20,00,000 - ₹ 30,00,000

    About Big Data Development RoleKey responsibilities for this position include performing hands-on big data development work, collaborating with junior engineers, reviewing code, and assisting in designing solutions using ETL processes.This involves working with Hadoop, Hive, HDFS, Apache Spark, and Kafka to integrate data from various sources. Strong...

  • Big Data Engineering Leader

    3 days ago

    Malappuram, Kerala, India beBeeDataEngineer Full time ₹ 24,00,000 - ₹ 36,00,000

    Job OverviewWe are seeking a skilled Big Data Engineering leader to join our team.The successful candidate will be responsible for designing, implementing, and maintaining large-scale big data systems. This includes leading the data platform and pipeline along with risk evaluation, providing technical leadership on data platform design, rollout, and...

  • Big Data Solutions Expert

    19 hours ago

    Malappuram, Kerala, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job OverviewWe are seeking a skilled Senior Big Data Engineer to join our team.The ideal candidate will have experience in designing and implementing big data analytics platforms using Spark, Kafka, Glue, EMR, Hudi, Schema Registry, and Data Lineage.In addition to technical expertise, the successful candidate will possess excellent problem-solving skills,...

  • Big Data Specialist

    4 days ago

    Malappuram, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title:Big Data SpecialistJob Description:We are seeking a highly skilled Big Data Specialist to lead the design, development, and deployment of large-scale data systems. The ideal candidate will have expertise in Python and cloud technologies, preferably Azure.Responsibilities:Design, develop, and deploy big data systems to drive business growth and...

  • Big Data Architect with Fabric

    7 days ago

    Malappuram, Kerala, India beBeeData Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title:Big Data Architect with FabricThe ideal candidate for this role will be a highly motivated and skilled data engineer who is passionate about designing, building, and maintaining large-scale data systems. This position requires expertise in big data technologies such as Hadoop, Spark, and NoSQL databases.

  • Senior Big Data Engineer

    2 weeks ago

    Malappuram, Kerala, India beBeeAnalyst Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Big Data Analyst We are seeking a highly skilled Data Analyst with expertise in SQL, Python, and Java, along with exposure to Generative AI tools and techniques.Key Responsibilities:Design, develop, and optimize SQL queries for data extraction, manipulation, and reporting.Build and maintain ETL pipelines for seamless data integration across...

  • Chief Big Data Architect

    3 days ago

    Malappuram, Kerala, India beBeeBigDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Overview:A high-performing Big Data Engineering Leader oversees the development of scalable data solutions, collaborating closely with cross-functional teams to drive business outcomes.Key Responsibilities:Design and implement complex data pipelines using batch processing and real-time streaming technologies.Collaborate with stakeholders to define data...

  • Big Data Specialist

    1 day ago

    Malappuram, Kerala, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Reimagining Data Infrastructure for Enhanced PerformanceWe are seeking a seasoned Data Engineer to join our team and contribute to the development of high-performance data pipelines that process massive amounts of data. The ideal candidate will have experience in designing, building, and maintaining scalable batch processing systems using Python and big data...

  • Big Data Solutions Architect

    1 week ago

    Malappuram, Kerala, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 21,00,000

    Job OpportunityWe are seeking a highly skilled data engineering professional to join our team.The ideal candidate will have a strong background in big data technologies and experience with PySpark, Hive, Spark, HBase, DQ tool, and Agile Scrum methodologies.Key responsibilities include designing, developing, and maintaining large-scale data processing...

  • Big Data Solutions Architect

    7 days ago

    Malappuram, Kerala, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Opportunity:We are seeking an experienced Big Data Developer to drive data insights and inform business decisions.The ideal candidate will have a strong background in big data technologies, including Scala, AWS, and Python or Java.Key Responsibilities:Design and develop scalable big data architectures using Hadoop and Spark.Analyze large datasets to...