Big Data Ingestion Specialist

4 days ago

Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

Azure Databricks is a leading platform for big data analytics and machine learning. We are seeking an experienced professional to join our team and work on building scalable and efficient data ingestion pipelines.

The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support the integration of various data sources into Azure Databricks.

Key Responsibilities:

  • Design, develop, and optimize data ingestion pipelines for integrating multiple sources into Databricks.
  • Implement and maintain CI/CD pipelines for data workflows.
  • Deploy and manage containerized applications using Kubernetes.
  • Ensure data integrity, security, and compliance throughout the ingestion process.
  • Collaborate with Data Engineers and other stakeholders to streamline data ingestion strategies.
  • Troubleshoot and optimize ingestion pipelines for performance and scalability.

Required Skills & Qualifications:

  • Proven experience in data ingestion and pipeline development.
  • Hands-on experience with CI/CD tools (GitHub Actions, Jenkins, Azure DevOps, etc.).
  • Strong knowledge of Kubernetes and container orchestration.
  • Experience with Databricks, Spark, and data lake architectures.
  • Proficiency in Python, Scala, or SQL for data processing.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP.

Preferred Qualifications:

  • Experience with Infrastructure as Code (Terraform, Helm, etc.).
  • Background in streaming data ingestion (Kafka, Kinesis, etc.).
  • Knowledge of data governance and security best practices.

Benefits:

  • Opportunity to work with cutting-edge technologies and contribute to innovative projects.
  • Collaborative and dynamic work environment.
  • Professional growth and development opportunities.

  • Big Data Solutions Expert

    15 hours ago

    Amrāvati, Maharashtra, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Unlock your expertise in data processing and analytics as a Big Data Engineer.Key Responsibilities:Develop scalable big data solutions using PySpark and Hive to drive business growth.Collaborate with cross-functional teams to integrate big data platforms with other systems, ensuring seamless data flow.Work closely with data scientists and analysts to build...

  • Big Data Architect

    2 days ago

    Amrāvati, Maharashtra, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    **Job Title:** Big Data ArchitectWe are seeking a highly skilled Big Data Architect to design, develop, and optimize large-scale data pipelines and distributed data processing systems.The ideal candidate will have a proven track record in building scalable cloud-based data architectures using Scala, Spark, Python, Apache Kafka, and NiFi.

  • Senior Big Data Specialist

    4 days ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title: Data EngineerSynopsis:As a seasoned data engineer, you will be responsible for designing and developing large-scale data processing systems using distributed computing frameworks like PySpark.Our ideal candidate is an expert in Big Data technologies, with hands-on experience in Python, AWS, and Spark.Mandatory Skills:PythonPySparkAWSBonus...

  • Chief Data Architect

    2 days ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Data Engineer RoleWe are seeking an experienced Data Engineer to design, build, and maintain robust data pipelines and ETL/ELT processes. This individual will develop data ingestion frameworks to collect data from various sources, implement data transformation and cleaning processes to ensure data quality and consistency, and optimize data pipeline...

  • Senior Financial Data Systems Specialist

    1 day ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Job TitleWe are seeking a highly skilled professional to join our growing team as a Big Data Engineer.Key Responsibilities:Design, develop, and manage end-to-end data pipelines for various financial datasets.Integrate third-party APIs and data feeds into internal systems.Build and optimize data ingestion, storage, and transformation workflows (batch and...

  • Data Engineer

    6 days ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 1,80,00,000

    Unlock Your Potential as a Data EngineerWe're seeking an experienced and innovative Data Engineer to join our team. As a key member of our data analytics platforms, you will be responsible for developing and deploying scalable data processing pipelines using Azure Data Factory, Azure Databricks, and other relevant technologies.Your primary focus will be on...

  • Big Data Platform Visionary

    2 days ago

    Amrāvati, Maharashtra, India beBeeDataLeader Full time US$ 1,25,000 - US$ 1,75,000

    Job OverviewWe are seeking a visionary leader to spearhead the development of a cutting-edge big data platform.Align with key client initiatives to drive strategic growth and collaboration

  • Cloud Data Governance Specialist

    20 hours ago

    Amrāvati, Maharashtra, India beBeeData Full time US$ 1,80,000 - US$ 2,50,000

    Job Title: Cloud Data Architect\We are seeking an experienced professional to lead our data architecture efforts on Google Cloud Platform (GCP). The ideal candidate will have a strong background in cloud data architecture, data warehousing, big data processing, and data integration.\Key Responsibilities:\\Design and architect end-to-end data solutions on...

  • Senior Big Data Project Coordinator

    20 hours ago

    Amrāvati, Maharashtra, India beBeeProject Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Technical Project Manager PositionWe seek an accomplished Technical Project Manager to spearhead large-scale projects within complex big data and analytics environments.

  • Expert Data Workflow Developer

    1 day ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Cloud Data SpecialistWe are looking for a highly skilled professional with expertise in designing, building, and optimizing large-scale data processing workflows.Job Description:As a Cloud Data Specialist, you will be responsible for designing, developing, and maintaining scalable ETL pipelines for batch data ingestion and egress using Azure Data Factory,...