Data Ingestion Expert

1 week ago

Kolkata, West Bengal, India beBeeIngestion Full time ₹ 1,00,00,000 - ₹ 1,50,00,000
Job Description

Azure Databricks Engineer is required to design, develop, and optimize data ingestion pipelines for integrating multiple sources into Databricks.

We seek an experienced engineer with a strong understanding of data ingestion processes, CI/CD, and Kubernetes. The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support scalable and efficient ingestion workflows.

The primary responsibility of this role includes:

  • Designing, developing, and optimizing data ingestion pipelines for integrating multiple sources into Databricks.
  • Implementing and maintaining CI/CD pipelines for data workflows.
  • Deploying and managing containerized applications using Kubernetes.
  • Ensuring data integrity, security, and compliance throughout the ingestion process.

This role requires a collaborative environment where engineers work closely with other stakeholders to streamline data ingestion strategies.

Required Skills & Qualifications

  • Proven experience in data ingestion and pipeline development.
  • Hands-on experience with CI/CD tools (GitHub Actions, Jenkins, Azure DevOps, etc.).
  • Strong knowledge of Kubernetes and container orchestration.
  • Experience with Databricks, Spark, and data lake architectures.
  • Proficiency in Python, Scala, or SQL for data processing.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP.

Preferred Qualifications

  • Experience with Infrastructure as Code (Terraform, Helm, etc.).
  • Background in streaming data ingestion (Kafka, Kinesis, etc.).

  • Expert Data Engineer

    18 hours ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 17,00,000

    Job Title: Expert Data EngineerWe are seeking a seasoned data engineer with expertise in Postgres SQL and hands-on experience in building scalable data warehouses and data lakes.Our ideal candidate will design, build, and maintain a high-performance data warehouse for analytics and reporting.Develop and manage batch and streaming data pipelines for...

  • Enterprise Data Cloud Expert

    7 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Opportunity: Salesforce Data Cloud Specialist About the RoleWe are seeking an experienced Salesforce Data Cloud expert to spearhead our data-first initiatives. This position requires in-depth knowledge of Salesforce Data Cloud (formerly Customer Data Platform), along with proven leadership skills to guide technical teams and drive scalable...

  • Data Expert

    3 days ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Key Data Engineer PositionWe are seeking a skilled data engineer to collaborate on client projects.The ideal candidate will have strong Python and SQL/database skills, experience with large datasets, and expertise in using the Pandas library.Key Responsibilities:Perform web scraping using various techniques and utilize Python's Pandas library for data...

  • Expert Data Architect for Scalable Infrastructure Solutions

    6 days ago

    Kolkata, West Bengal, India beBeeSnowflake Full time ₹ 25,00,000 - ₹ 40,00,000

    Senior Snowflake Developer and Infrastructure Integration ExpertWe're seeking a highly skilled Senior Snowflake Developer and Infrastructure Integration Expert to oversee full lifecycle implementations.The ideal candidate will have experience with secure platform setup, cloud integration, and high-performance data pipeline development.Lead the setup of the...

  • Unlock Business Potential as a Data Expert

    3 days ago

    Kolkata, West Bengal, India beBeeData Full time US$ 18,00,000 - US$ 20,00,000

    Drive Business Growth as a Data StrategistMercor is revolutionizing talent acquisition by leveraging cutting-edge models to predict job performance with unprecedented accuracy.Our innovative platform powers hiring at top AI labs and has achieved remarkable success, scaling from $1M to $100M in just 11 months.As a key member of our team, you will play a...

  • Senior Cloud Data Architect

    13 hours ago

    Kolkata, West Bengal, India beBeeDataEngineer Full time ₹ 1,75,00,000 - ₹ 2,25,00,000

    Cloud Data EngineerJob Description:We seek an experienced Cloud Data Engineer to design, develop and maintain scalable data pipelines and ETL/ELT solutions using cloud services.The ideal candidate will have expertise in S3, Redshift, Glue, Athena, EMR, Lake Formation, Lambda, Kinesis, Step Functions, CloudWatch, RDS and DynamoDB.They will also possess strong...

  • Advanced Data Analytics Expert

    6 days ago

    Kolkata, West Bengal, India beBeeData Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    Senior Associate - Data Analytics AdvisorWe are seeking a skilled Senior Associate to join our data analytics advisory team. As a key member of our team, you will be responsible for designing and developing advanced data solutions using Snowflake.The ideal candidate will have a strong background in SQL and Snowflake, with experience in designing and managing...

  • Senior Data Architect

    4 days ago

    Kolkata, West Bengal, India beBeeDatabase Full time ₹ 15,00,000 - ₹ 17,50,000

    Job OverviewNoSQL Database Administrator RoleResponsibilities:Database Optimization: Manage and optimize NoSQL databases, primarily ElasticSearch and Kafka.Infrastructure Design: Design and maintain Kafka brokers, zookeepers, and connectors ensuring high availability and scalability.Data Ingestion: Configure and manage ingestion tools like Logstash, Fluentd,...

  • AI/ML Expert for Advanced Cybersecurity Projects

    1 week ago

    Kolkata, West Bengal, India beBeeCybersecurity Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Expert AI/ML Lead for Cybersecurity DevelopmentWe are seeking a seasoned AI/ML lead to join our cybersecurity team based in Chennai.Lead the fine-tuning and domain adaptation of open-source Large Language Models (LLMs) using frameworks like vLLM, HuggingFace, DeepSpeed, and PEFT techniques.Develop data pipelines to ingest, clean, and structure cybersecurity...

  • L3 Support Engineer – Elasticsearch DB Expert

    3 weeks ago

    Kolkata, West Bengal, India Vehere Full time

    We are looking for an L3 Support Engineer with deep expertise in Elasticsearch and the ELK stack to provide operational support and drive the stability, performance, and scalability of large-scale cluster deployments. The ideal candidate will play a vital role in diagnosing complex system issues, optimizing performance, and ensuring the reliability of...