Data Pipeline Engineer

3 days ago

Rajkot, Gujarat, India beBeeElasticsearch Full time ₹ 15,00,000 - ₹ 25,00,000
About this Role

This position involves continually evaluating the performance of our Elasticsearch clusters to identify potential issues, planning for upcoming high-load events, implementing security patches, testing and upgrading software incrementally, and enhancing our monitoring and alert infrastructure.

  • You will be responsible for maintaining other parts of the data pipeline, which may include serverless or server-based systems feeding data into the Elasticsearch pipeline.
  • We are constantly optimizing our cost-performance balance, so testing new hosts or configurations is an ongoing focus.
  • Our work often employs declarative tools like Puppet, as well as various scripting mechanisms depending on the target environment. We aim to automate tasks wherever possible and strive for a 'build once/run everywhere' system.
Key Expertise You Will Bring
  • Experience with managing Elasticsearch clusters in various environments, including public cloud, Kubernetes, and private datacenters.
  • An opportunity to collaborate with diverse types of infrastructure and operations teams.
  • Built long-lasting partnerships across the organization to foster collaboration between Product, Engineering, and Operations teams.
  • Participated in on-call rotations and demonstrated ability to address escalated issues promptly.
  • Required Qualifications: Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
  • Primary Expertise: Administration and Management of Elasticsearch Clusters.
  • Secondary Expertise: Administration and Management of Splunk Clusters.
  • Strong Knowledge in provisioning and Configuration Management tools like Puppet, Ansible, Rundeck, etc.
  • Experience in building Automations and Infrastructure as Code using Terraform, Packer, or CloudFormation templates.
  • Experience with monitoring and logging tools like Splunk, Prometheus, PagerDuty, etc.
  • Experience with scripting languages like Python, Bash, Go, Ruby, Perl, etc.
  • Experience with CI/CD tools like Jenkins, Pipelines, Artifactory, etc.
  • An inquisitive mind able to learn about data distribution within complex systems and interpret that data effectively.
  • The skills to perform effective troubleshooting, following problems to their resolution.
  • Experience with Docker and Kubernetes.

  • Senior Data Pipeline Engineer

    3 days ago

    Rajkot, Gujarat, India beBeeBackendDeveloper Full time ₹ 20,00,000 - ₹ 30,00,000

    Job DescriptionWe are seeking a skilled Backend Developer to join our team and contribute to the development of core infrastructure that powers real-time pricing and analytics in DeFi. This role is ideal for someone who enjoys low-level protocol work, understands DeFi mechanics, and can build robust data pipelines that power real-time pricing engines.

  • Data Pipeline Architect

    7 days ago

    Rajkot, Gujarat, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Unlock Data Engineering PotentialThis role presents an exciting opportunity to leverage technical expertise in designing, developing, and maintaining robust data pipelines, warehouses, and analytics platforms. The successful candidate will collaborate with cross-functional teams to ensure seamless data flow from various marketing channels, ad platforms, and...

  • Data Pipeline Architect

    23 hours ago

    Rajkot, Gujarat, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    About this role:We are seeking an experienced Data Engineer to join our team.This is a great opportunity for someone who wants to design, build and maintain robust data pipelines using Azure cloud services.The ideal candidate will have strong SQL skills, experience with Fabric, and hands-on experience with Azure Data Services.Responsibilities include...

  • Mastering Data Ingestion Pipelines

    1 week ago

    Rajkot, Gujarat, India beBeeEngineering Full time ₹ 80,00,000 - ₹ 1,60,00,000

    Job Title:Big Data EngineerWe are seeking an experienced Big Data Engineer to join our team. The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support scalable and efficient data ingestion workflows.Key Responsibilities:Design, develop, and optimize data ingestion pipelines for integrating...

  • Cloud Native Data Pipeline Engineer

    10 hours ago

    Rajkot, Gujarat, India beBeedataengineering Full time ₹ 12,00,000 - ₹ 25,00,000

    Job Title:Data Engineer - Cloud ArchitectOverview:We are seeking skilled Data Engineers to design and build robust data pipelines that power cloud-native solutions. The role focuses on developing efficient ETL/ELT workflows, ensuring data quality, security, and scalability to support AI/ML model training, inference, and intelligent decision-making...

  • Chief Data Pipeline Architect

    13 hours ago

    Rajkot, Gujarat, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineering Role OverviewA Data Engineer plays a pivotal role in our analytics division, responsible for designing and developing robust data pipelines and ETL processes using data platforms for the organization's centralized data warehouse.

  • Chief Data Pipeline Architect

    3 days ago

    Rajkot, Gujarat, India beBeeData Full time US$ 80,000 - US$ 1,50,000

    Job Title: Azure Data EngineerWe are seeking a skilled data engineer to design and implement scalable data pipelines using Azure Databricks.The ideal candidate will have strong experience with Azure Databricks and Apache Spark, as well as proficiency in developing Azure Function Apps using C#, Python, or JavaScript.Key Responsibilities:Design and implement...

  • Build Data Pipelines

    3 days ago

    Rajkot, Gujarat, India beBeeDataEngineering Full time ₹ 80,00,000 - ₹ 1,00,00,000

    Job Title: Data ArchitectWe are seeking a highly skilled data architect to join our team.Design, build, and maintain robust data pipelines for processing channel activity data, focusing on digital channels and integrating multiple sources of data.Work on AWS cloud infrastructure (including services like Pyspark, Glue, EMR, Lambda, Redshift, Athena, and other...

  • Financial Data Pipelines Developer

    7 days ago

    Rajkot, Gujarat, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are seeking a skilled Data Engineer to design and build robust data pipelines that process high-volume financial data.The ideal candidate will have hands-on experience in designing, developing, and managing end-to-end data pipelines for stocks, cryptocurrencies, and other financial datasets.Expertise in Python, SQL, and distributed data systems is...

  • AWS Data Pipeline Specialist

    10 hours ago

    Rajkot, Gujarat, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Dear Candidate, we are seeking a highly skilled professional to join our team as an AWS Data Engineer.The ideal candidate will have deep expertise in building scalable data pipelines using Apache Spark, PySpark, SQL, and Python, along with hands-on experience in the AWS ecosystem.Key Responsibilities:Design, develop, and maintain robust ETL/ELT pipelines on...