Data Pipeline Specialist

5 days ago

Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

Big Data Expertise Job Description

We seek a highly skilled Big Data Expert to create and maintain scalable data systems.

  • Job Summary:
  • • Design, develop, and implement large-scale data pipelines using Apache Spark and Python.
  • • Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
  • • Ensure data accuracy, quality, and security by implementing robust data governance practices.

Key Responsibilities:

  • Design and Develop Data Pipelines:
  • • Use Apache Spark to design and implement efficient data pipelines that meet business requirements.
  • • Implement data processing techniques such as data transformation, aggregation, and filtering.
  • Collaborate with Cross-Functional Teams:
  • • Work closely with data scientists, analysts, and product managers to understand business requirements and deliver high-quality solutions.
  • • Communicate technical concepts to non-technical stakeholders and provide guidance on data-driven decision-making.
  • Ensure Data Quality and Security:
  • • Implement robust data governance practices to ensure data accuracy, completeness, and integrity.
  • • Develop and enforce data security policies and procedures to protect sensitive information.

Requirements:

  • Experience:
  • • 5-8 years of experience in designing and implementing large-scale data systems.
  • • Strong hands-on experience with Apache Spark (RDDs, DataFrames, Spark SQL, PySpark).
  • Skill Set:
  • • Deep understanding of Spark cluster architecture and performance tuning.
  • • Proficiency in AWS ecosystem: Glue, S3, Lambda, Redshift, CloudWatch.
  • • Solid expertise in Snowflake: Snowpipe, Streams & Tasks, Time Travel, and data sharing.
  • • Strong programming skills in Python and SQL.
  • • Experience with CI/CD tools and infrastructure-as-code (Terraform or CloudFormation).

What We Offer:

  • Opportunities for Growth:
  • • Opportunities to work on cutting-edge projects and technologies.
  • • Professional growth and development opportunities.

  • Data Pipeline Quality Assurance Specialist

    3 days ago

    Amrāvati, Maharashtra, India beBeePipeline Full time ₹ 1,50,00,000 - ₹ 2,01,00,000

    Job Title:Data Pipeline Quality Assurance SpecialistJob Description:We are seeking an experienced Data Pipeline Quality Assurance Specialist to join our team. As a key member of our data operations team, you will play a vital role in ensuring the integrity and quality of our data pipelines.This is an excellent opportunity for a motivated and detail-oriented...

  • Data Pipeline Specialist

    5 days ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 25,00,000 - ₹ 30,00,000

    Unlock Data ExcellenceKey Responsibilities:Design and build scalable data pipelines handling structured and unstructured data using cloud services.Own end-to-end ETL/ELT processes ensuring reliability, scalability, and high availability.Collaborate closely with AI teams to support model training, retrieval-augmented generation, and analytics...

  • Data Pipeline Specialist

    5 days ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineering RoleWe are seeking a skilled professional to develop and optimize data pipelines, process large datasets, and work with relational databases. The ideal candidate will have experience in ETL tools, data processing technologies, data orchestration, and Python scripting.

  • Data Pipeline Specialist

    7 days ago

    Amrāvati, Maharashtra, India beBeeMachineLearning Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title:As a senior machine learning engineer, you will be responsible for developing and optimizing complex data pipelines using machine learning engineering principles.Key Responsibilities:Designing and implementing data pipelines using AWS services such as Glue and SageMaker.Collaborating with data science teams to review model-ready datasets and...

  • Mastering Azure Data Pipeline Architectures

    1 week ago

    Amrāvati, Maharashtra, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Azure Data Factory SpecialistWe seek an experienced professional to support data integration and ETL needs. The ideal candidate will have expertise in ADF pipelines, data orchestration, and transformations for smooth, scalable, and secure data flows.Design, build, and maintain Azure Data Factory pipelines for ETL and ELT processes.Integrate data from...

  • Senior Data Pipeline Specialist

    2 weeks ago

    Amrāvati, Maharashtra, India beBeeData Full time ₹ 10,00,000 - ₹ 20,00,000

    Job Title: Data EngineerRole Overview:We are seeking a skilled professional to design, develop, and maintain scalable data pipelines that process large volumes of structured and unstructured data.The ideal candidate will have a strong background in data engineering, cloud technologies, and data architecture, collaborating closely with cross-functional teams...

  • AI Data Pipeline Specialist

    1 week ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 18,00,000

    We're seeking an experienced Data Engineer to join our team.The role involves designing, building, and maintaining production-level data pipelines that handle terabytes of structured and unstructured data.Own end-to-end ETL/ELT processes, ensuring reliability, scalability, and high availability.Work with AWS services (S3, EC2, RDS, DynamoDB, Athena,...

  • High-Performance Data Pipeline Specialist

    18 hours ago

    Amrāvati, Maharashtra, India beBeeDataEngineering Full time US$ 1,25,000 - US$ 1,75,000

    Job Title: Senior Data EngineerWe are seeking a highly skilled Data Engineering Expert to design and implement high-performance data pipelines.The ideal candidate will have experience with big data technologies, such as Spark and Hadoop, and be proficient in Python programming. Strong database performance optimization skills are also required.In this role,...

  • Chief Data Pipeline Architect

    7 days ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Cloud Data Engineer Position OverviewWe are seeking an experienced Cloud Data Engineer to design, build and maintain scalable data pipelines in the cloud.Develop data pipelines using AWS services such as EMR, Glue, and PySpark to efficiently process large datasets.Collaborate with cross-functional teams to ensure smooth data flow and processing.Work closely...

  • Data Engineer

    2 weeks ago

    Amrāvati, Maharashtra, India beBeeDataEngineer Full time US$ 1,80,000 - US$ 2,50,000

    Job OverviewWe are seeking a seasoned data professional with expertise in designing and building scalable, event-driven data pipelines that support real-time clinical workflows.This critical role involves creating systems that process clinical encounter data, retrieve associated medical documents, and deliver validated information to machine learning systems...