Advanced Data Pipelining Specialist

2 weeks ago

Varanasi, Uttar Pradesh, India beBeeDeveloper Full time ₹ 14,00,000 - ₹ 17,00,000
Senior Data Developer

Are you looking for a challenging opportunity to work with complex data pipelines and scalable architectures?

This role is perfect for an experienced ETL developer who can design, develop, and maintain efficient, high-quality data integration solutions.

  • Design and implement robust, scalable ETL processes using industry-leading tools and technologies.
  • Develop reusable components that support modular, configuration-driven architecture.
  • Collaborate with cross-functional teams to understand data requirements and provide technical expertise.

As a senior data developer, you will play a key role in shaping the company's data strategy and ensuring seamless data integration across systems.

Key Responsibilities1. ETL Design and Development:

Develop efficient, scalable SSIS packages to extract, transform, and load data between systems.

Implement business rules and data transformations using data flow and control flow logic.

Optimize ETL processes for performance, scalability, and reliability.

2. Data Integration and Transformation:

Integrate data from multiple heterogeneous sources, including relational databases, flat files, APIs, and cloud-based services.

Implement data quality checks, data validation, and data cleansing techniques.

Manage incremental loads, full loads, and slowly changing dimensions (SCD) as required.

3. SQL and Database Development:

Write complex T-SQL queries, stored procedures, and functions to support data transformations and staging logic.

Perform joins, unions, aggregations, filtering, and windowing operations effectively for data preparation.

Ensure referential integrity and proper indexing for performance.

4. Performance Tuning:

Optimize SSIS packages by tuning buffer sizes, using parallelism, and minimizing unnecessary transformations.

Tune SQL queries and monitor execution plans for efficient data movement and transformation.

Implement efficient data loads for high-volume environments.

5. Error Handling and Logging:

Develop error-handling mechanisms and event logging in SSIS using Event Handlers and custom logging frameworks.

Implement restartability, checkpoints, and failure notifications in workflows.

6. Testing and Quality Assurance:

Conduct unit and integration testing of ETL pipelines.

Validate data outputs against source and business rules.

Support QA teams in user acceptance testing (UAT) and defect resolution.

7. Deployment and Scheduling:

Package, deploy, and version SSIS solutions across development, test, and production environments.

Schedule ETL jobs using SQL Server Agent or enterprise job schedulers (e.g., Control-M, Tidal).

Monitor and troubleshoot job failures and performance issues.

8. Documentation and Maintenance:

Maintain documentation for ETL designs, data flow diagrams, transformation logic, and job schedules.

Update job dependencies and maintain audit trails for data pipelines.

9. Collaboration and Communication:

Collaborate with data architects, business analysts, and reporting teams to understand data needs.

Provide technical support and feedback during requirements analysis and post-deployment support.

Participate in sprint planning, status reporting, and technical reviews.

10. Compliance and Best Practices:

Ensure ETL processes comply with data governance, security, and privacy regulations.

Follow team coding standards, naming conventions, and deployment protocols.

  • Senior Data Pipeline Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 90,00,000 - ₹ 1,80,00,000

    Data Engineering ProfessionalWe are seeking a highly skilled Data Engineer to join our team. The ideal candidate will have expertise in building, deploying, and managing data pipelines using Python and PySpark.

  • Data Pipeline Architect

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title:Data Pipeline ArchitectCreate and implement scalable, event-driven data pipelines that support real-time clinical workflows.Design and integrate systems with EHR/EMR platforms (e.g., Epic) using FHIR and HL7 standards.Develop high-quality systems to ingest, track, and update clinical events in real-timeEnsure downstream analytics systems receive...

  • Senior Data Pipeline Specialist

    1 week ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Unlock the full potential of your data career with this exciting opportunity.Job DescriptionWe are seeking a skilled Data Engineer to join our organization. As a key member of our team, you will be responsible for designing, developing, and maintaining large-scale data pipelines and systems.Key Responsibilities:Designing and implementing scalable data...

  • Advanced Analytics Specialist

    1 week ago

    Varanasi, Uttar Pradesh, India beBeeMachineLearning Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job Title: Advanced Analytics SpecialistWe are seeking a talented and motivated Advanced Analytics Specialist to join our team. In this role, you will be responsible for developing and implementing advanced analytics solutions that drive business growth and improve operational efficiency.Key Responsibilities:Design and develop complex analytical models and...

  • Transforming Business Through Advanced Data Solutions

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 16,00,000

    Big Data SpecialistWe are seeking a seasoned Big Data Specialist to collaborate with our team. As a key member of the group, you will be responsible for designing, building, and maintaining large-scale data pipelines that process structured and unstructured data.Key Responsibilities:Develop end-to-end ETL/ELT processes to ensure high availability and...

  • Senior Cloud Data Pipeline Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Cloud Data EngineerWe are seeking a skilled cloud data engineer to design, build and maintain scalable data pipelines and ETL solutions. The ideal candidate will have hands-on experience with AWS services, PySpark, Airflow and Python.Key Responsibilities:Design and develop scalable data pipelines using AWS services.Collaborate closely with data analysts,...

  • Advanced Data Strategist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 30,00,000

    Job Title: Advanced Data Strategist">">Develop and implement sophisticated data-driven solutions to drive business growth.">Collaborate with cross-functional teams to turn data into actionable insights.">Design and deploy advanced AI/ML models to support strategic objectives.">">Key Responsibilities:">">Develop and deploy machine learning models using...

  • Expert Data Pipeline Architect

    1 week ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineering ExpertiseWe are seeking a highly skilled Data Engineer to join our team. As a key member of our data engineering group, you will play a vital role in designing and building scalable data pipelines and architectures.This position offers the chance to work closely with talented teams, solve real-world data challenges, and make a direct impact...

  • Data Pipelines Engineer

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeSoftware Full time ₹ 90,00,000 - ₹ 1,50,00,000

    Job Description">Overview:We are seeking a skilled Python Developer to join our team. This full-time position will involve building automated data pipelines, implementing analytical algorithms, and developing REST APIs.Key Responsibilities:Design and implement automated data pipelines using Google Cloud Platform.Develop analytical algorithms to drive...

  • Cloud Data Specialist

    1 week ago

    Varanasi, Uttar Pradesh, India beBeeCloudData Full time ₹ 12,00,000 - ₹ 17,50,000

    Cloud Data SpecialistWe are seeking a highly skilled Cloud Data Specialist to join our team. The ideal candidate will have expertise in cloud-based analytics, big data processing, and advanced visualization technologies.The successful candidate will design, develop, and maintain scalable ETL/ELT pipelines using cloud services, Python programming, and...