Senior Data Pipeline Developer

3 days ago

Dombivli, Maharashtra, India beBeeEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

Job Title

We are seeking a highly skilled and experienced Platform Engineer to join our team. The ideal candidate will have a strong background in Python, with expertise in performance and concurrency, as well as experience with API engineering and large-scale data pipelines.

  • Design and Build Shared Component Library/SDK: Design, build, and maintain a shared component library/SDK for pipelines, including ingestion, parsing, extraction, validation, enrichment, and publishing.
  • Create Pluggable Interfaces: Create pluggable interfaces so multiple teams can swap extractors, OCR providers, and EMR publishers without code rewrites.

Responsibilities

  1. Define Patterns/Templates: Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  2. Implement Concurrency Best Practices: Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
  3. Establish SLOs/Alerts: Establish SLOs/alerts for throughput, latency, error rates; set up DLQs and recovery patterns.
  4. Mentor Developers: Mentor developers, lead design reviews, codify best practices, write clear docs and examples.
  5. Partner with ML Engineers: Partner with ML engineers on the future LLM/SLM path (evaluation harness, safety/PII, cost/perf).

Expertise You'll Bring:

  • 7+ Years of Python Experience: 7+ years of Python experience with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing), profiling and memory tuning.
  • Observability Expertise: Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
  • LLM-based Solutions: Must have implemented LLM-based solutions and supported them in production.
  • API Engineering: API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
  • Strong Architecture/Design Skills: Strong architecture/design skills: clean interfaces, packaging shared libs, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
  • Large-Scale Data Pipelines: 3+ years building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
  • Document Processing: Document processing: OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, text normalization.
  • LLM/SLM Integration: LLM/SLM integration experience (e.g., OpenAI/Azure AI, local SLMs), prompt/eval frameworks, PII redaction/guardrails.
  • Cloud and Tooling: Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
  • Security/Compliance Mindset: Security/compliance mindset (HIPAA), secrets management, least-privilege access.

  • Data Pipeline Architect

    2 days ago

    Dombivli, Maharashtra, India beBeePython Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Python EngineerWe are seeking a seasoned Senior Python Engineer to lead our engineering group. As a key member of the team, you will be responsible for designing and developing scalable data pipelines using Python and Pandas.You will work closely with Data Scientists, Analysts, and Engineering teams to support data-driven applications and reporting...

  • Data Pipeline Architect

    2 days ago

    Dombivli, Maharashtra, India beBeeAzure Full time ₹ 12,00,000 - ₹ 20,10,000

    Job DescriptionThreatXIntel, a cybersecurity company dedicated to protecting businesses from cyber threats, is seeking an experienced Freelance Azure Data Factory (ADF) Specialist to support our data integration and ETL needs.The ideal consultant will have strong hands-on expertise in Azure ADF pipelines, data orchestration, and transformations to ensure...

  • Senior Data Pipeline Specialist

    4 days ago

    Dombivli, Maharashtra, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineering RoleJob OverviewThis is a key role within our organization, focusing on designing and developing data pipelines to ingest data from various sources into our data warehouse. The ideal candidate will have experience with AWS services such as Lambda, DMS, and Glue for data ingestion and transformation processes.Key ResponsibilitiesDesign and...

  • Expert Data Pipeline Specialist

    3 days ago

    Dombivli, Maharashtra, India beBeeData Full time ₹ 9,00,000 - ₹ 15,00,000

    Job OverviewWe are seeking a skilled Data Engineer to design, develop and maintain large-scale data pipelines that can handle structured and unstructured data.This role involves key responsibilities such as implementing data pipelines, building ETL/ELT workflows and ensuring data quality across all pipelines.

  • Data Pipeline Specialist

    5 days ago

    Dombivli, Maharashtra, India beBeeTalend Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Talend ETL DeveloperWe are seeking a qualified professional to take on the role of a Talend ETL Developer with expertise in Talend Open Studio for an immediate contractual position.Below are the key responsibilities and requirements for this job:Key Responsibilities:- Design and optimize complex ETL pipelines using Talend Data Integration- Debug...

  • Highly Skilled Data Pipeline Architect

    3 days ago

    Dombivli, Maharashtra, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    About Foodsmart:We are a leading provider of personalized nutrition and foodcare solutions.Our platform is designed to foster healthier eating habits, drive lasting behavior change, and deliver long-term health outcomes.We serve 2.2 million members on a tailored journey to eating well while saving time and money.We partner with national and regional...

  • Data Developer Position

    3 days ago

    Dombivli, Maharashtra, India beBeeData Full time ₹ 18,00,000 - ₹ 24,00,000

    Senior Data DeveloperWe are seeking a skilled Senior Data Developer to join our team.Job Description:The successful candidate will be responsible for designing, developing and maintaining large-scale ETL processes using Databricks PySpark. This includes extracting, transforming and loading data from various sources into data lakes and warehouses.Key...

  • Data Pipeline Specialist – Cloud Engineering

    2 days ago

    Dombivli, Maharashtra, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,50,000

    Freelance Data Engineer OpportunityJoin a collaborative environment where you can utilize your expertise in designing and optimizing large-scale data pipelines.Role OverviewAs an experienced Freelance Data Engineer, you will work closely with internal teams to design, build, and maintain ETL/ELT pipelines using PySpark, AWS, and Databricks. The ideal...

  • Senior Scalable Data Developer

    3 days ago

    Dombivli, Maharashtra, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Scalable Data Pipelines Developer OpportunityAt a leading organization, we seek an experienced Scalable Data Pipelines Developer to join our dynamic team. This is an exciting chance to work on designing and implementing efficient data pipelines using Spark SQL, DataFrame, and RDD APIs.Key Responsibilities:Design and develop batch and streaming data pipelines...

  • Senior Data Engineer

    5 days ago

    Dombivli, Maharashtra, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Overview">As a senior data engineering professional, you will be responsible for designing and developing scalable, secure, and high-performance data solutions on Databricks.">Key Responsibilities">Design, develop, and maintain large-scale data pipelines and ETL processes to support business growth and decision-making.Lead the architecture and...