Senior Data Pipeline Architect

4 days ago

Aligarh, Uttar Pradesh, India beBeeEngineer Full time ₹ 1,60,00,000 - ₹ 2,01,00,000
Lead Python Platform Engineer

You will be responsible for designing and building scalable data pipelines using Apache Beam.

  • Design and build a shared component library/SDK for ingestion, parsing/OCR, extraction, validation, enrichment, and publishing.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
  • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
  • Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
Required Skills and Qualifications:
  • 7+ years of experience with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing).
  • Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
  • Must have implemented Large Language Model based solutions and supported them in production.
  • API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
  • Strong architecture/design skills: clean interfaces, packaging shared libraries, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
  • 3+ years of experience building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
Benefits:
  • Competitive salary and benefits package.
  • Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications.
  • Opportunity to work with cutting-edge technologies.
  • Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards.
  • Annual health check-ups.
  • Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents.
Additional Information:

We are committed to providing a positive and inclusive work environment.

  • Senior Data Solutions Architect

    2 days ago

    Aligarh, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Solutions ArchitectWe are seeking a seasoned data expert to design and implement scalable data platforms using modern technologies. As a key member of our team, you will be responsible for architecting and managing data lakes and warehouses, ensuring seamless data flow across the organization.Data Engineering & Architecture:Design, build, and...

  • Senior Data Pipeline Architect

    5 days ago

    Aligarh, Uttar Pradesh, India beBeeDataIntegrator Full time ₹ 60,00,000 - ₹ 1,00,00,000

    Key Data Integration RoleJob Summary:We are seeking a seasoned data integration professional to join our team. As an O9 Integration Consultant, you will be responsible for designing and building robust data pipelines to support business operations.The ideal candidate will have extensive experience in ETL processes, proficiency in databases, and strong...

  • Senior Data Architect

    1 minute ago

    Aligarh, Uttar Pradesh, India beBeeData Full time US$ 1,50,000 - US$ 1,75,000

    We are seeking a highly skilled Senior Data Engineer to join our team as a key contributor in designing, constructing, and optimizing large-scale data processing systems that enable advanced analytics and business intelligence.Key ResponsibilitiesDesign scalable and robust data pipelines that collect, process, and store large volumes of structured and...

  • Senior Data Pipeline Specialist

    4 days ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    Data Engineer PositionAs a senior data engineer, you will play a critical role in designing and implementing data pipelines that drive business growth. Your expertise in Python and AWS will enable you to develop scalable and efficient solutions for large datasets.The ideal candidate will have strong knowledge of ETL tools such as Glue and Talend, as well as...

  • Chief Data Pipeline Architect

    2 days ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 90,00,000 - ₹ 1,20,00,000

    Transform Data into InsightsWe're seeking a skilled Data Engineer to join our team. The ideal candidate has 1–2 years of experience in data engineering and is excited to solve tough data problems in a fast-paced environment.The RoleDesign, build, and maintain production-level data pipelines that handle terabytes of structured and unstructured data.Own...

  • Senior Data Architecture Specialist

    5 days ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Technical Architect - DataJob DescriptionThe ideal candidate for this senior technical architect position is an experienced professional with a strong background in leading data engineering teams, designing and implementing end-to-end data pipelines, and integrating various data sources and targets.Architect and design scalable data warehouse...

  • Expert Data Pipeline Architect

    5 days ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Job OverviewWe are seeking a skilled Data Engineer with 4–7 years of experience to build and optimize scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. The role involves working across scrum teams to develop data solutions, ensure data governance with Unity Catalog, and support real-time and batch processing.

  • Chief Data Engineering Architect

    5 days ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Senior Data Engineer Position OverviewWe are seeking an experienced Senior Data Engineer to design and develop our modern data platform. This is a fantastic opportunity for a skilled professional to lead the development of scalable, efficient ETL/ELT pipelines, data flows, and lakehouse architectures.Responsibilities:Design and implement optimized ETL/ELT...

  • Senior Data Architect

    9 hours ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineering Full time US$ 9,00,000 - US$ 12,00,000

    Join Our Team as a Senior Data EngineerWe are seeking an experienced Senior Data Engineer to join our team.The ideal candidate will have a strong background in data engineering, experience with big data technologies such as Hadoop and Spark, and a proven track record of designing and implementing scalable data pipelines.Key Responsibilities:Design,...

  • Senior Data Pipeline Specialist

    1 week ago

    Aligarh, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are currently seeking an experienced Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and optimizing data processing pipelines using Azure Cloud technologies.Key ResponsibilitiesDesign, develop, and maintain scalable ETL pipelines for batch data ingestion and egress using Azure Data Factory,...