Senior Data Engineering Leader

5 days ago

Jamnagar, Gujarat, India beBeeEngineering Full time US$ 1,20,000 - US$ 1,50,000
Job Description

Our team is looking for a senior engineer to lead the development of our data pipelines. The successful candidate will have hands-on experience with Apache Beam, Databricks and other big data technologies.

  • Design and build a shared component library/SDK for pipelines: ingestion, parsing/OCR, extraction (RegEx now, LLM/SLM later), validation, enrichment, publishing.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Create pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.
  • Define repo strategy - shared/child repos for each use case
Required Skills and Qualifications

To be successful in this role, you will need:

  • 7+ years Python with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing), profiling and memory tuning.
  • Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
  • Must have implemented LLM based solutions and supported them in production
  • API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
  • Strong architecture/design skills: clean interfaces, packaging shared libs, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
  • 3+ years building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
  • Document processing: OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, text normalization.
  • LLM/SLM integration experience (e.g., OpenAI/Azure AI, local SLMs), prompt/eval frameworks, PII redaction/guardrails.
  • Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
  • Security/compliance mindset (HIPAA), secrets management, least-privilege access.
Benefits

We offer a competitive salary and benefits package, as well as opportunities for career growth and professional development.

  • Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
  • Opportunity to work with cutting-edge technologies
  • Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
  • Annual health check-ups
  • Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents
Work Environment

We value diversity and inclusion in the workplace and welcome applications from all qualified individuals.

  • We support hybrid work and flexible hours to fit diverse lifestyles.
  • Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
  • If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

  • Data Engineering Leader

    4 days ago

    Jamnagar, Gujarat, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job DescriptionWe are seeking a skilled Senior Data Engineer Lead to design, develop and optimize ETL/ELT data pipelines using Apache Spark (PySpark or Scala), AWS Glue, and Azure Data Factory.Responsibilities include working with structured and unstructured data to build scalable ingestion and transformation workflows across cloud platforms. You will...

  • Senior Data Science Leader

    5 days ago

    Jamnagar, Gujarat, India beBeeDataScience Full time ₹ 2,50,00,000 - ₹ 3,50,00,000

    Job Title: Senior Data Science LeaderDescription:We are seeking a seasoned leader to define and shape our data science initiatives, driving innovation in AI/ML, building scalable data solutions, and impacting our business.Key Responsibilities:Lead the development of strategic plans for data-driven business growth, aligning with company objectives.Design and...

  • Senior Data Analytics Leader

    5 days ago

    Jamnagar, Gujarat, India beBeeData Full time ₹ 75,00,000 - ₹ 1,50,00,000

    Senior Data Analytics LeaderAbout the RoleLeads and manages delivery across multiple client engagements, prioritizing accounts and customer value to ensure critical ones receive attention.

  • Data Engineering Leader

    3 hours ago

    Jamnagar, Gujarat, India beBeeDataEngineer Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Job Title: Data Engineer LeaderAbout the RoleWe are seeking a seasoned data professional to lead our team in designing and developing scalable data pipelines using Oracle Data Flow or Apache Spark (PySpark), with a solid understanding of data patterns, Python, and SQL.The ideal candidate will have 3-12 years of experience in building and managing data lakes,...

  • Visionary Data Engineering Leader

    3 hours ago

    Jamnagar, Gujarat, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,00,00,000

    Role Summary:We are seeking an experienced Senior Data Engineer to lead our data engineering efforts. This role will involve designing, developing, and implementing data solutions that drive business value.Key Responsibilities:Laying the foundation for our projects through effective design, development, and implementationMentoring team members and fostering...

  • Chief Engineer and Academic Leader

    4 days ago

    Jamnagar, Gujarat, India beBeeVisionary Full time ₹ 2,00,00,000 - ₹ 3,00,00,000

    Engineering Leadership PositionWe are seeking a highly skilled and visionary leader to join our faculty as Director of Engineering. This is an excellent opportunity for someone with 15+ years of experience in academia, research, or R&D leadership.The successful candidate will provide academic and strategic leadership to the engineering program, building...

  • Senior Director of Data Strategy

    4 hours ago

    Jamnagar, Gujarat, India beBeeDataStrategy Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Senior Director of Data StrategyWe are seeking a visionary leader to drive our data migration initiatives as the Senior Director of Data Strategy.The ideal candidate will have extensive experience in leading complex data migration projects, with a deep understanding of data migration methodologies, tools, and best practices.This role requires a...

  • Senior Cloud Data Engineer

    4 days ago

    Jamnagar, Gujarat, India beBeeEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Cloud Data Engineer PositionAbout the Role:We are seeking an experienced and skilled Senior Cloud Data Engineer to join our team. The ideal candidate will have a strong background in designing, implementing, and optimizing data pipelines using cloud-based technologies.The successful candidate will be responsible for developing scalable and efficient data...

  • Data Solutions Leader

    3 hours ago

    Jamnagar, Gujarat, India beBeeAzure Full time ₹ 1,20,00,000 - ₹ 1,80,00,000

    Job DescriptionWe are seeking a seasoned professional to lead our Azure Data Support team.Team Leadership: Technical expertise in ITIL processes and incident management frameworks is essential for this role. As a mentor and leader, you will guide a team of engineers in efficiently handling support activities.Technical Oversight: Oversee the monitoring and...

  • Airflow Senior Data Engineer Opportunity

    3 hours ago

    Jamnagar, Gujarat, India beBeeData Full time ₹ 10,00,000 - ₹ 19,00,000

    Senior Data Engineer PositionThe ideal candidate will have a strong background in data engineering, with expertise in Airflow and Python. They will be responsible for designing, developing, and maintaining large-scale data processing pipelines using Snowflake.Key Responsibilities:Design and develop complex data processing pipelines using Airflow and...