Senior Engineering Leader for Large Scale Data Systems

3 days ago

Rajkot, Gujarat, India beBeeEngineer Full time ₹ 1,56,12,996 - ₹ 2,03,61,948
Lead Data Engineer Opportunity

We are seeking an experienced Lead Data Engineer to join our team.

This is a key leadership role, responsible for driving the development of large-scale data pipelines using Apache Beam and Databricks. The successful candidate will have hands-on experience with performance and concurrency, observability expertise, and a strong understanding of cloud technologies.

Key Responsibilities:
  • Architecture and Reuse: Design and build a shared component library/SDK for pipelines: ingestion, parsing/OCR, extraction, validation, enrichment, publishing.
  • Define patterns/templates: For Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Create pluggable interfaces: So multiple teams can swap extractors, OCR providers, and EMR publishers without code rewrites.
  • Define repository strategy: Shared/child repos for each use case.
Performance and Reliability:
  • Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
  • Instrument services: With Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
  • Implement concurrency best practices: Asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
  • Implement robust LLM API rate limiting/governance: Enforce provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
  • Establish SLOs/alerts: For throughput, latency, error rates; set up DLQs and recovery patterns.
Team Enablement:
  • Mentor developers: Lead design reviews, codify best practices, write clear docs and examples.
  • Partner with ML engineers: On the future LLM/SLM path (evaluation harness, safety/PII, cost/perf).

  • Large Scale Data Engineer Wanted

    2 days ago

    Rajkot, Gujarat, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 15,00,000

    Job TitleWe are seeking a skilled Freelance Data Engineer to join our team on a project basis. The ideal candidate will have experience in designing and building large-scale data pipelines using Python, SQL, Linux/Bash, PySpark, AWS, and Databricks.The successful freelancer will collaborate with internal teams and clients to deliver robust, scalable, and...

  • Senior Big Data Engineer

    3 days ago

    Rajkot, Gujarat, India beBeeBigDataEngineer Full time ₹ 75,00,000 - ₹ 1,50,00,000

    Senior Big Data EngineerWe are seeking a highly skilled Senior Big Data Engineer to join our team. The successful candidate will be responsible for designing, building, and maintaining large-scale data systems.

  • Senior Data Systems Engineer

    3 days ago

    Rajkot, Gujarat, India beBeeData Full time US$ 1,00,000 - US$ 1,40,000

    Job TitleDevelop and maintain large-scale data systems using Python and AWS cloud services. This includes creating, implementing, and optimizing ETL processes for vast datasets.Key Responsibilities:Ensure data integrity, security, and accessibility by adhering to best practices and regulations.Collaborate with cross-functional teams to meet business data...

  • Data Engineering Leader

    3 days ago

    Rajkot, Gujarat, India beBeeDataEngineer Full time ₹ 12,45,550 - ₹ 26,12,714

    Job Description\Sigmoid is empowering enterprises to make smarter, data-driven decisions by blending advanced data engineering with AI consulting.We collaborate with leading data-rich organizations across sectors to solve complex business challenges.Our global team specializes in cloud data modernization, predictive analytics, generative AI, and...

  • Senior Gameplay Engineer

    1 day ago

    Rajkot, Gujarat, India beBeeStrategy Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Gameplay Engineer - Real-time Multiplayer StrategyWe are seeking a skilled Gameplay Engineer to join our team and help design and implement core gameplay and real-time multiplayer systems for our high-scale strategy experiences.

  • High-Level Data System Administrator

    4 days ago

    Rajkot, Gujarat, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data Specialist will oversee large-scale data systems and collaborate with various stakeholders to ensure successful project delivery.The role involves working closely with business partners, data engineers, operations teams, and IT groups across the organization to deliver high-quality results.Administer and maintain scalable cloud environments and...

  • Strategic Data Architect

    2 days ago

    Rajkot, Gujarat, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Build Scalable Data SolutionsWe're seeking a skilled Senior Data Engineer to join our team of experts. In this role, you'll design and construct large-scale data processing systems that enable advanced analytics and business intelligence.Key Responsibilities:

  • Senior Big Data Architect

    5 days ago

    Rajkot, Gujarat, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Azure Data EngineerThe role of the data engineer involves working with large datasets and designing efficient systems to store, process, and retrieve data.About the Role: As a data engineer, you will be responsible for architecting, building, testing, and deploying large-scale data processing systems. You will work closely with cross-functional...

  • High Performance Data Architect

    3 days ago

    Rajkot, Gujarat, India beBeeDataEngineering Full time ₹ 1,99,75,500 - ₹ 2,14,79,000

    Senior Data Engineer PositionWe are seeking a seasoned data engineering leader to drive the optimization and scalability of our data delivery pipelines. As a key member of our team, you will be responsible for architecting and maintaining robust high-throughput data pipelines and infrastructure.This role requires 6+ years of professional experience in data...

  • Data Engineer

    2 days ago

    Rajkot, Gujarat, India beBeeDataEngineer Full time ₹ 3,00,00,000 - ₹ 6,00,00,000

    Job Title: Data EngineerWe are seeking a skilled data engineer to design, develop and maintain large-scale data processing systems.The ideal candidate will have a strong background in big data technologies such as PySpark, Hive, Spark, and HBase. Experience with data ingestion from disparate sources onto the big data platform is highly desirable.We require...