Senior Data Engineer

4 days ago

Konkan Division, India Celebal Technologies Full time

Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: •Design, develop, and maintain real-time and batch data pipelines in Databricks. •Integrate Apache Kafka with Databricks using Structured Streaming. •Implement robust data ingestion frameworks using Databricks Autoloader. •Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers. •Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs. •Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic. •Develop reusable components for merge/upsert operations and window function-based transformations. •Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques. •Collaborate with cross-functional teams to ensure data availability, reliability, and consistency. Must Have: •Apache Kafka: Integration, topic management, schema registry (Avro/JSON). •Databricks & Spark Structured Streaming: oProcessing Modes: Append, Update, Complete oOutput Modes: Memory, Console, File, Kafka, Delta oCheckpointing and fault tolerance •Databricks Autoloader: Schema inference, schema evolution, incremental loads. •Medallion Architecture implementation expertise. •Performance Optimization: oData partitioning strategies oCaching and persistence oAdaptive query execution and cluster configuration tuning •SQL & Spark SQL: Proficiency in writing efficient queries and transformations. •Data Governance: Schema enforcement, data quality checks, and monitoring. Good to Have: •Strong coding skills in Python and PySpark. •Experience working in CI/CD environments for data pipelines. •Exposure to cloud platforms (AWS/Azure/GCP). •Understanding of Delta Lake, time travel, and data versioning. •Familiarity with orchestration tools like Airflow or Azure Data Factory. Please share your CVs shailendra.sharma@celebaltech.com Immediate Joiners Preferred

  • Senior Electrical Design Engineer

    4 days ago

    Konkan Division, India Integrated Personnel Services Limited Full time

    skills : - Senior Electrical Engineer Location : - Navi Mumbai Prepare Basis of Design, Load list, Single Line Diagram, Schematics. Power distribution from 33kV / 22kV and downstream based on power ratings of equipment to various voltage levels 11kV / 6.6kV / 3.3kV & 415V. Other national standard voltage levels for GID project. Calculations for Transformer...

  • Lead - Data Science

    4 days ago

    Konkan Division, India ICICIDirect Full time

    Experience: 12- 16 Years About the Role: We are looking for a seasoned professional with deep expertise in analytics and AI, particularly within the BFSI domain. The ideal candidate will bring strong hands-on experience in Python-based statistical and machine learning model development, along with exposure to AWS AI stack and GenAI technologies. This role...

  • Senior Manager

    4 days ago

    Konkan Division, India eClerx Full time

    eClerx is hiring for Senior Process Manager in FCC Job location: Navi Mumbai The ideal candidate should possess strong business understanding of finance industry with special focus on KYC lifecycle, AML, Due Diligence, periodic refresh, KYC review, and remediation. He/ She should have complete knowledge on ultimate beneficiary ownership, source of...

  • Senior Data Engineer

    4 days ago

    Bangalore Division, India USEReady Full time

    Job Title: Senior Databricks Engineer Experience Level: 5-8 Years Job Summary As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakehouse Platform. You will leverage your deep expertise in Spark, Delta Lake, and cloud technologies to build scalable and...

  • GTM Engineer

    4 days ago

    Konkan Division, India Pixelo Digital Full time

    How to Apply: If you find the job opportunity relevant and exciting, please share your additional details here: This will help the hiring team reach out to you at the earliest. About Pixelo: As a new age B2B demand generation and product company, we specialize in accelerating sales for our clients through innovative marketing strategies. At Pixelo, we take...

  • Co-Founder & Senior Partner

    4 days ago

    Konkan Division, India ExoTalent Consultancy Services LLP Full time

    Co-Founder & Partner – ExoTalent Position: Co-Founder & Partner – ExoTalent Location: Navi Mumbai (Onsite, with travel across India & Middle East) Engagement Type: Equity + Revenue Share (Investor–Co-Founder role) Experience Required: 15–25 years in Talent Acquisition, Recruitment, RPO Sales & Operations Start Date: Immediate Industry : Recruitment/...

  • Senior Data Engineer

    4 days ago

    Bangalore Division, India Societe Generale Global Solution Centre Full time

    Responsibilities: The role is a Senior Data Engineer who will help shape the future of data-driven auditing. With 7 - 10 years of experience, you will design and maintain scalable data pipelines, ensure compliance with global data regulations (including GDPR), and empower auditors/inspectors with high-quality data and advanced analytics. You will masterize...

  • Senior Data Engineer

    4 days ago

    Pune Division, India Persistent Systems Full time

    About Position: A highly skilled Data Engineering professional with 6 to 8 years of experience in designing, developing, and maintaining robust data solutions. Proven ability to work across the full software development lifecycle, from requirement gathering to deployment and support. Adept at working in Agile environments and collaborating with...

  • Senior Data Platform Engineer

    2 weeks ago

    Delhi Division, India Invoria Global Solutions Full time

    Invoria Global Solutions (IGS) is associated with a network of strategic consulting partners who work closely with top enterprise clients across India. This opportunity is being offered through our strategic partner, who holds a direct engagement with the end client. Selected candidates will be on our partner’s payroll and will be deployed to the...

  • Senior Data Engineer

    4 days ago

    Bangalore Division, India Zeta Full time

    About Zeta Zeta is a Next-Gen Banking Tech company that empowers banks and fintechs to launch banking products for the future. It was founded by and Ramki Gaddipati in 2015. Our flagship processing platform - Zeta Tachyon - is the industry’s first modern, cloud-native, and fully API-enabled stack that brings together issuance, processing, lending, core...