Kaleidofin - Lead Data Engineer - ETL/PySpark

1 week ago


Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

What youll do ?

  • You will take on the role of technical expert and a lead member of our Engineering team.
  • Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc.
  • Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
  • Designing, building, and operationalizing large-scale enterprise data solutions using Hadoop based technologies along with AWS, Spark, Hive, & Data Lake, using Hive, PySpark and Python Programming.
  • Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various use cases built on the platform.
  • Responsible to Ingest data from files, streams and databases and process the data with Hadoop,Scala, SQL/NOSQL Database, Spark, ML, IoT.
  • Develop programs in Scala and Python as part of data cleaning and processing.
  • It includes data modelling, data ingestion, transformation, data consumption patterns and optimizing complex queries, creating efficient UDFs to extend the functionalities.
  • You will be involved in product feature development and will be working in close partnership with other engineering teams.
  • You will be responsible for mentoring other team members and ensuring high availability platform and stability for batch and stream processing systems.

Who you need to be ?

  • Experience - Min. 6 Years.
  • Bachelor's degree in computer science or computing-related discipline from premier institutes.
  • The candidate should have data processing ability (ETL techniques) using have scripting experience.
  • Native Data engineer with 3 to 8 years of hands-on experience with SQL and NOSQL databases such as HBase, Cassandra or MongoDB.
  • Extensive 2+ years experience working with Hadoop and related processing frameworks such as

Spark, Hive, Kafka etc.

  • Experience working with REST and SOAP based APIs to extract data for data pipelines.
  • Experience with cloud-native ETL languages/frameworks (Scala, Python, Databricks, AWS Glue).
  • Experience in working with Real time data streams and Kafka Platform.
  • Working knowledge with workflow orchestration tools like Apache Airflow design and deploy DAGs.
  • Location : Bangalore.

)



  • Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    What youll do ? - You will take on the role of technical expert and a lead member of our Engineering team. - Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc. - Responsible to design and develop distributed, high volume, high velocity multi-threaded...


  • Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 1,20,000 - ₹ 2,60,000 per year

    Lead -Software Engineer- BangaloreWho are we?Kaleidofin is a fintech focused on the mission to build a digital ecosystem ensuring finance for everyone, everywhere. We provide the ki credit platform, combining credit health assessment (ki score), middleware rails and best-in-class risk management (ki view) to enable the financial ecosystem to serve customers...

  • Pyspark Lead

    2 weeks ago


    Bengaluru, Karnataka, India Krazy Mantra HR Solutions Pvt. Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for a skilled PySpark Lead with 5 to 8 years of experience, based in Bangalore. The ideal candidate will have expertise in PySpark, SQL, ETL/Data Engineering Pipeline, Cloud understanding, CI/CD, Data Modelling, Airflow, Kafka, EMR/Databricks, Snowflake, and Hive.Roles and ResponsibilityDesign and develop scalable data pipelines using PySpark...

  • Data Engineer

    6 days ago


    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Migrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix systems....

  • Lead Data Engineer

    1 week ago


    Bengaluru, Karnataka, India Mount Talent Consulting Full time ₹ 2,00,000 - ₹ 3,00,000 per year

    LOOKING FOR IMMEDIATE/ EARLY JOINERS ONLY.Role: Lead Data engineerLocation: BengaluruMode: 5 days work from office (Alternate Saturday work from home)Budget: 20 LPA- 30 LPA Max.A Data Engineer with PySpark experience develops, maintains, and optimizes scalable data pipelines and infrastructure, focusing on data extraction, transformation, and loading...

  • ETL Data Architect

    1 week ago


    Bengaluru, Karnataka, India EduRun Group Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    As a candidate for this role, you should have over 10 years of experience in designing and implementing scalable data solutions. This includes creating end-to-end ETL/ELT pipelines using Databricks (PySpark) and Snowflake. You will be responsible for building and managing secure, scalable data platforms using PostgreSQL and DynamoDB that are tailored to the...


  • Bengaluru, Karnataka, India EdgeVerve Systems Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Pipeline/ETL (Extract, Transform, Load) processes, API integration, scripting languages (Python), and big data technologies (Trio, Iceberg, DuckDB/Parquet).Database design, data modeling, and data warehousing. SQL and at least one cloud platform.Analytical tools (Superset, Power BI), statistical analysis, and SQL. To derive insights from data.Data governance...


  • Bengaluru, Karnataka, India NatWest Group Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Join us as a Data Engineer, PySparkThis is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless, digital first customer experiencesYou'll be simplifying the bank by developing innovative data driven solutions, using insight to be commercially successful, and keeping our customers' and the bank's data...

  • Data Engineer, Pyspark

    21 hours ago


    Bengaluru, Karnataka, India NatWest Group Full time ₹ 1,50,00,000 - ₹ 2,50,00,000 per year

    Join us as a Data EngineerWe're looking for someone to build effortless, digital first customer experiences to help simplify our organisation and keep our data safe and secureDay-to-day, you'll develop innovative, data-driven solutions through data pipelines, modelling and ETL design while inspiring to be commercially successful through insightsIf you're...

  • Data Engineer lead

    1 week ago


    Bengaluru, Karnataka, India Innova ESI Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Role: Data Engineer leadExperience: 10+ YearsLocation: BangaloreNotice: Immediate JoinersJob Description:Job Description – Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL)Role Overview:Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog,...