Kaleidofin - Lead Data Engineer - ETL/PySpark

2 days ago

Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

What youll do ?

  • You will take on the role of technical expert and a lead member of our Engineering team.
  • Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc.
  • Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
  • Designing, building, and operationalizing large-scale enterprise data solutions using Hadoop based technologies along with AWS, Spark, Hive, & Data Lake, using Hive, PySpark and Python Programming.
  • Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various use cases built on the platform.
  • Responsible to Ingest data from files, streams and databases and process the data with Hadoop,Scala, SQL/NOSQL Database, Spark, ML, IoT.
  • Develop programs in Scala and Python as part of data cleaning and processing.
  • It includes data modelling, data ingestion, transformation, data consumption patterns and optimizing complex queries, creating efficient UDFs to extend the functionalities.
  • You will be involved in product feature development and will be working in close partnership with other engineering teams.
  • You will be responsible for mentoring other team members and ensuring high availability platform and stability for batch and stream processing systems.

Who you need to be ?

  • Experience - Min. 6 Years.
  • Bachelor's degree in computer science or computing-related discipline from premier institutes.
  • The candidate should have data processing ability (ETL techniques) using have scripting experience.
  • Native Data engineer with 3 to 8 years of hands-on experience with SQL and NOSQL databases such as HBase, Cassandra or MongoDB.
  • Extensive 2+ years experience working with Hadoop and related processing frameworks such as

Spark, Hive, Kafka etc.

  • Experience working with REST and SOAP based APIs to extract data for data pipelines.
  • Experience with cloud-native ETL languages/frameworks (Scala, Python, Databricks, AWS Glue).
  • Experience in working with Real time data streams and Kafka Platform.
  • Working knowledge with workflow orchestration tools like Apache Airflow design and deploy DAGs.
  • Location : Bangalore.

)

  • Lead Software Engineer

    2 days ago

    Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 1,20,000 - ₹ 2,60,000 per year

    Lead -Software Engineer- BangaloreWho are we?Kaleidofin is a fintech focused on the mission to build a digital ecosystem ensuring finance for everyone, everywhere. We provide the ki credit platform, combining credit health assessment (ki score), middleware rails and best-in-class risk management (ki view) to enable the financial ecosystem to serve customers...

  • Pyspark Lead

    7 days ago

    Bengaluru, Karnataka, India Krazy Mantra HR Solutions Pvt. Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for a skilled PySpark Lead with 5 to 8 years of experience, based in Bangalore. The ideal candidate will have expertise in PySpark, SQL, ETL/Data Engineering Pipeline, Cloud understanding, CI/CD, Data Modelling, Airflow, Kafka, EMR/Databricks, Snowflake, and Hive.Roles and ResponsibilityDesign and develop scalable data pipelines using PySpark...

  • Data Engineer

    7 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Job Duties:Migrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix...

  • Kaleidofin - Backend Developer - Java Technologies

    2 weeks ago

    Bengaluru, Karnataka, India Kaleidofin Private Limited Full time

    Role : Backend DeveloperLocation : Bangalore, Chennai, IndiaExperience : 3 - 6 YearsEmployment Type : Full-timeJob Overview : We are seeking extremely smart and passionate Backend Developers with 3-6 years of experience to join our team. You'll be instrumental in building large-scale, enterprise-ready systems that will significantly impact millions of...

  • Lead Data Engineer

    2 days ago

    Bengaluru, Karnataka, India Mount Talent Consulting Full time ₹ 2,00,000 - ₹ 3,00,000 per year

    LOOKING FOR IMMEDIATE/ EARLY JOINERS ONLY.Role: Lead Data engineerLocation: BengaluruMode: 5 days work from office (Alternate Saturday work from home)Budget: 20 LPA- 30 LPA Max.A Data Engineer with PySpark experience develops, maintains, and optimizes scalable data pipelines and infrastructure, focusing on data extraction, transformation, and loading...

  • Python Pyspark Data Engineer

    4 weeks ago

    Bengaluru, Karnataka, India DXC Technology Full time

    Python Pyspark Data Engineer Job Location Hyderabad Bangalore Chennai Kolkata Noida Gurgaon Pune Indore Mumbai Strong Python Skills - Proficient in Python for data manipulation automation and building reusable components Data Pipeline Development - Experience designing and maintaining ETL data pipelines using tools like Airflow or...

  • Data Engineer, PySpark

    7 days ago

    Bengaluru, Karnataka, India RBS Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Join us as a Data Engineer, PySparkThis is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless, digital first customer experiencesYou'll be simplifying the bank by developing innovative data driven solutions, using insight to be commercially successful, and keeping our customers' and the bank's data...

  • Data Engineer, PySpark

    2 days ago

    Bengaluru, Karnataka, India NatWest Group Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Join us as a Data Engineer, PySparkThis is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless, digital first customer experiencesYou'll be simplifying the bank by developing innovative data driven solutions, using insight to be commercially successful, and keeping our customers' and the bank's data...

  • ETL Data Architect

    2 days ago

    Bengaluru, Karnataka, India EduRun Group Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    As a candidate for this role, you should have over 10 years of experience in designing and implementing scalable data solutions. This includes creating end-to-end ETL/ELT pipelines using Databricks (PySpark) and Snowflake. You will be responsible for building and managing secure, scalable data platforms using PostgreSQL and DynamoDB that are tailored to the...

  • Data Engineer lead

    3 days ago

    Bengaluru, Karnataka, India Innova ESI Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Role: Data Engineer leadExperience: 10+ YearsLocation: BangaloreNotice: Immediate JoinersJob Description:Job Description – Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL)Role Overview:Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog,...