
Kaleidofin - Lead Data Engineer - ETL/PySpark
3 days ago
What youll do ?
- You will take on the role of technical expert and a lead member of our Engineering team.
- Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc.
- Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
- Designing, building, and operationalizing large-scale enterprise data solutions using Hadoop based technologies along with AWS, Spark, Hive, & Data Lake, using Hive, PySpark and Python Programming.
- Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various use cases built on the platform.
- Responsible to Ingest data from files, streams and databases and process the data with Hadoop,Scala, SQL/NOSQL Database, Spark, ML, IoT.
- Develop programs in Scala and Python as part of data cleaning and processing.
- It includes data modelling, data ingestion, transformation, data consumption patterns and optimizing complex queries, creating efficient UDFs to extend the functionalities.
- You will be involved in product feature development and will be working in close partnership with other engineering teams.
- You will be responsible for mentoring other team members and ensuring high availability platform and stability for batch and stream processing systems.
Who you need to be ?
- Experience - Min. 6 Years.
- Bachelor's degree in computer science or computing-related discipline from premier institutes.
- The candidate should have data processing ability (ETL techniques) using have scripting experience.
- Native Data engineer with 3 to 8 years of hands-on experience with SQL and NOSQL databases such as HBase, Cassandra or MongoDB.
- Extensive 2+ years experience working with Hadoop and related processing frameworks such as
Spark, Hive, Kafka etc.
- Experience working with REST and SOAP based APIs to extract data for data pipelines.
- Experience with cloud-native ETL languages/frameworks (Scala, Python, Databricks, AWS Glue).
- Experience in working with Real time data streams and Kafka Platform.
- Working knowledge with workflow orchestration tools like Apache Airflow design and deploy DAGs.
- Location : Bangalore.
(ref:hirist.tech)
-
Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per yearWhat youll do ?You will take on the role of technical expert and a lead member of our Engineering team.Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc.Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing...
-
Lead- Data Engineer
2 days ago
Greater Bengaluru Area, India Kaleidofin Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per yearWho we are?Kaleidofin is a fintech focused on the mission to build a digital ecosystem ensuring finance for everyone, everywhere. We provide the ki credit platform, combining credit health assessment (ki score), middleware rails and best-in-class risk management (ki view) to enable the financial ecosystem to serve customers better. Via credit health scores...
-
Lead Software Engineer
2 days ago
Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 1,20,000 - ₹ 2,60,000 per yearLead -Software Engineer- BangaloreWho are we?Kaleidofin is a fintech focused on the mission to build a digital ecosystem ensuring finance for everyone, everywhere. We provide the ki credit platform, combining credit health assessment (ki score), middleware rails and best-in-class risk management (ki view) to enable the financial ecosystem to serve customers...
-
Pyspark Lead
7 days ago
Bengaluru, Karnataka, India Krazy Mantra HR Solutions Pvt. Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per yearWe are looking for a skilled PySpark Lead with 5 to 8 years of experience, based in Bangalore. The ideal candidate will have expertise in PySpark, SQL, ETL/Data Engineering Pipeline, Cloud understanding, CI/CD, Data Modelling, Airflow, Kafka, EMR/Databricks, Snowflake, and Hive.Roles and ResponsibilityDesign and develop scalable data pipelines using PySpark...
-
Data Engineer
2 weeks ago
Bengaluru, India NTT Data Full timeJob Description NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer - ETL (BODS) to join our team in Bangalore, Karntaka (IN-KA), India (IN). Job Duties: . Migrate ETL...
-
Etl (Talend/pyspark)
1 week ago
Bengaluru, India DEzen Technology Solutions Pvt Ltd. Full time**Recruitment Company Name**: - DEzen Technology Solutions Pvt Ltd. **Requirements**: - 4 -7 Years - ETL (Talend/PySpark) **Skills**: - ETL (Talend/PySpark) - ETL Developer Talend / Pyspark Upload Resume **Job Details**: - **Role**: ETL (Talend/PySpark) - **Job Type**: Full Time - **Location**: Bangalore - **Offered Salary**: open - **Posted**:...
-
Data Engineer
7 days ago
Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 10,00,000 - ₹ 25,00,000 per yearJob Duties:Migrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix...
-
Bengaluru, Karnataka, India Kaleidofin Private Limited Full timeRole : Backend DeveloperLocation : Bangalore, Chennai, IndiaExperience : 3 - 6 YearsEmployment Type : Full-timeJob Overview : We are seeking extremely smart and passionate Backend Developers with 3-6 years of experience to join our team. You'll be instrumental in building large-scale, enterprise-ready systems that will significantly impact millions of...
-
Lead Data Engineer
2 days ago
Bengaluru, Karnataka, India Mount Talent Consulting Full time ₹ 2,00,000 - ₹ 3,00,000 per yearLOOKING FOR IMMEDIATE/ EARLY JOINERS ONLY.Role: Lead Data engineerLocation: BengaluruMode: 5 days work from office (Alternate Saturday work from home)Budget: 20 LPA- 30 LPA Max.A Data Engineer with PySpark experience develops, maintains, and optimizes scalable data pipelines and infrastructure, focusing on data extraction, transformation, and loading...
-
Senior Data Engineer – Python
20 hours ago
Bengaluru, India Xebia Full timeWe’re Hiring: Senior Data Engineer – Python & PySparkLocation: Bangalore (Hybrid – 3 days office per week)We are looking for an experienced Senior Data Engineer with a strong background in Python (with OOPs concepts), PySpark, and building test cases . The ideal candidate must have 6+ years of hands-on experience and be available to join ...