
Kaleidofin - Lead Data Engineer - ETL/PySpark
2 weeks ago
What youll do ?
- You will take on the role of technical expert and a lead member of our Engineering team.
- Designing and building data pipelines from data ingestion to consumption within a hybrid big data architecture, using Cloud Native AWS, NOSQL, SQL etc.
- Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
- Designing, building, and operationalizing large-scale enterprise data solutions using Hadoop based technologies along with AWS, Spark, Hive, & Data Lake, using Hive, PySpark and Python Programming.
- Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various use cases built on the platform.
- Responsible to Ingest data from files, streams and databases and process the data with Hadoop,Scala, SQL/NOSQL Database, Spark, ML, IoT.
- Develop programs in Scala and Python as part of data cleaning and processing.
- It includes data modelling, data ingestion, transformation, data consumption patterns and optimizing complex queries, creating efficient UDFs to extend the functionalities.
- You will be involved in product feature development and will be working in close partnership with other engineering teams.
- You will be responsible for mentoring other team members and ensuring high availability platform and stability for batch and stream processing systems.
Who you need to be ?
- Experience - Min. 6 Years.
- Bachelor's degree in computer science or computing-related discipline from premier institutes.
- The candidate should have data processing ability (ETL techniques) using have scripting experience.
- Native Data engineer with 3 to 8 years of hands-on experience with SQL and NOSQL databases such as HBase, Cassandra or MongoDB.
- Extensive 2 years experience working with Hadoop and related processing frameworks such as
Spark, Hive, Kafka etc.
- Experience working with REST and SOAP based APIs to extract data for data pipelines.
- Experience with cloud-native ETL languages/frameworks (Scala, Python, Databricks, AWS Glue).
- Experience in working with Real time data streams and Kafka Platform.
- Working knowledge with workflow orchestration tools like Apache Airflow design and deploy DAGs.
- Location : Bangalore.
-
Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per yearDescriptionWhat youll do :As a Software Development Manager, you will be responsible for shaping product features and driving the technical execution of the product roadmap.You will work closely with your team to build systems that stress scalability, accuracy, and robustness. This is a hands-on leadership role you will actively participate in design,...
-
Data Engineer with SQL/ETL/Pyspark
16 hours ago
Bengaluru, Karnataka, India Infosys Finacle Full time ₹ 12,00,000 - ₹ 36,00,000 per yearMandate Skills - SQL, ETL ,hadoop, pyspark, Apache, KafkaRequired Skills5+ years of relevant Data engineering experience using Python or Java as tech be well versed into HadoopDevelopment expertise with 2+ years of experience, in SQL, ETL processes and other similar technologiesSkilled in Identifying and optimizing database queriesStrong understanding of...
-
Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per yearDescription :What youll do : As a Software Development Manager, you will be responsible for shaping product features and driving the technical execution of the product roadmap. You will work closely with your team to build systems that stress scalability, accuracy, and robustness. This is a hands-on leadership role you will actively participate in...
-
Data Engineer
1 week ago
Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per yearMigrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix systems....
-
DevOps Engineer
16 hours ago
Bengaluru, Karnataka, India Kaleidofin Private Limited Full time ₹ 15,00,000 - ₹ 25,00,000 per yearWho we are?Kaleidofin is a fintech focused on the mission to build a digital ecosystem ensuring finance for everyone, everywhere. We provide the ki credit platform, combining credit health assessment (ki score), middleware rails and best-in-class risk management (ki view) to enable the financial ecosystem to serve customers better. Via credit health scores...
-
Hiring For Pyspark developer + ETL
2 weeks ago
Bengaluru, Karnataka, India Altimetrik Full time ₹ 1,50,000 - ₹ 2,50,000 per year,Greeting from Altimetrik We are looking for a highly skilled and experienced Data Engineer join our dynamic team.Technical Skills & Qualifications:Mandatory Skills : Python, Pyspark ,SQL , ETL , Spark ,Hive, Data Ware House/Datamart design , AWS/AZUREExperience : 5 to 10 yrs only . 5+ MandatoryLocation : Bangalore only.Educational Qualification:Bachelors...
-
Bengaluru, Karnataka, India Kaleidofin Private Limited Full timeRole : Backend DeveloperLocation : Bangalore, Chennai, IndiaExperience : 3 - 6 YearsEmployment Type : Full-timeJob Overview : We are seeking extremely smart and passionate Backend Developers with 3-6 years of experience to join our team. You'll be instrumental in building large-scale, enterprise-ready systems that will significantly impact millions of...
-
Data Engineer, Pyspark
6 days ago
Bengaluru, Karnataka, India NatWest Group Full time ₹ 1,50,00,000 - ₹ 2,50,00,000 per yearJoin us as a Data EngineerWe're looking for someone to build effortless, digital first customer experiences to help simplify our organisation and keep our data safe and secureDay-to-day, you'll develop innovative, data-driven solutions through data pipelines, modelling and ETL design while inspiring to be commercially successful through insightsIf you're...
-
ETL Automation Tester
6 days ago
Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 8,00,000 - ₹ 12,00,000 per yearMandatory SkillsSQL,ELT/ETL Testing, Python, Data Validation, Pyspark/Pytest/Junit/TestNG and Azure Data services, Databricks QA, Data Lake QAGood to have SkillsDevops, Azure cloud Devops, Data governance, API testing
-
Data Engineer lead
2 weeks ago
Bengaluru, Karnataka, India Expian Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per yearRole & responsibilities8+ years of experience as a Data Engineer, with a focus on Databricks and cloud-based data platforms, with a minimum of 4 years of experience in writing unit/end-to-end tests for data pipelines and ETL processes on Databricks.Hands-on experience in PySpark programming for data manipulation, transformation, and analysis.Strong...