Data Engineer

4 weeks ago

Bengaluru, India TalentOnLease Pvt Ltd. Full time

Description :- Big Data Orchestration : Airflow, Spark on Kubernetes, Yarn, Oozie- Big Data Processing : Hadoop, Kafka, Spark & Spark Structured Streaming- Experience on SOLID & DRY principles with Good Software Architecture & Design implementation experience- Advanced Scala experience (e. Functional Programming, using Case classes, Complex Data Structures & Algorithms)- Proficient in developing automated frameworks for unit & integration testing- Proficient with Kubernetes, Docker, Helm and related container technologies- Proficient in deploying and managing Spark workloads on Kubernetes clusters- Candidates should have hands on experience in Spark and Scala at least 5+ years with good data engineering concepts- Design, develop, and maintain data pipelines and ETL workflows for large-scale distributed systems. - Implement and manage Big Data orchestration solutions using Apache Airflow, Oozie, Spark on Kubernetes, and YARN.- Work extensively on data processing frameworks including Hadoop, Kafka, Spark, and Spark Structured Streaming. - Develop and optimize real-time and batch data processing systems ensuring scalability, reliability, and high performance. - Apply SOLID and DRY software engineering principles to ensure maintainable and high-quality code. - Architect and implement data solutions using best practices for software design and distributed system patterns. - Write efficient, modular, and testable code using Scala (Functional Programming, Case Classes, Data Structures & Algorithms). - Develop automated unit and integration testing frameworks to ensure code quality and data pipeline reliability.- Manage Spark workloads on Kubernetes clusters and use Docker, Helm, and other container technologies for deployment automation. - Collaborate with cross-functional teams including Data Scientists, DevOps, and Business Analysts to ensure data availability and consistency. - Monitor, troubleshoot, and optimize data workflows for performance and cost efficiency.Proficient in Big Data orchestration tools such as Airflow, Spark on Kubernetes, Yarn, Oozie. - Strong experience with Hadoop, Kafka, Spark Structured Streaming, and related Big Data ecosystems. - Proficiency in Functional Programming and advanced Scala concepts including case classes, higher-order functions, and immutability. (ref:hirist.tech)

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, Karnataka, India Data Engineer Academy Full time US$ 4,80,000 - US$ 12,00,000 per year

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in cloud...

  • Senior Data Architect

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in cloud...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & Databricks Remote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in cloud...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in cloud...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking For- Minimum 9-15 years of experience in cloud...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in cloud...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & DatabricksRemote | Part Time | Flexible Timings | Pay: $600 – $1000/MonthlyAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 9-15 years of experience in cloud...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & Databricks Remote | Part Time | Flexible Timings | Pay: $600 – $1000/Monthly About the Opportunity We are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms. What We're Looking For - Minimum 9-15 years of experience in...

  • Data Engineering Subject Matter Expert

    2 weeks ago

    Bengaluru, India Data Engineer Academy Full time

    We're Hiring: Data Engineering Subject Matter Expert (SME) – Snowflake,Cloud & Databricks Remote | Part Time | Flexible Timings | Pay: $600 – $1000/Monthly About the Opportunity We are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms. What We're Looking For Minimum 9-15 years of experience in...