Scala Spark Data Engineer

Company DescriptionRsquareInfo E-Solution Pvt. Ltd. (RIS) is a leading consultancy service for IT project solutions, founded in 2019. We specialize in Mobile App Development, Website Development, Testing, BPM, RPA, and Salesforce. Our mission is to offer cost-effective solutions in today's competitive market without compromising on quality. Our team, consisting of nationally and internationally experienced, well-qualified engineers, is dedicated to managing inquiries professionally and promptly.Role DescriptionWe are seeking a Senior Data Engineer with deep expertise in Scala-based Spark development and end-to-end deployment of data pipelines on Kubernetes cluster, orchestrated via Airflow. The ideal candidate should have a strong software engineering foundation, excellent understanding of distributed systems, proficient in software design, modern project/code structuring skills, with good understanding on CI/CD processes and implementation which enables them to deliver reliable, scalable and robust data solutions. Should have overall experience of minimum 6-8 years with minimum 5Years in Hadoop, Spark.Responsibilities:Design & implement robust, scalable, batch & real-time data engineering solutions using Apache Spark (Scala) & Spark structure streaming. Architect well-structured Scala projects using reusable, modular, and testable codebases aligned with SOLID principles and clean architecture principles & practices. Develop, Deploy & Manage Spark jobs on Kubernetes clusters, ensuring efficient resource utilization, fault tolerance, and scalability. Orchestrate data workflows using Apache Airflow — manage DAGs, task dependencies, retries, and SLA alerts. Write and maintain comprehensive unit tests and integration tests for Pipelines / Utilities developed. •Work on performance tuning, partitioning strategies, and data quality validation. Use and enforce version control best practices (branching, PRs, code review) and continuous integration (CI/CD) for automated testing and deployment. Write clear, maintainable documentation (README, inline docs, docstrings). Participate in design reviews and provide technical guidance to peers and junior engineers.Technical Skills: Primary: Languages: Scala, Java Big Data Orchestration: Airflow, Spark on Kubernetes, Yarn, Oozie Big Data Processing: Hadoop, Kafka, Spark & Spark Structured Streaming. Experience on SOLID & DRY principles with Good Software Architecture & Design implementation experience. Advanced Scala experience (e.g. Functional Programming, using Case classes, Complex Data Structures & Algorithms) Proficient in developing automated frameworks for unit & integration testing. Experience with Docker and Helm and related container technologies. Proficient in deploying and managing Spark workloads on Kubernetes clusters. Experience in evaluation and implementation of Data Validation & Data Quality Devops experience in Jenkins, Maven, Github, Github actions, CI/CD Nice To HaveFamiliarity with data governance frameworks and compliance practices. Experience with wide range of Range of Big Data tools, databases & frameworks e.g. Kafka Connect, Apache Arrow, Apache Iceberg, Flink SQL & NoSQL DB: Cassandra, Dremio, Presto Security & Secrets Management like Ansible Vault, Snyk, Git Secrets, Hashi Corp Vault. Qualification6-8 years of proven experience on building large scale distributed systems.Should have led a team in tech lead/module lead role Should have good mentorship experience Should have good communication and very good documentation skills Should show maturity & understand the requirement and convert to high quality technical requirements Should Code and Design end to end data flow and deliver on time Be resilient & flexible to work across multiple teams and internal teams Should help to Implement best practices in Data Architecture and Enterprise Software Development.Should have extensive experience in working in Agile Data Engineering Teams Work very closely with Engineering Manager, TPM, Product Manager and Stake Holders across functions. Be a quick learner and be more proactive in understanding the wider Business requirements and linking to the various other concepts in the Domain. Help implement better solutions independently and faster with better ownership. Help automating manual operational tasks and focus on creating reusable assets and propel innovation. Work very closely with team members and able to have healthy relationship, be innovative and able to come up with ideas and reusable components & frameworks Should be ready to Support 24x7 as per the Rota.If you’re passionate about designing seamless user experiences and want to be part of a dynamic, growing team, we’d love to hear from you.This Role requires candidate to be 3-days in client office. Hence ,candidate should be based in Bangalore or should be ready to relocate within 15 daysInterested Candidates Kindly send your resume along with your design portfolio to ankita@rsquareinfo.com or apply directly via LinkedIn.Note: This is an immediate requirement. So immediate joiner with max 15 days of notice preferred.