Kubernetes Spark Administrator

2 weeks ago


Mumbai, Maharashtra, India timesjobs Full time
Please share your resume at ruchita.-Experience: 4 Years

Location :
Mumbai

Job Description:

Spark Cluster Deployment:
Deploy, configure, and maintain Apache Spark clusters on Kubernetes, ensuring scalability, reliability, and performance.

Application Deployment:
Collaborate with data engineers and data scientists to deploy Spark applications and workloads, ensuring they run efficiently.

Monitoring and Optimization:
Implement monitoring solutions to track cluster performance, resource utilization, and application health. Proactively identify and resolve performance bottlenecks.

Resource Management:
Manage cluster resources, including CPU, memory, and storage allocation, to ensure optimal utilization and cost efficiency.

Security:
Implement and maintain security measures, including authentication, authorization, and encryption, to protect sensitive data and Spark clusters.

Backup and Recovery:
Develop and maintain backup and recovery strategies to ensure data integrity and availability in case of failures.

Documentation:
Maintain clear and comprehensive documentation of Spark cluster configurations, deployment procedures, and best practices.

Troubleshooting:
Quickly diagnose and resolve issues related to Spark clusters, applications, and Kubernetes infrastructure.

Collaboration:

Work closely with cross-functional teams, including data engineers, data scientists, and DevOps, to understand application requirements and optimize Spark clusters accordingly.



Qualifications:

Proven experience deploying and managing Apache Spark on Kubernetes in a production environment.
Proficiency in containerization technologies, particularly Docker and Kubernetes.
Strong knowledge of Spark architecture, including cluster, driver, and worker nodes.
Familiarity with Spark tuning, optimization, and performance monitoring.
Experience with resource management tools like Kubernetes Resource Quotas and LimitRanges.
Understanding of data processing and analytics workflows.
Excellent problem-solving and troubleshooting skills.
Strong communication and collaboration skills.

Preferred

Qualifications:

Experience with Spark cluster orchestration tools like Helm.
Knowledge of Spark ecosystem components such as Spark SQL, Spark Streaming, and MLlib.
Familiarity with cloud-based solutions (e.g., AWS, Azure, Google Cloud).
Certification in Kubernetes (e.g., Certified Kubernetes Administrator CKA).
Knowledge of CI/CD pipelines and infrastructure as code (IaC) tools (e.g., Terraform).
Scripting skills in languages like Python, Bash, or Shell.
Understanding of DevOps practices and automation.

Kindly share your updated resume in word format.
  • Cateina Technologies

    4 weeks ago


    Mumbai, Maharashtra, India Cateina Technologies Full time

    Job Title : Red Hat OpenShift AdministratorJob Summary : As a Red Hat OpenShift Administrator, you will be responsible for designing, deploying, and maintaining Red Hat OpenShift Container Platform environments. Your primary focus will be on ensuring the availability, scalability, and security of containerized applications while collaborating with...

  • Linux Administrator

    4 weeks ago


    Mumbai, Maharashtra, India Careerfit.ai Full time

    Responsibilities :- Install, configure, and maintain Linux operating systems and server infrastructure to ensure high availability and performance of critical systems.- Monitor system performance, identify and troubleshoot issues, and optimize Linux servers to enhance efficiency and reliability.- Manage user accounts, permissions, and access controls on...


  • Mumbai, Maharashtra, India Sampoorna Consultants Pvt. Ltd Full time

    Key Responsibilities : - Design, build, and maintain scalable and efficient data pipelines.- Implement the complete ELT process to enhance our data platform's performance and reliability.- Collaborate with cross-functional teams to gather requirements and deliver high-quality data solutions.- Optimize data flow and collection to improve data accuracy and...

  • Lead Data Engineer

    4 weeks ago


    Mumbai, Maharashtra, India Sampoorna Consultants Pvt. Ltd Full time

    Client is looking out for a technical delivery leader on platform and project builds for building and maintaining optimised and highly available data pipelines that facilitate data management and data analytics solutions.The role will drive the development of data processing frameworks and architectural approaches that scale to handle the business's...


  • Mumbai, Maharashtra, India SAMTA INFOTECH PRIVATE LIMITED Full time

    Job Description :We are seeking a DevOps Engineer with an experience of 2 to 4 years, to join our iNDX team. The individual will be responsible for processes related to application programming, automation, and deployments.This regular, full-time position will be office-based at our office in Powai.Responsibilities :- Strong background in Linux/Unix...


  • Mumbai, Maharashtra, India timesjobs Full time

    Job Title: Platform DevOps EngineerLocation: Mumbai, IndiaEmployment Type: Full TimeWork Arrangement: HybridAt Nielsen, we believe that career growth is a partnership. You ultimately own, fuel and set the journey. By joining our team of nearly 14,000 associates, you will become part of a community that will help you to succeed. We champion you because when...

  • DevOps Engineer

    4 weeks ago


    Mumbai, Maharashtra, India Armour Corporate Services Full time

    Assignment Brief:Job Title : DevOps / Onsite EngineerLocation : Mumbai Reporting to : Enterprise ArchitectNature of Work : Monday to Friday, regular work from office Job Overview:The successful candidate will be responsible for designing, implementing, and maintaining the infrastructure and software systems required to support our development and production...


  • Mumbai, Maharashtra, India Olive Trees Consulting Full time

    Job Description :- Our IT client based in Malad, Mumbai is looking for a Data Mesh Engineer. The main purpose of the role is to design, build, and maintain data pipelines and infrastructure using Microsoft Fabric components. You'll play a crucial role in enabling data mesh architecture within the organization, ensuring scalability, reliability, and...

  • Intern

    3 weeks ago


    Mumbai, Maharashtra, India Cornerstone OnDemand Full time

    In this role you will be...Perform daily system monitoring, verifying the integrity and availability of all hardware, server resources, systems and key processes, reviewing system and application logs, and verifying completion of scheduled jobs such as backups.Research and recommend innovative and automated approaches for system administration tasks.Identify...

  • Software Engineer

    2 weeks ago


    Mumbai, Maharashtra, India Nielsen Full time

    At Nielsen, we believe that career growth is a partnership. You ultimately own, fuel and set the journey. By joining our team of nearly 14,000 associates, you will become part of a community that will help you to succeed. We champion you because when you succeed, we do too. Embark on a new initiative, explore a fresh approach, and take license to think big,...


  • Mumbai, Maharashtra, India Ergode IT Services pvt. ltd. Full time

    Key Responsibilities :- Collaborate with development, operations, and other cross-functional teams to understand infrastructure requirements and implement solutions to meet business needs- Assist in the design, implementation, and maintenance of CI/CD pipelines for software delivery and deployment automation- Configure and manage cloud infrastructure and...


  • Mumbai, Maharashtra, India Mithra Full time

    As Senior DevSecOps Engineer, you will be responsible for DevSecOps platform and solutions for AWS and on-prem environments. You will lead design, estimate, develop and deploy cutting-edge DevSecOps solutions and services to support infrastructure/platform as a service.Your role will also be to ensure that design adequately represents and support the needs...


  • Mumbai, Maharashtra, India Talent Leads HR Solutions Pvt. Ltd. Full time

    Job Description :Skill, Knowledge &Trainings : - Site Reliability Engineer will be responsible to develop and implement services that improve Software development Life Cycle. - Build automations which will help optimize software delivery. - Improve reliability, quality, and time - to - market of our suite of software solutions. - Will be responsible for...

  • DevOps Engineer III

    1 week ago


    Bangalore/Mumbai, Maharashtra, India Xaltworks Consulting Full time

    Job Requirement :What you'll do :- Bridging the gaps b/w core infra, security, QA and development team.- Owning the end-to-end Availability, Performance, Capacity of applications and their infrastructure and creating/maintaining the respective observability with Prometheus/New Relic/ELK/Loki.- Providing 24X7 infra & app support, building processes and...


  • Mumbai, Maharashtra, India ABB Full time

    Digital Software ArchitectTake your next career step at ABB with a global team that is energizing the transformation of society and industry to achieve a more productive, sustainable future.At ABB, we have the clear goal of driving diversity and inclusion across all dimensions: gender, LGBTQ+, abilities, ethnicity and generations. Together, we are embarking...

  • Cloud Data Architect

    4 weeks ago


    Mumbai, Maharashtra, India EDGESOFT Full time

    Job Description : Cloud Data Architect - Data Warehousing/Data Lake for Enterprise Position Overview : We are seeking a talented and experienced Cloud Data Architect to lead the design and implementation of a robust, scalable, and efficient cloud-based data warehousing and data lake solution for our enterprise. As a Cloud Data Architect, you will play a...

  • Akasa Air

    22 hours ago


    Mumbai, Maharashtra, India SNV AVIATION PRIVATE LIMITED Full time

    Role Description :This is a full-time on-site role for a Site Reliability Engineer at Akasa Air located in Mumbai. As a Site Reliability Engineer, you will be responsible for ensuring the reliability and performance of our systems and infrastructure. This includes troubleshooting issues, developing and maintaining software, managing system administration...


  • Mumbai, Maharashtra, India NDX P2P Private Limited Full time

    Job Role - AWS DevOps EngineerJob Description :As an AWS DevOps Engineer, you will work with development teams daily to address their infra concerns and deploy appropriate AWS infra solutions. You will also continuously monitor services and perform necessary infra upgrades. You will work as an independent contributor ready to take on new challenges and...


  • Bangalore/Mumbai, Maharashtra, India Growel Softech Pvt. Ltd. Full time

    JD : Roles and responsibilities :- Maintain and configure development tools such as CI/CD, environment automation, code quality, and testing tools- Design, engineer, and support Adobe Target products and necessary infrastructure.- Install/configure/maintenance of Adobe Target Infrastructure- Installation/configuration/maintenance of Apache web server and...


  • Mumbai, Maharashtra, India Capabiliq IT Services Private Limited Full time

    Position Overview : We are seeking a talented and experienced Technical Architect (Staff/Principal Engineer- Application) to join our dynamic team. The ideal candidate will work closely with an Engineering Manager and be responsible for one or more software products. The Technical Architect will provide leadership, architectural guidance, and technical...