Apache Spark Administrator

1 week ago

India Marlabs Full time

**Date**:Oct 11, 2024**Location**:Any Marlabs Office Location, IN**Company**:Marlabs Innovations Pvt Ltd**Description**:

- Spark Administrator - 6 Weeks (Starting November 1st 2024)- Key Responsibilities- Install and configure the operating system (Linux distributions such as Ubuntu, CentOS, or RHEL) on cluster nodes, ensuring consistency across the environment.
- Set up networking for Spark clusters, including IP addressing, DNS configuration, and ensuring communication between cluster nodes.
- Configure firewall rules and VPNs for secure communication between cluster nodes and client machines.
- Install and configure Apache Spark and its dependencies (Java, Scala, Hadoop components) on all cluster nodes.
- Set up cluster management components, such as YARN or Mesos, to efficiently manage cluster resources.
- Configure cluster nodes for optimal performance, including tuning JVM parameters, setting up swap memory, and managing CPU and memory resources.
- Configure cluster security, including SSL/TLS for encrypted communication, Kerberos authentication, and user permissions.
- Perform cluster testing to ensure successful communication between nodes and efficient job execution.
- Design and implement network architecture, ensuring high availability and failover capabilities for the cluster.
- Monitor network performance and ensure optimal bandwidth utilization between cluster nodes.

  • Java developer with Spark

    2 days ago

    Hyderabad, India VHL Technologies Inc Full time

    Job Description Looking for immediate joiners, 30 days or less 3-45 years experince Key Requirements: Experience with the following technologies: Java, REST APIs, and Apache Spark Practical expertise in building and managing streaming applications Proven track record in developing microservices and integrating with batch data sources Strong hands-on...

  • Senior Database Administrator

    2 days ago

    India CES Full time

    We are seeking a highly skilled and proactive Senior Database Administrator to join our team. This hybrid role blends traditional DBA responsibilities with modern data engineering tasks to support our Comparative data loads and ensure optimal performance of critical database systems. You’ll play a key role in scaling our data infrastructure, diagnosing...

  • Senior Database Administrator

    2 weeks ago

    India CES Full time

    We are seeking a highly skilled and proactive Senior Database Administrator to join our team. This hybrid role blends traditional DBA responsibilities with modern data engineering tasks to support our Comparative data loads and ensure optimal performance of critical database systems. You'll play a key role in scaling our data infrastructure, diagnosing...

  • Senior Database Administrator

    3 days ago

    India CES Full time

    We are seeking a highly skilled and proactive Senior Database Administrator to join our team. This hybrid role blends traditional DBA responsibilities with modern data engineering tasks to support our Comparative data loads and ensure optimal performance of critical database systems. You’ll play a key role in scaling our data infrastructure,...

  • Senior Database Administrator

    2 days ago

    India CES Full time

    We are seeking a highly skilled and proactive Senior Database Administrator to join ourteam. This hybrid role blends traditional DBA responsibilities with modern dataengineering tasks to support our Comparative data loads and ensure optimalperformance of critical database systems. You’ll play a key role in scaling our datainfrastructure, diagnosing...

  • Senior Database Administrator

    1 day ago

    India CES Full time

    We are seeking a highly skilled and proactive Senior Database Administrator to join our team. This hybrid role blends traditional DBA responsibilities with modern data engineering tasks to support our Comparative data loads and ensure optimal performance of critical database systems. You’ll play a key role in scaling our data infrastructure, diagnosing...

  • Data Cloud Solutions Architect

    6 days ago

    India beBeeData Full time ₹ 15,00,000 - ₹ 28,00,000

    Cloud Data Solutions ArchitectThe ideal candidate for this role will be an experienced and results-driven Tech Lead with a strong background in Java and Apache Spark to oversee the development and deployment of cloud-based data solutions.Key Responsibilities:Design, develop, and deploy scalable and reliable data processing systems using Java and Apache Spark...

  • Data Migration Lead

    4 days ago

    India LTIMindtree Full time

    Job Description: We are looking for an experienced and results-driven Tech Lead with a strong background in Java and Apache Spark to lead the development and deployment of cloud-based data solutions. This role involves overseeing the technical direction of projects, mentoring team members, and delivering scalable and efficient systems using cloud and big...

  • Senior Data Infrastructure Specialist

    2 weeks ago

    India beBeeDatabase Full time ₹ 18,00,000 - ₹ 26,00,000

    Job Overview:We are seeking a highly skilled and proactive Senior Database Administrator to enhance our data infrastructure. This role blends traditional DBA responsibilities with modern data engineering tasks to support comparative data loads and ensure optimal performance of critical database systems.You'll play a key role in scaling our data architecture,...

  • Data Engineering Lead

    1 week ago

    India beBeeDataEngineering Full time ₹ 8,00,000 - ₹ 12,00,000

    Job Title: Data Engineering LeadJob DescriptionWe are seeking an experienced Data Engineering Lead to lead our data engineering team in setting up and implementing a data lake on cloud, as well as developing standardized data models. The ideal candidate will have experience with big data technologies and be able to develop scalable data pipelines.In this...