Softility - Big Data Engineer - Apache Spark

2 weeks ago

Hyderabad, India Softility Full time

Big Data Engineer :


- Design, build, and optimize robust and scalable data pipelines for both batch and real-time data ingestion using Apache Spark (Streaming + Batch), Apache Nifi, and Apache Kafka.

Data Storage and Management :


- Manage and maintain data storage solutions on Hadoop/HDFS. Implement data models and schemas in Hive for a data warehouse and reporting layer.


- Work with HBase for specific use cases requiring fast, random-access to large datasets, leveraging Phoenix/SQL line for SQL-based access.

- Workflow Orchestration: Develop and manage complex data workflows and dependencies using Apache Oozie.

ETL and Data Integration :


- Utilize Informatica for traditional ETL workflows and Apache Sqoop to efficiently transfer data between RDBMS and the Hadoop ecosystem.

Resource Management :


- Work with YARN to manage cluster resources, monitor job execution, and ensure high availability and fault tolerance.

Monitoring and Maintenance :


- Monitor the health and performance of the data platform using tools like Hue and New Relic. Proactively identify and resolve issues.

Collaboration :


- Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.

Cloud and Advanced Skills (Good to Have) :

- Experience with containerization and cloud-native solutions, particularly Anthos, for deploying and managing applications.

- Familiarity with data observability and logging platforms like Cribl for advanced data collection and routing.

Qualifications :

Experience : Proven experience as a Big Data Engineer or a similar role.

Technical Skills :

- Strong expertise in Hadoop and HDFS.

- Proficiency in Apache Spark for both batch and stream processing.

- Hands-on experience with Apache Hive and HBase.

- Knowledge of data ingestion tools like Apache Kafka and Apache Nifi.

- Experience with Apache Oozie or other workflow schedulers.

- Familiarity with Apache Sqoop for RDBMS integration.

- Understanding of YARN for cluster resource management.

- Proficiency in at least one scripting language (e.g., Python, Scala).

- Familiarity with monitoring tools like Hue and New Relic.

- Experience with Informatica is a plus.

Soft Skills :

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration abilities.

- Ability to work in a fast-paced, agile environment.

Education :

- Bachelors or Masters degree in Computer Science, Data Science, or a related field.


(ref:hirist.tech)

  • Big Data Developer

    6 days ago

    Hyderabad, Telangana, India Softility, Inc. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Company DescriptionSoftility is a digital transformation technology services company specializing in innovative solutions across cloud adoption and modernization, data engineering and analytics, AI-driven operations, digital service management, and cybersecurity. Based in Herndon, Virginia, USA, with an Innovation & Development Center in Hyderabad, India, we...

  • Big Data Engineer

    6 days ago

    Hyderabad, Telangana, India Softility, Inc. Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Company DescriptionSoftility is a digital transformation technology services company specializing in cloud adoption, data engineering, AI-driven operations, digital service management, and cybersecurity. We are the preferred partner for many Fortune 500 companies, SLED entities, healthcare institutions, major enterprises, and federal agencies. Headquartered...

  • Spark Data Engineer

    1 day ago

    Bengaluru, Hyderabad, Pune, India KPI Partners Full time US$ 80,000 - US$ 1,50,000 per year

    KPI Partners is a leading provider of analytics and data solutions that help businesses leverage their data for strategic decision-making. We are committed to delivering innovative solutions and exceptional service to our clients. Job Summary: We are seeking a skilled Spark Data Engineer to join our dynamic team. The ideal candidate will have a strong...

  • Apache Spark

    3 days ago

    Bengaluru, Chennai, Hyderabad, India Advent Global Solutions Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    *Please read complete JD if matching only apply and #Whatsappme *Must have skills : Apache SparkGood to have skills : NAMinimum 5 year(s) of experience is requiredEducational Qualification : 15 years full time educationSummary: As an Application Developer, you will be involved in designing, building, and configuring applications to meet business process and...

  • Big Data Engineer

    1 week ago

    Hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data Engineer 📍 Locations: Hyderabad | Chennai | Pune | Bangalore 🛠 Skills: Scala, Python, Apache Spark, PySpark, AWS, CI/CD 📈 Experience: 3 to 11 years 🔹 Scala Developer 📍 Multiple Locations 🛠 Strong hands-on experience in Scala & distributed data processing

  • Big Data Engineer

    7 days ago

    Hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data EngineerLocations: Hyderabad | Chennai | Pune | BangaloreSkills: Scala, Python, Apache Spark, PySpark, AWS, CI/CDExperience: 3 to 11 yearsScala DeveloperMultiple LocationsStrong hands-on experience in Scala & distributed data processing

  • Big Data Developer

    1 week ago

    Hyderabad, India Sourcingxpress Full time

    Company : : Visit Type : Consulting FirmCompany Type : Model : Stage : : IT Description : We are looking for an experienced Senior Big Data Engineer (Scala/Spark) with 8+ years of expertise in designing and developing large-scale, distributed data processing pipelines. The role requires strong knowledge of Scala, Apache Spark (Core, SQL,...

  • Senior Data Engineer

    3 days ago

    Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture,Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 15th 2025)Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and...

  • Big Data Engineer

    1 week ago

    hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data Engineer 📍 Locations: Hyderabad | Chennai | Pune | Bangalore 🛠 Skills: Scala, Python, Apache Spark, PySpark, AWS, CI/CD 📈 Experience: 3 to 11 years🔹 Scala Developer 📍 Multiple Locations 🛠 Strong hands-on experience in Scala & distributed data processing

  • Big Data Engineer

    1 week ago

    Hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data Engineer 📍 Locations: Hyderabad | Chennai | Pune | Bangalore 🛠 Skills: Scala, Python, Apache Spark, PySpark, AWS, CI/CD 📈 Experience: 3 to 11 years🔹 Scala Developer 📍 Multiple Locations 🛠 Strong hands-on experience in Scala & distributed data processing