Softility - Big Data Engineer - Apache Spark

7 days ago

Hyderabad, India Softility Full time

Big Data Engineer :- Design, build, and optimize robust and scalable data pipelines for both batch and real-time data ingestion using Apache Spark (Streaming + Batch), Apache Nifi, and Apache Kafka.Data Storage and Management :- Manage and maintain data storage solutions on Hadoop/HDFS. Implement data models and schemas in Hive for a data warehouse and reporting layer. - Work with HBase for specific use cases requiring fast, random-access to large datasets, leveraging Phoenix/SQL line for SQL-based access.- Workflow Orchestration: Develop and manage complex data workflows and dependencies using Apache Oozie.ETL and Data Integration : - Utilize Informatica for traditional ETL workflows and Apache Sqoop to efficiently transfer data between RDBMS and the Hadoop ecosystem.Resource Management : - Work with YARN to manage cluster resources, monitor job execution, and ensure high availability and fault tolerance.Monitoring and Maintenance : - Monitor the health and performance of the data platform using tools like Hue and New Relic. Proactively identify and resolve issues.Collaboration : - Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.Cloud and Advanced Skills (Good to Have) :- Experience with containerization and cloud-native solutions, particularly Anthos, for deploying and managing applications.- Familiarity with data observability and logging platforms like Cribl for advanced data collection and routing.Qualifications :Experience : Proven experience as a Big Data Engineer or a similar role.Technical Skills :- Strong expertise in Hadoop and HDFS.- Proficiency in Apache Spark for both batch and stream processing.- Hands-on experience with Apache Hive and HBase.- Knowledge of data ingestion tools like Apache Kafka and Apache Nifi.- Experience with Apache Oozie or other workflow schedulers.- Familiarity with Apache Sqoop for RDBMS integration.- Understanding of YARN for cluster resource management.- Proficiency in at least one scripting language (e.g., Python, Scala).- Familiarity with monitoring tools like Hue and New Relic.- Experience with Informatica is a plus.Soft Skills :- Excellent problem-solving and analytical skills.- Strong communication and collaboration abilities.- Ability to work in a fast-paced, agile environment.Education :- Bachelors or Masters degree in Computer Science, Data Science, or a related field. (ref:hirist.tech)

  • Softility - Java Architect - Microservices Architecture/Spring Boot

    5 days ago

    Hyderabad, India Softility Full time

    Job Title : Java ArchitectExperience : 12 to 20 yearsLocation : HyderadadCompany : SoftilityAbout the Role :Softility is seeking a highly skilled and experienced Java Architect to join our innovative team. This role is perfect for an expert in Core Java and modern microservices frameworks who is passionate about delivering high-quality, scalable, and...

  • Apache Spark Admin

    2 weeks ago

    Madhapur, Hyderabad, Telangana, India Glansa Solutions Full time

    An Apache Spark administrator's responsibilities include: - **Designing and implementing Spark jobs**: Defining, scheduling, monitoring, and controlling processes - **Optimizing Spark jobs**: Maximizing speed and scalability while remaining data-use compliant - **Managing data pipelines**: Managing data pipelines and acquisition processes - **Performing...

  • Java Architect

    5 days ago

    Hyderabad, Telangana, India Softility Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title : Java ArchitectExperience : 12 to 20 years Location : Hyderadad Company : SoftilityAbout the Role : Softility is seeking a highly skilled and experienced Java Architect to join our innovative team. This role is perfect for an expert in Core Java and modern microservices frameworks who is passionate about delivering high-quality, scalable,...

  • Data Engineer

    4 weeks ago

    Hyderabad, India BOHIYAANAM TECHNOLOGY LLP Full time

    Description :Key Responsibilities :- Design, develop, and deploy end-to-end data engineering solutions using Databricks, Apache Spark, PySpark, Python, and SQL.- Build scalable and efficient ETL/ELT pipelines for data ingestion, transformation, and integration from various sources.- Work with data warehousing solutions and ensure high-performance data...

  • Big Data Operations Engineer

    3 days ago

    Hyderabad, Telangana, India S M Software Solutions Inc Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Description – Big Data Operations EngineerLocation :Onsite, Hyderabad SoftilityExperience: 5+ yearsRole OverviewWe are looking for a skilledBig Data Operations Engineerwith hands-on experience in supporting and maintaining large-scale data platforms. The role involves ensuring platform availability, managing incidents, monitoring performance through...

  • Grid Dynamics

    4 weeks ago

    Hyderabad, India GRID DYNAMICS PRIVATE LIMITED Full time

    Description : Big Data Functions : - Design and implement data pipelines for migration from HDFS/Hive to cloud object storage (e.g., S3, Ceph). - Optimize Spark (and optionally Flink) jobs for performance and scalability in a Kubernetes environment.- Ensure data consistency, schema evolution, and governance with Apache Iceberg or equivalent table formats. -...

  • Big Data Engineer

    4 weeks ago

    Hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data Engineer 📍 Locations: Hyderabad | Chennai | Pune | Bangalore 🛠 Skills: Scala, Python, Apache Spark, PySpark, AWS, CI/CD 📈 Experience: 3 to 11 years 🔹 Scala Developer 📍 Multiple Locations 🛠 Strong hands-on experience in Scala & distributed data processing

  • Big Data Engineer

    1 week ago

    Hyderabad, Telangana, India Techno Facts Solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Experience in developing and delivering scalable big data pipelines using Apache Spark and Databricks on AWS.Position Requirements :Must Have : Build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimize ETL/ELT processes for structured and unstructured data. Knowledge of Lakehouse architecture for efficient data...

  • Big data engineer

    4 weeks ago

    Hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data EngineerLocations: Hyderabad | Chennai | Pune | BangaloreSkills: Scala, Python, Apache Spark, Py Spark, AWS, CI/CDExperience: 3 to 11 yearsScala DeveloperMultiple LocationsStrong hands-on experience in Scala & distributed data processing

  • Big Data Engineer

    4 weeks ago

    Hyderabad, India Rapinno Tech Solutions Pvt.Ltd. Full time

    Big Data Engineer 📍 Locations: Hyderabad | Chennai | Pune | Bangalore 🛠 Skills: Scala, Python, Apache Spark, PySpark, AWS, CI/CD 📈 Experience: 3 to 11 years🔹 Scala Developer 📍 Multiple Locations 🛠 Strong hands-on experience in Scala & distributed data processing