Senior Big Data Engineer for Large-Scale Hadoop Cluster

6 days ago

Hyderabad, Telangana, India Aricent Full time

Job Description

We are seeking a skilled Senior Big Data Engineer to design and build solutions for a Fortune 500 client program. The project involves creating a data standardized and curation-based Hadoop cluster.

About the Role

This is a high-visibility, fast-paced key initiative that will integrate data across internal and external sources, provide analytical insights, and integrate with critical customer systems. You will be responsible for designing, building, and unit testing applications on the Spark framework using Python.

Key Responsibilities:

  • Design and build PySpark based applications for batch and streaming requirements, utilizing in-depth knowledge of Hadoop and NoSQL databases.
  • Develop and execute data pipeline testing processes to validate business rules and policies.
  • Optimize performance of built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDDs.
  • Choose appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec for optimized data access requirements.
  • Design and build real-time applications using Apache Kafka Spark Streaming.
  • Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
  • Develop data tokenization libraries and integrate with Hive Spark for column-level obfuscation.
  • Process large amounts of structured and unstructured data, including integrating data from multiple sources.
  • Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories.
  • Participate in agile development process, documenting and communicating issues and bugs relative to data standards in scrum meetings.
  • Work collaboratively with onsite and offshore teams.
  • Develop technical documentation for artifacts delivered.
  • Solve complex data-driven scenarios and triage towards defects and production issues.
  • Learn, unlearn, and relearn concepts with an open and analytical mindset.
  • Participate in code release and production deployment.
  • Challenge and inspire team members to achieve business results in a fast-paced environment.

Requirements:

  • Bachelor's degree in Computer Science, Statistics, or related field from an accredited college or university.
  • Minimum 3 years of extensive experience in designing, building, and deploying PySpark-based applications.
  • Expertise in handling complex large-scale Big Data environments preferably (20Tb+).
  • Minimum 3 years of experience in HIVE, YARN, HDFS preferably on Hortonworks Data Platform.
  • Good implementation experience of OOPS concepts.
  • Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
  • Able to build abstracted, modularized reusable code components.
  • Hands-on experience generating/parsing XML, JSON documents, and REST API request/responses.

Estimated Salary Range:

$120,000 - $180,000 per annum.

  • Big Data Cluster Architect

    3 weeks ago

    Hyderabad, Telangana, India Medtronic Full time

    Transforming Data into InsightsWe are seeking an experienced Senior Cloud Data Engineer to join our PCS Software Solutions team at Medtronic. As a key member of our team, you will be responsible for the design, implementation, and management of our Big Data infrastructure, which leverages HBase on Hortonworks and Cloudera Data Platform (CDP).About the...

  • Senior Big Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Vidpro Consultancy Services Full time

    Job Title: Senior Big Data EngineerJob Summary: Vidpro Consultancy Services is seeking a highly skilled Senior Big Data Engineer to join our team. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining large-scale big data architectures using Java, Python, and Hadoop technologies.Responsibilities:Design...

  • Senior Big Data Developer

    3 days ago

    Hyderabad, Telangana, India Virtusa Full time

    Virtusa seeks a seasoned Senior Big Data Developer to spearhead the development of large-scale data processing systems using Hadoop and Spark. As a key member of our team, you will design, implement, and deploy complex big data solutions that drive business growth and innovation.Job SummaryWe are looking for an experienced Senior Big Data Developer with a...

  • Senior Big Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Findernest Software Services Pvt Ltd Full time

    About the RoleWe are seeking a highly skilled Senior Big Data Engineer to join our team at Findernest Software Services Pvt Ltd. As a Senior Big Data Engineer, you will be responsible for designing, developing, and maintaining scalable big data solutions that drive business growth and decision-making.Key ResponsibilitiesDesign and implement big data...

  • Senior Big Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India Nisum Full time

    Job Title: Senior Big Data EngineerAbout the Role:We are seeking a highly skilled Senior Big Data Engineer to join our team at Nisum. As a key member of our engineering team, you will be responsible for designing, building, and maintaining large-scale data processing systems using Hadoop, Python, and other big data technologies.Key Responsibilities:- Design...

  • Big Data Engineer with Hadoop and Spark Expertise

    1 week ago

    Hyderabad, Telangana, India Virtusa Full time

    About the RoleWe are seeking a highly skilled Big Data Engineer to join our team at Virtusa. As a key member of our engineering team, you will design, develop, and maintain large-scale data processing pipelines using big data technologies such as Hadoop, MapReduce, Hive, and Spark.ResponsibilitiesDesign and implement data processing pipelines using Python...

  • Big Data Engineer, Data Modeling Expert

    1 month ago

    Hyderabad, Telangana, India data Full time

    Unlock the Power of Data with UsWe're seeking an experienced Big Data Engineer to join our team and contribute to the development of innovative analytics and data products. As a key member of our engineering team, you will work closely with Data Scientists, Data Engineers, Product Managers, and Software Architects to design and implement complex data product...

  • Senior Big Data Engineer

    1 month ago

    Hyderabad, Telangana, India Virtusa Full time

    Job Title: Senior Big Data EngineerJob Summary:We are seeking a Senior Big Data Engineer to join our team at Virtusa. The ideal candidate will have a strong background in designing, building, and supporting data applications using Spark, Sqoop, and Hive.Key Responsibilities:Design and develop complex data applications using Spark, Sqoop, and Hive.Work with...

  • Big Data Systems Engineer

    3 weeks ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Data EngineerJob Summary:We are seeking a skilled Data Engineer to design, build, and maintain large-scale data systems. The ideal candidate will have expertise in big data ecosystems, data processing, and data storage.Key Responsibilities:Design and develop scalable data systems for business applications.Work closely with data scientists to...

  • Big Data Engineer

    1 month ago

    Hyderabad, Telangana, India Vidpro Consultancy Services Full time

    About the Role:Vidpro Consultancy Services is seeking a highly skilled Big Data Engineer to join our team. As a key member of our data architecture team, you will be responsible for designing and implementing large-scale Big Data architectures using Java and Python.Key Responsibilities:Design and implement Big Data architectures using Hadoop and SparkDevelop...

  • Big Data Engineer

    1 month ago

    Hyderabad, Telangana, India Nisum Full time

    About the RoleWe are seeking a highly skilled Big Data Engineer to join our team at Nisum. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems.Key ResponsibilitiesAnalyze complex business requirements and generate technical specifications to design or redesign complex software...

  • Big Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India data Full time

    Overview: We're seeking a skilled Big Data Engineer to join our team at data.ai. As a key member of our engineering team, you will be responsible for designing and implementing innovative data solutions that drive business growth. If you're passionate about working with large data sets, have experience with big data technologies, and are eager to...

  • Big Data Engineer

    1 month ago

    Hyderabad, Telangana, India TechSophy Full time

    Job Title: Big Data EngineerJob Summary:We are seeking a highly skilled Big Data Engineer to join our team at TechSophy. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems.Key Responsibilities:Analyze large data sets to identify errors in the map.Design and implement complex...

  • Big Data Engineer for Ivy Comptech

    3 days ago

    Hyderabad, Telangana, India Ivy comptech Full time

    About the RoleIvy Comptech is seeking a highly skilled Big Data Engineer to join our team in Hyderabad, India. As a Senior Data Analyst, you will play a crucial role in various phases and activities of the software development process or software testing.Key Responsibilities:Strong understanding of Apache Druid and Apache Hadoop Architecture.Ability to learn...

  • Big Data and Cloud Infrastructure Specialist

    1 month ago

    Hyderabad, Telangana, India Experian Full time

    Job Title: Big Data and Cloud Infrastructure SpecialistJob Summary:We are seeking a highly skilled Big Data and Cloud Infrastructure Specialist to join our team at Experian. The ideal candidate will have a deep understanding of Linux, networking fundamentals, and security, as well as experience working with AWS cloud platform and infrastructure.Key...

  • Data Engineer for Large-Scale Big Data Solutions

    3 days ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    We are seeking an experienced data engineer to join our team at Alignity Solutions in Mumbai. The ideal candidate will have expertise in Apache Spark and Python programming, focusing on building scalable, high-performance data processing pipelines.Job Description:The role of a PySpark Developer involves developing, testing, and maintaining large-scale data...

  • Linux and Big Data Infrastructure Manager

    3 weeks ago

    Hyderabad, Telangana, India Experian Full time

    Job Description:We are seeking an experienced professional to join our team as a Linux and Big Data Infrastructure Manager. The ideal candidate will have a deep understanding of Linux, networking fundamentals, and security, as well as experience working with AWS cloud platform and infrastructure.Key Responsibilities:Manage large Big Data clusters in...

  • GCP Big Data Engineer

    3 weeks ago

    Hyderabad, Telangana, India Tiger Analytics Full time

    About the Role:Tiger Analytics is looking for a Senior Data Engineer to join our Chennai office. As a Senior Data Engineer, you will be responsible for designing and building solutions for near real-time stream processing and batch processing on the Big Data platform.Responsibilities:Design and build solutions for near real-time stream processing and batch...

  • Big Data Engineer

    2 months ago

    Hyderabad, Telangana, India TECHSTAR SOFTWARE DEVELOPMENT INDIA PVT LTD Full time

    Responsibilities of the Candidate :- Be responsible for the design and development of big data solutions. - Partner with domain experts, product managers, analysts, and data scientists to develop Big Data pipelines in Hadoop- Be responsible for moving all legacy workloads to a cloud platform- Work with data scientists to build Client pipelines using...

  • Big Data Engineer

    3 weeks ago

    Hyderabad, Telangana, India Mempage Full time

    About the Role : We are seeking a highly skilled Big Data Engineer to contribute to the design, development, and maintenance of our data infrastructure. As a key member of our team, you will be responsible for building and optimizing data pipelines to support our business analytics and machine learning initiatives. Your expertise will be instrumental in...