Big Data Manager

2 weeks ago

Noida, Uttar Pradesh, India TalentXo Full time

Job Summary :

We are looking for an experienced and highly skilled Big Data Engineer to join our data engineering team. In this role, you will be responsible for designing, developing, and maintaining scalable big data solutions to support data-driven decision-making across the organization. You should have strong experience working with distributed systems, cloud-based platforms, and big data technologies, with a keen focus on performance, security, and scalability.

Key Responsibilities :

- Design and implement scalable and efficient big data pipelines using Apache Spark and Hadoop.

- Develop, test, and maintain data integration workflows using AWS services such as EMR, Glue, S3, Athena, IAM, Lambda, and CloudWatch.

- Optimize and manage data processing applications using Spark Core, Spark SQL, Spark Streaming, and RDD APIs to process large-scale data sets (terabytes).

- Perform advanced SQL queries and performance tuning using Hive and Impala frameworks.

- Design and implement physical table structures in big data environments to support efficient data access and processing.

- Leverage ElasticSearch (OpenSearch) and Kibana to build interactive dashboards and monitor data pipeline performance.

- Utilize resource management tools such as YARN or Mesos for efficient job execution in distributed environments.

- Automate and orchestrate workflows using tools such as Airflow, AWS Data Pipeline, and Autosys.

- Work with key/value data stores like HBase for real-time and batch processing.

- Collaborate with data scientists, analysts, and other stakeholders to ensure data quality, availability, and usability.

- Ensure best practices in security, reliability, and governance of big data infrastructure.

Required Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or a related technical field.

- A minimum of 5 years of experience in Big Data Engineering or Data Analysis.

- Proficiency in Python, SQL, and Apache Spark.

- Strong understanding of Hadoop and other distributed computing frameworks.

Preferred Skills and Experience :

- Hands-on experience with AWS services such as EMR, Glue (serverless), S3, Athena, IAM, Lambda, and CloudWatch.

- Expertise in Spark Streaming, DataFrame API, Dataset API, and advanced Spark SQL.

- Advanced SQL tuning experience using Hive/Impala.

- Experience building and managing Kibana Dashboards integrated with ElasticSearch (OpenSearch).

- Familiarity with resource management tools like Yarn or Mesos.

- Experience with job scheduling tools such as Autosys, AWS Data Pipeline, or Airflow.

- Exposure to key/value data stores such as HBase.

- Solid understanding of data security, performance optimization, and big data table design principles.

(ref:hirist.tech)

  • Big Data Architect

    5 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    About the PositionWe are seeking an experienced Big Data Architect to design and develop our company's data infrastructure. The ideal candidate will have a strong background in big data and experience with data architecture.Key ResponsibilitiesDesigning and developing a scalable data architecture to meet business needs;Developing and implementing data...

  • Big Data Architect

    1 week ago

    Noida, Uttar Pradesh, India Infogain Full time

    Job DescriptionWe are seeking a skilled Data Engineer to join our team at Infogain, a leading digital platform and software engineering company. As a Data Engineer, you will play a crucial role in designing, building, and maintaining large-scale data processing systems.Key Responsibilities:Design and implement data pipelines using Apache Spark, Azure Data...

  • AWS Big Data Architect

    2 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    Big Data ArchitectureWe are seeking an experienced AWS Big Data Architect to design and implement scalable and efficient data architectures that meet business needs. The ideal candidate will have a strong understanding of big data tools, cloud services, and data engineering principles.Key Responsibilities:• Design and develop data pipelines that integrate...

  • Senior Big Data Professional

    16 hours ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    Job Description:We are seeking an experienced Big Data professional to help us develop and maintain our large-scale data processing systems.The ideal candidate will have a strong background in Spark Core, Spark-SQL, Scala-programming, and streaming datasets. Proficiency in handling complex big data platforms is essential for this role.Demonstrated expertise...

  • Big Data Specialist

    5 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    Job OverviewWe are seeking an experienced Big Data Specialist to design, implement, and optimize big data pipelines in Databricks. The ideal candidate will have a strong background in data engineering and be proficient in Apache Spark for distributed data processing and real-time analytics.The selected candidate will collaborate with data scientists,...

  • Big Data Specialist

    4 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    We are seeking a highly skilled Big Data Specialist to manage and maintain large-scale data processing environments.Main Responsibilities:Maintain and manage Hadoop clusters using tools like IBM BigInsights, Cloudera/Hortonworks Ambari, Ganglia, etc.Tune performance of Hadoop clusters and MapReduce routines for optimal efficiency.Expertise in working with...

  • Big Data Solutions Developer

    8 hours ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    Job DescriptionWe are seeking a talented Big Data Solutions Developer to join our team. As a key member of our big data team, you will design, develop, and deploy big data solutions using Hadoop, Hive, and Spark technologies.Key ResponsibilitiesDesign and develop scalable and secure big data solutions using Hadoop, Hive, and Spark technologies.Implement data...

  • Big Data Transformation Expert

    2 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    Salary\We are looking for a big data transformation expert to join our team. As a big data transformation expert, you will be responsible for leading end-to-end data migration projects, ensuring data integrity, and collaborating with cross-functional teams to align data migration processes with business objectives.\Job Description\The big data transformation...

  • Big Data Solutions Architect

    4 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    Job Title: Big Data Solutions ArchitectAbout the PositionWe are looking for a seasoned Big Data Solutions Architect to lead our team in designing and implementing big data solutions using Hadoop services like Hive, Spark, Scala, and Sqoop. The ideal candidate will have a deep understanding of big data technologies and be able to guide team members in cloud...

  • Big Data Engineer

    5 days ago

    Noida, Uttar Pradesh, India beBee Careers Full time

    About the RoleWe are seeking a seasoned professional to lead our Big Data Engineering efforts. The ideal candidate will possess in-depth knowledge of Hadoop, Spark, and their ecosystem components.Key ResponsibilitiesDesign, develop, and maintain large-scale big data architectures using Hadoop and Spark.Collaborate with cross-functional teams to implement...