Big Data Engineer

3 weeks ago

Delhi, India Pert Telecom Solutions Full time
Job Description : We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for this role.
Key Responsibilities:
Develop and implement data ingestion pipelines and ETL processes using PySpark and Python capable of managing and processing large volumes of data, ranging in the millions of records.
Design and develop APIs using Python to facilitate data integration and processing.
Deploy, configure, and maintain Hadoop ecosystem components including HDFS , YARN, Hive, and HBase in on-premises environments.
Optimize the performance of Big Data applications and queries, ensuring efficient resource utilization.
Manage and monitor cluster health , including tuning and troubleshooting performance issues.
Ensure data security , integrity , and availability within the Big Data infrastructure.
Collaborate with cross-functional teams to integrate data analytics solutions and support business requirements.
Required Skills and Qualifications:
Bachelor’s degree in computer science, Engineering, or a related field;
Proven experience 3+ years working with Big Data technologies such as Hadoop, PySpark, and MySQL in on-premises environments and handling large-scale data volumes, including datasets with millions of records, in on-premises environments.
Strong programming skills in Python, with experience in developing APIs.
Hands-on experience in Linux.
Expertise in deploying, managing, and optimizing Big Data clusters in a local data center.
Proficiency in monitoring and performance tuning of Hadoop and PySpark clusters.
Ability to troubleshoot and resolve complex issues related to data processing and infrastructure.
Excellent problem-solving skills and the ability to work independently as well as part of a team.
Strong communication skills with the ability to convey technical concepts to non-technical stakeholders.
Experience in dockers
Preferred Skills:
Experience with Apache Airflow, Kafka, Apache Flink, and Apache NiFi for data orchestration and streaming.
Certification in Hadoop or related Big Data technologies.
Application Process: Interested candidates should submit their resume and a cover letter outlining their qualifications and experience in Big Data engineering and on-premises cluster management and include details of at least one relevant project involving the handling of large data volumes (in millions of records), you have worked on, including a brief description of the project, your role and the technologies used. This is mandatory requirement for our screening process .

  • Big Data Engineer

    3 weeks ago

    delhi, India Pert Telecom Solutions Full time

    Job Description : We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for...

  • Big data engineer

    3 weeks ago

    Delhi, India Pert Telecom Solutions Full time

    Job Description : We are seeking a skilled Big Data Engineer with deep expertise in Py Spark, Python, My SQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential...

  • Big Data Engineer

    3 weeks ago

    delhi, India Pert Telecom Solutions Full time

    Job Description:We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for this...

  • Big Data Engineer

    3 weeks ago

    Delhi, India Pert Telecom Solutions Full time

    Job Description:We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for this...

  • Data Warehouse Engineer | Data Warehouse Engineer | delhi

    1 week ago

    delhi, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Data Warehouse Engineer

    1 week ago

    Delhi, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Big Data engineer

    18 hours ago

    Delhi, India Tata Consultancy Services Full time

    Role ; Big Data engineer Location : Pune, Chennai, MumbaiExp : 3-6 YearRequired Technical Skill Set** Hadoop, Python, PySpark, HIVEMust-Have**· Hands-on experience of Hadoop, Python, PySpark, Hive, Big Data Eco System Tools. · Should be able to develop, tweak queries and work on performance enhancement. · Solid understanding of object-oriented...

  • Big Data Automation Engineer

    15 hours ago

    Delhi, Delhi, India PubMatic Full time

    Job DescriptionWe are seeking a highly skilled Big Data Automation Engineer to join our team at PubMatic. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining automation frameworks for big data workflows.Responsibilities:Designing and implementing automation frameworks using Python to automate big data...

  • Big Data Engineer

    2 weeks ago

    Delhi, Delhi, India Wipro Full time

    We are seeking a highly skilled Big Data Engineer to join our team at Wipro. As a Big Data Engineer, you will play a key role in designing, developing, and maintaining large-scale data processing systems.The ideal candidate will have a strong background in Hadoop and related technologies such as Scala. You will work closely with cross-functional teams to...

  • Data Engineer for Big Data Solutions

    3 days ago

    Delhi, Delhi, India LTIMindtree Full time

    We are seeking an experienced Data Engineer to join our team at LTIMindtree. As a key member of our big data solutions group, you will be responsible for designing and developing scalable data platforms using modern technologies like Spark, HBase, Hive, and other Hadoop ecosystem tools.About the RoleIn this role, you will work closely with cross-functional...

  • Big Data Engineer

    1 month ago

    Delhi, Delhi, India Talent Pro Full time

    Hadoop Developer Job DescriptionWe are currently looking for a skilled and experienced Hadoop Developer to join our team at Talent Pro.As a Hadoop Developer, you will be responsible for designing, developing, and maintaining large-scale big data processing pipelines using the Hadoop ecosystem (HDFS, MapReduce) and Spark.The ideal candidate will have a strong...

  • Big Data Developer

    20 hours ago

    Delhi, India Impetus Full time

    Lead Software EngineerLocations- Indore, Noida, HyderabadJob DescriptionWe are looking for a Lead Big Data Developer to design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets.Roles & ResponsibilitiesKey Responsibilities- Architect Data Solutions: Design...

  • Engineering Manager

    3 weeks ago

    Delhi, Delhi, India PROTEGRITY INDIA PRIVATE LIMITED Full time

    About the RoleProtegrity - Engineering Manager - Big DataProtegrity is a pioneering company in data protection, and we're seeking a seasoned Engineering Manager to lead our Big Data team. This role requires a strategic thinker with expertise in data security and a passion for mentoring and guiding technical teams.Key ResponsibilitiesManage the research and...

  • Cloud Data Engineer

    15 hours ago

    Delhi, Delhi, India Tata Consultancy Services Full time

    Job Role: Cloud Data Engineer - Big Data Solutions SpecialistAbout the Role: Tata Consultancy Services is seeking a skilled Cloud Data Engineer to join our team in Chennai. As a key member of our big data solutions team, you will be responsible for developing and deploying scalable big data pipelines on Google Cloud Platform (GCP).Responsibilities: Develop...

  • Big Data Developer

    4 weeks ago

    delhi, India Manuh Technologies Full time

    Big Data Engineer with 5+ years of experience on technologies Cloudera, Hadoop, sqoop, Spark, PySpark, Impala, SQL Previous experince into big data projects using hadoop and spark. Atleast one project experience into Cloudera hadoop with Impala and Hue Strong Programming experience in Python and PySpark Hands on experience in SQL using Impala Experience in...

  • Senior Data Engineer

    2 weeks ago

    Delhi, Delhi, India PRONEXUS CONSULTING PRIVATE LIMITED Full time

    **Job Title:** Senior Data Engineer - Big Data Pipeline SpecialistAt Pronexus Consulting Private Limited, we are seeking an experienced and highly skilled Senior Data Engineer - Big Data Pipeline Specialist to join our team. This role will involve designing, developing, and implementing large-scale data pipelines using various technologies such as Spark,...

  • Big Data Developer

    3 weeks ago

    delhi, India Manuh Technologies Full time

    Big Data Engineer with 5+ years of experience on technologies Cloudera, Hadoop, sqoop, Spark, PySpark, Impala, SQL Previous experince into big data projects using hadoop and spark.Atleast one project experience into Cloudera hadoop with Impala and HueStrong Programming experience in Python and PySparkHands on experience in SQL using ImpalaExperience in...

  • Big Data Developer

    4 weeks ago

    delhi, India Manuh Technologies Full time

    Big Data Engineer with 5+ years of experience on technologies Cloudera, Hadoop, sqoop, Spark, PySpark, Impala, SQLPrevious experince into big data projects using hadoop and spark.Atleast one project experience into Cloudera hadoop with Impala and HueStrong Programming experience in Python and PySparkHands on experience in SQL using ImpalaExperience in...

  • Big Data Developer

    4 weeks ago

    Delhi, India Manuh Technologies Full time

    Big Data Engineer with 5+ years of experience on technologies Cloudera, Hadoop, sqoop, Spark, PySpark, Impala, SQLPrevious experince into big data projects using hadoop and spark.Atleast one project experience into Cloudera hadoop with Impala and HueStrong Programming experience in Python and PySparkHands on experience in SQL using ImpalaExperience in...

  • Data Engineering Expert for Big Data Systems

    3 days ago

    Delhi, Delhi, India NEXUS CONSULTING SOLUTIONS Full time

    We are seeking a skilled Data Engineer to join our team at NEXUS CONSULTING SOLUTIONS. This role requires experience in working with large data sets and a strong understanding of big data systems.Job DescriptionA successful candidate will have hands-on experience in Python and Scala programming languages. Additionally, they should be familiar with the Spark...