Big Data Engineer

2 months ago


India Pert Telecom Solutions Full time

Job Description : We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for this role. 

Key Responsibilities:  

  • Develop and implement data ingestion pipelines and ETL processes using PySpark and Python capable of managing and processing large volumes of data, ranging in the millions of records. 
  • Design and develop APIs using Python to facilitate data integration and processing. 
  • Deploy, configure, and maintain Hadoop ecosystem components including HDFS , YARN, Hive, and HBase in on-premises environments. 
  • Optimize the performance of Big Data applications and queries, ensuring efficient resource utilization. 
  • Manage and monitor cluster health , including tuning and troubleshooting performance issues. 
  • Ensure data security , integrity , and availability within the Big Data infrastructure. 
  • Collaborate with cross-functional teams to integrate data analytics solutions and support business requirements. 

Required Skills and Qualifications:  

  • Bachelor’s degree in computer science, Engineering, or a related field; 
  • Proven experience 3+ years working with Big Data technologies such as Hadoop, PySpark, and MySQL in on-premises environments and handling large-scale data volumes, including datasets with millions of records, in on-premises environments. 
  • Strong programming skills in Python, with experience in developing APIs. 
  • Hands-on experience in Linux. 
  • Expertise in deploying, managing, and optimizing Big Data clusters in a local data center. 
  • Proficiency in monitoring and performance tuning of Hadoop and PySpark clusters. 
  • Ability to troubleshoot and resolve complex issues related to data processing and infrastructure. 
  • Excellent problem-solving skills and the ability to work independently as well as part of a team. 
  • Strong communication skills with the ability to convey technical concepts to non-technical stakeholders. 
  • Experience in dockers 

Preferred Skills:  

  • Experience with Apache Airflow, Kafka, Apache Flink, and Apache NiFi for data orchestration and streaming. 
  • Certification in Hadoop or related Big Data technologies. 

Application Process: Interested candidates should submit their resume and a cover letter outlining their qualifications and experience in Big Data engineering and on-premises cluster management and include details of at least one relevant project involving the handling of large data volumes (in millions of records), you have worked on, including a brief description of the project, your role and the technologies used. This is mandatory requirement for our screening process



  • India ITI Data Full time

    Job Title: Big Data Solutions Architect Company Overview:ITI Data is a leading provider of data solutions and services. About the Role:We are seeking an experienced Big Data Solutions Architect to join our team. The successful candidate will have a strong background in designing and implementing large-scale data architectures using Apache Spark, Hadoop, and...


  • India NEXUS CONSULTING SOLUTIONS Full time

    Nexus Consulting Solutions seeks an experienced Big Data Systems Engineer to join our team. This is a challenging and rewarding role that requires expertise in designing, developing, and maintaining large-scale big data systems.We are looking for someone with hands-on experience in Python and Scala programming languages. Familiarity with the Spark framework,...


  • India ITI Data Full time

    We are seeking an experienced PySpark Data Engineer to join our team at ITI Data. As a key member of our team, you will design and build scalable data solutions for one of our Fortune 500 Client programs.About the Role:The ideal candidate will have extensive experience in working with Hadoop, NoSQL databases, and Spark framework on Python. You will be...


  • India Radioactive Technologies Full time

    At Radioactive Technologies, we're on the hunt for a skilled Energetic Big Data Engineer to join our team. The salary for this role is around $140,000 per year.We're looking for someone with 10 years of experience in data transformation and aggregation using Scala/Spark jobs. As a Full Time Contract Opportunity, you'll have the chance to work fully...

  • Big Data Developer

    3 weeks ago


    India Impetus Full time

    Lead Software Engineer Locations- Indore, Noida, Hyderabad Job Description We are looking for a Lead Big Data Developer to design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets. Roles & Responsibilities Key Responsibilities ...


  • Anywhere in India/Multiple Locations US Technology International Pvt Ltd Full time

    Job Title: Senior Big Data EngineerJob Summary:US Technology International Pvt Ltd is seeking a Senior Big Data Engineer to lead the development and implementation of big data solutions. The ideal candidate will have expertise in Hadoop, Spark, and other big data technologies, as well as strong leadership and communication skills.Key Responsibilities:Design,...


  • India LTIMindtree Full time

    Job OverviewWe are seeking a highly skilled Data Engineer to join our team at LTIMindtree. The ideal candidate will have extensive experience working with Big Data technologies, particularly Spark and Java.


  • India DigitalTek Solutions Full time

    Company OverviewDigitalTek Solutions is a trusted provider of managed services, staff augmentation, and IT consulting. With over a decade of expertise, we deliver tailored solutions to Fortune 1000 companies across North America.We specialize in connecting top talent with opportunities in the IT, Engineering, and Digital sectors, serving various industries...


  • India Coffeee Full time

    We are seeking an experienced Big Data Engineer to join our innovative team at Coffeee. As a key member of our team, you will be responsible for developing and implementing Hadoop-based solutions that drive business growth.The ideal candidate will have hands-on experience with the Hadoop ecosystem, including HDFS, Hive, Pig, and Spark. You will also be...

  • Big Data Developer

    4 weeks ago


    India Luxoft Full time

    Project Description:As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine...

  • Big Data Developer

    4 weeks ago


    India Luxoft Full time

    Project Description: As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered by retail stores, restaurants, banks, and other consumer-focused companies. The challenge will be to create high-performance algorithms, cutting-edge analytical techniques including machine...

  • Big Data Engineer

    4 weeks ago


    Anywhere in India/Multiple Locations D-TechWorks Pvt Ltd Full time

    We are looking for Aws big data engineer In Remote , India Position : Aws big data engineerExperience : 7 to 9 YearLocation : Remote , India Mandate skills : BigData Python Pyspark AWS Glue Airflow Redshift Responsibilities :- Ensure effective Design, Development, Validation and Support activities in line with client needs and architectural requirements.,-...


  • India QntmLogic, LLC. Full time

    Job DescriptionWe are seeking a highly skilled Senior Python Developer for our Big Data development team at QntmLogic, LLC. in Bangalore/Pune (Hybrid). The ideal candidate will have hands-on experience with PySpark, Hadoop, Tableau, and SQL Server development.The successful candidate will be responsible for designing and building big data solutions using...


  • India Cynosure Corporate Solutions Full time

    Cynosure Corporate Solutions is seeking a highly skilled Data Engineer to lead our big data solutions team. This role requires expertise in designing, building, and optimizing complex data pipelines using Hadoop/Spark ecosystem.About the Role:We are looking for an experienced professional who can work collaboratively with cross-functional teams to deliver...

  • Big Data Architect

    11 hours ago


    India Bloom Consulting Services Full time

    Not Disclosed Experience **12 - 16 Years** Offered Salary **Lakh** Notice Period **Not Disclosed** **Role: Big Data Architect** **Location: Bangalore** **Skills required**: Total 12 to 16 years of experience in BI, analytics, visualization, **data** integration, designing and developing **data **warehouses (ETL, **data**bases, front-end...

  • Big Data Architect

    11 hours ago


    India Bloom Consulting Services Full time

    NA Experience **12 - 16 years** Offered Salary Notice Period **Not Disclosed** **Role: Big Data Architect** **Location: Bangalore** **Skills required**: Total 12 to 16 years of experience in BI, analytics, visualization, **data** integration, designing and developing **data **warehouses (ETL, **data**bases, front-end tools). 9+ years of...

  • Big Data Architect

    8 months ago


    India Princeton IT Services Full time

    **Job Title**: Big Data Architect **Location**:Kakinada (work from office) or Bangalore (work from office) **Shift**: UK shift preferred **Job Overview**: We are seeking a highly skilled and motivated Big Data Architect to join our team. This individual will play a crucial role in designing and implementing scalable and efficient big data solutions,...


  • India Recro Full time

    About the RoleAt Recro, we are looking for a highly skilled Big Data Solutions Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing and developing scalable data pipelines and solutions using Azure services.We are seeking someone with strong experience in SQL for querying, analyzing, and managing data,...


  • india iitjobs, Inc. Full time

    Hello Folks,Are you a tech enthusiast or a sales ace looking for a new opportunity? Look no further! We're hiring for various positions.We are hiring Data Engineer/Data Scientist who enjoys working to join an innovative and growing organisation with experience on Python, SQL, Building data pipelines in big data environments, Tableau, Data Visualization and...

  • Big Data Developer

    4 weeks ago


    India Luxoft Full time

    Senior Data Engineer Job DescriptionWe are seeking a highly skilled Senior Data Engineer to join our team at Luxoft. As a key member of our Data Engineering & Analytics team, you will play a crucial role in developing data-driven solutions that drive business growth.About the RoleThis is an exciting opportunity for a seasoned data engineer to leverage their...