Big data engineer

14 hours ago

Hyderabad, India Virtusa Full time

Big data engineer - CREQ Description Position: Big data engineer Primary: big data concepts, aws, pyspark Location: HYD Create Trigger based automation framework for Data Migration Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same Integrate CDMS framework with Lake and Data bridge API Data migration from S3 Managed to Hadoop On prem Jobs for Daily and Bulk loads Test support for AVRO to test lake features Test support for Compression types like LZO, .ENC to test lake features ABINITIO integration: Build feature to create operation trigger for ABI pipeline Movement to new datacenter -SQL server migration Carlstadt to Ashburn (DR switchover) Develop and maintain data platforms using Python. Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity. Collaborate with cross functional teams to understand data requirements and design solutions that meet business needs . Implement and manage agents for monitoring, logging, and automation within AWS environments. Handling migration from PySpark to AWS. (Secondary) Resource must have hands on development experience with various Ab Initio components such as Rollup Scan join Partition by key Partition by Round Robin Gather Merge Interleave Lookup etc. Must have experience with SQL database programming SQL performance tuning relational model analysis. Good knowledge in developing UNIX scripts Oracle SQLPLSQL. Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging. Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management. Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Primary Location Hyderabad, Andhra Pradesh, India Job Type Experienced Primary Skills Big Data, Python, Spark Years of Experience 12 Qualification Education: Any degree or equivalent Experience: 6+ years Travel No

  • Big Data Engineer

    2 days ago

    Hyderabad, Telangana, India Sky Systems, Inc. (SkySys) Full time

    Role:Big Data EngineerPosition Type:Full-Time Contract (40hrs/week)Contract Duration:Long TermWork Schedule: 8 hours/day (Mon-Fri)Location: Hybrid - Hyderabad, India (3x days onsite/week)We are looking for a skilledBig Data Engineer (5+ years' experience)to join Experian's dynamic data engineering team. You will design and optimize scalable data pipelines,...

  • Big Data Engineer

    4 days ago

    Hyderabad, Telangana, India IDESLABS PRIVATE LIMITED Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    We are looking for a skilled Big Data Engineer with 6 to 22 years of experience. The ideal candidate will have expertise in Pyspark, Azure Data Bricks, and workflows. This position is available as a contract role across Pan India.Roles and ResponsibilityDesign and develop scalable big data systems using Pyspark and Azure Data Bricks.Implement and manage...

  • Big Data Engineer

    2 days ago

    Hyderabad, Telangana, India Harp Technologies Services Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Job Title: Bigdata + PysparkLocation: HyderabadExperience Level: 5-8 YearsNotice Period: Immediate to 30 DaysInterview Mode: L1 Virtual / L2 F2F (Mandatory)Work Mode: HybridJob SummaryWe are seeking a Specialist with 5 to 8 years of experience in the Big Data Hadoop Ecosystem to design and implement scalable data solutions.Detailed Job DescriptionWork...

  • Big Data Engineer

    3 weeks ago

    Hyderabad, Bengaluru, India Fusion Plus Solutions Full time

    Job Description We are looking for a skilled Big Data Engineer with 5 to 7 years of hands-on experience in Big Data technologies. The ideal candidate should have strong expertise in Hadoop, HDFS, Hive, SQL, and Unix, and be able to work on large-scale data processing systems. Roles and Responsibilities: - Work on design, development, and optimization of Big...

  • Big Data Engineer

    7 days ago

    Hyderabad, Telangana, India HARP Technologies And Services Pvt Ltd Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Description : - Skill : Bigdata + Pyspark - Location : Hyderabad only - Experience : 5-9 Years - Notice Period : Immediate to 30 Days - Interview Mode : L1 Virtual / L2 F2F (Mandatory) - Work Mode : Hybrid - Skills required : Bigdata (Hadoop Hive, Impala, Spark) , pyspark, Python, Oracle, Exadata (RDBMS),...

  • Big Data QA/Test Engineer

    7 days ago

    Hyderabad, Telangana, India deltaclass technology solutions Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Role : Bigdata QA Test Engineer Exp : 6+Yrs Notice :Immediate to 15 Days Location:Hyderabad (Hybrid) Big Data QA/Test engineer – 3 Openings Hyderabad or Chennai (Hybrid) Big Data QA/Test Engineer AWS + Data-bricks + Big Data + Python + DevOps Big Data QA/Test Engineer AWS + Data-bricks + Big Data + Python + DevOps Big Data QA/Test Engineer AWS +...

  • Big Data Engineer

    1 week ago

    Hyderabad, Telangana, India Pennant Technologies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: Senior Big Data EngineerDepartment: Data & AnalyticsEmployment Type: Full‑timeRole OverviewWe are seeking an experienced Big Data Engineer to join our growing Data & Analytics team. In this role, you'll design, build, and optimize our modern data platform—enabling scalable pipelines, reliable data lakes, and robust ML operations. You'll work...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, Chennai, Hyderabad, India Net2source Llp Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Hiring: Big Data/Hadoop Engineer (PySpark, Hive) | 4 to 6 YearsWe are looking for talented Big Data/Hadoop Engineer for one of our premium clientsWork Locations: Bangalore (KA) | Chennai (TN) | Hyderabad (TG)Experience: 4 to 6 YearsRequired Skills:Strong hands-on experience in PySpark and HiveExpertise in Big Data & Hadoop Ecosystem toolsAbility to design...

  • Big Data QA/Test Engineer

    1 week ago

    Hyderabad, Telangana, India deltaclass technology solutions Full time ₹ 10,00,000 - ₹ 15,00,000 per year

    Currently we are looking for Bigdata QA test Engineer. Responsibilities Role : Bigdata QA Test Engineer Exp : 6+Yrs Notice :Immediate to 15 Days Location:Hyderabad (Hybrid) Big Data QA/Test engineer – 3 Openings Hyderabad or Chennai (Hybrid) Big Data QA/Test Engineer AWS + Data-bricks + Big Data + Python + DevOps Big Data QA/Test Engineer AWS +...

  • Big Data Engineer

    1 week ago

    Hyderabad, Telangana, India Softility, Inc. Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Company DescriptionSoftility is a digital transformation technology services company specializing in cloud adoption, data engineering, AI-driven operations, digital service management, and cybersecurity. We are the preferred partner for many Fortune 500 companies, SLED entities, healthcare institutions, major enterprises, and federal agencies. Headquartered...