Big data engineer

21 hours ago

Hyderabad, India Virtusa Full time

Big data engineer - CREQ Description Position: Big data engineer Primary: big data concepts, aws, pyspark Location: HYD Create Trigger based automation framework for Data Migration Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same Integrate CDMS framework with Lake and Data bridge API Data migration from S3 Managed to Hadoop On prem Jobs for Daily and Bulk loads Test support for AVRO to test lake features Test support for Compression types like LZO, .ENC to test lake features ABINITIO integration: Build feature to create operation trigger for ABI pipeline Movement to new datacenter -SQL server migration Carlstadt to Ashburn (DR switchover) Develop and maintain data platforms using Python. Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity. Collaborate with cross functional teams to understand data requirements and design solutions that meet business needs . Implement and manage agents for monitoring, logging, and automation within AWS environments. Handling migration from PySpark to AWS. (Secondary) Resource must have hands on development experience with various Ab Initio components such as Rollup Scan join Partition by key Partition by Round Robin Gather Merge Interleave Lookup etc. Must have experience with SQL database programming SQL performance tuning relational model analysis. Good knowledge in developing UNIX scripts Oracle SQLPLSQL. Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging. Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management. Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Primary Location Hyderabad, Andhra Pradesh, India Job Type Experienced Primary Skills Big Data, Python, Spark Years of Experience 12 Qualification Education: Any degree or equivalent Experience: 6+ years Travel No

  • Big Data Engineer

    1 week ago

    hyderabad, India beBeeData Full time

    Job Description:We are seeking a skilled Big Data Engineer to join our team. The ideal candidate will have expertise in designing, developing, and maintaining large-scale data processing systems.Required Skills and Qualifications:4-7 years of experience in big data engineeringStrong background in programming languages such as Java, Python, or ScalaExperience...

  • Big data Engineer

    2 days ago

    Hyderabad, Telangana, India Zorba AI Full time

    Role OverviewWe are looking fortwo Junior-level Big Data Engineerswith strong hands-on experience inAWS, Python, Apache Airflow, and Big Data technologies. The ideal candidates will design, build, and maintainscalable data pipelines and workflowsin a cloud-based environment, ensuring data reliability, performance, and security.Key ResponsibilitiesDesign,...

  • Big Data Engineer

    4 days ago

    Hyderabad, Telangana, India Ascendion Engineering Full time

    Big Data Engineer:(Note - One in-person interview is needed after first round selection, Location - Hyderabad)BS degree in computer science, computer engineering or equivalent3-5 years of experience delivering enterprise software solutionsFamiliar with Spark, Scala, Python, AWS Cloud technologies2+ years of experience across multiple Hadoop / Spark...

  • Big Data QA/Test Engineer

    2 days ago

    Hyderabad, Telangana, India deltaclass technology solutions Full time

    Role : Bigdata QA Test Engineer Exp : 6+Yrs Notice :Immediate to 15 Days Location:Hyderabad (Hybrid) Big Data QA/Test engineer – 3 Openings Hyderabad or Chennai (Hybrid) Big Data QA/Test Engineer AWS + Data-bricks + Big Data + Python + DevOps Big Data QA/Test Engineer AWS + Data-bricks + Big Data + Python + DevOps Big Data QA/Test Engineer AWS +...

  • Big Data QA/Test Engineer

    9 hours ago

    Hyderabad, Telangana, India deltaclass technology solutions Full time

    Currently we are looking for Bigdata QA test Engineer. Responsibilities Role : Bigdata QA Test Engineer Exp : 6+Yrs Notice :Immediate to 15 Days Location:Hyderabad (Hybrid) Big Data QA/Test engineer – 3 Openings Hyderabad or Chennai (Hybrid) Big Data QA/Test Engineer AWS + Data-bricks + Big Data + Python + DevOps Big Data QA/Test Engineer AWS +...

  • Big Data Python Engineer

    21 hours ago

    Hyderabad, India Virtusa Full time

    Big Data Python Engineer - CREQ Description Role GDT WS Big Data Python Engineer Exp 5+ Location Pune/Hyderabad MUST Have Skills Design and implement data processing pipelines using Python + SCALA programming and big data technologies such as Hadoop, MapReduce, Hive, and Spark. Airflow Jenkins Understand business requirements to code up data assets in...

  • Big Data Engineer

    5 days ago

    Bengaluru, Chennai, Hyderabad, India Net2source Llp Full time

    Hiring: Big Data/Hadoop Engineer (PySpark, Hive) | 4 to 6 YearsWe are looking for talented Big Data/Hadoop Engineer for one of our premium clientsWork Locations: Bangalore (KA) | Chennai (TN) | Hyderabad (TG)Experience: 4 to 6 YearsRequired Skills:Strong hands-on experience in PySpark and HiveExpertise in Big Data & Hadoop Ecosystem toolsAbility to design...

  • Big Data

    6 days ago

    Hyderabad, India Greenwich Associates Full time

    Please enter job description - 5+ years software development experience in an enterprise level engineering environment demonstrating increasing levels of technical leadership. 3+ years of work experience with ETL, Data Modeling, and Data Architecture. Experience with Big Data technologies such as Hadoop/Hive/Spark with Spark as primary. Fluent in one or more...

  • AWS Big Data Engineer

    21 hours ago

    Hyderabad, India Tiger Analytics Full time

    Job Description We are looking for a Senior Data Engineer to be based out of our Chennai, Bangalore & Hyderabad offices. This role involves a combination of hands-on contribution, customer engagement, and technical team management. As a Senior Data Engineer, you will Design and build solutions for near real-time stream processing as well as batch processing...

  • Big Data Engineer

    4 days ago

    Hyderabad, Telangana, India IDESLABS PRIVATE LIMITED Full time

    We are looking for a skilled Big Data Developer with 6-8 years of experience to join our team in Hyderabad/Pune. The ideal candidate will have strong knowledge of Unix/BigData Scripting and a solid understanding of BigData (CDP/Hive) Environment.Roles and ResponsibilityDesign, develop, and implement big data solutions using Hadoop/CDP, Hive.Collaborate with...