Big data engineer

6 months ago


Hyderabad, India Virtusa Full time
Big data engineer - CREQ189284 Description Position: Big data engineer Primary: big data concepts, aws, pyspark Location: HYD Create Trigger based automation framework for Data Migration
Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same
Integrate CDMS framework with Lake and Data bridge API
Data migration from S3 Managed to Hadoop On prem
Jobs for Daily and Bulk loads
Test support for AVRO to test lake features
Test support for Compression types like LZO, .ENC to test lake features
ABINITIO integration: Build feature to create operation trigger for ABI pipeline
Movement to new datacenter -SQL server migration
Carlstadt to Ashburn (DR switchover)

Develop and maintain data platforms using Python.
Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity.
Collaborate with cross functional teams to understand data requirements and design solutions that meet business needs .
Implement and manage agents for monitoring, logging, and automation within AWS environments.
Handling migration from PySpark to AWS.
(Secondary) Resource must have hands on development experience with various Ab Initio components such as Rollup Scan join Partition by key Partition by Round Robin Gather Merge Interleave Lookup etc.
Must have experience with SQL database programming SQL performance tuning relational model analysis.
Good knowledge in developing UNIX scripts Oracle SQLPLSQL.
Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging.
Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management.
Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Primary Location Hyderabad, Andhra Pradesh, India Job Type Experienced Primary Skills Big Data, Python, Spark Years of Experience 12 Qualification Education: Any degree or equivalent Experience: 6+ years Travel No

  • Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Solutions Engineer to join our team in Hyderabad, India. As a key member of our engineering team, you will be responsible for designing and implementing complex data products components based on requirements with possible technical solutions.You will work closely with Data Scientists, Data Engineers,...

  • Big Data Engineer

    4 months ago


    Hyderabad, India NucleusTeq Full time

    Big Data Engineer Responsibilities: Big Data Engineer with 5+ years of experienceExperience in leading a team of engineers and a good attitude toward learning the domain and implementationStrong exposure and expertise in Spark (Primary), Scala/Java (Scala Primary), Airflow Orchestration and AWS.Finalizing the scope of the system and delivering...

  • Big Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India BlueByte Technologies Full time

    About the RoleAs a Big Data Engineer at BlueByte Technologies, you will be responsible for creating efficient and scalable solutions using Google BigQuery. Your primary focus will be on designing and implementing data pipelines that can handle petabytes of data across various domains.You will work closely with software engineers, data scientists, product...


  • Hyderabad, Telangana, India Tata Consultancy Services Full time

    Job DescriptionAbout the Role:We are seeking an experienced Senior Big Data Engineer to join our team at Tata Consultancy Services. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale big data environments using PySpark.Key Responsibilities:Design and implement complex big data...

  • Big Data Engineer

    7 months ago


    Hyderabad, India TechSophy Full time

    Job Description: Roles&Responsibilities : 1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map. 2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into...


  • Hyderabad, India Virtusa Full time

    Big Data Python Engineer - CREQ191701 Description Role GDT WS Big Data Python Engineer Exp 5+ Location Pune/Hyderabad MUST Have Skills Design and implement data processing pipelines using Python + SCALA programming and big data technologies such as Hadoop, MapReduce, Hive, and Spark. Airflow Jenkins Understand business requirements to code up data assets...

  • AWS Big Data Engineer

    6 months ago


    Hyderabad, India Tiger Analytics Full time

    Job Description We are looking for a Senior Data Engineer to be based out of our Chennai, Bangalore & Hyderabad offices. This role involves a combination of hands-on contribution, customer engagement, and technical team management. As a Senior Data Engineer, you will Design and build solutions for near real-time stream processing as well as batch...

  • Big data engineer

    2 months ago


    Hyderabad, India Anblicks Full time

    Experience- 4+ yearsExperience in hadoop/big data technologies.Experience in spark.Experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;Experience with containerization and related technologies (e.g. Docker,...

  • Big Data Engineer

    2 months ago


    hyderabad, India Anblicks Full time

    Experience- 4+ yearsExperience in hadoop/big data technologies.Experience in spark.Experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;Experience with containerization and related technologies (e.g. Docker,...

  • Big Data Engineer

    2 months ago


    Hyderabad, India Anblicks Full time

    Experience- 4+ yearsExperience in hadoop/big data technologies.Experience in spark.Experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;Experience with containerization and related technologies (e.g. Docker,...


  • Hyderabad, Telangana, India Tata Consultancy Services Full time

    About Tata Consultancy ServicesTata Consultancy Services is a leading IT services company that empowers global businesses through its cutting-edge technology and innovative solutions.Compensation PackageThe estimated salary for this role is around $120,000 - $180,000 per annum, depending on the location and individual qualifications.About the RoleJob...


  • Hyderabad, Telangana, India Anblicks Full time

    We are looking for a skilled Big Data Engineer to join our team at Anblicks. This is a full-time opportunity with a competitive salary and excellent benefits.Key Responsibilities:Design, develop, and deploy large-scale data processing systems using Hadoop and Spark technologies.Collaborate with cross-functional teams to identify business requirements and...

  • Big Data Engineer

    2 months ago


    Hyderabad, India Anblicks Full time

    Experience- 4+ yearsExperience in hadoop/big data technologies.Experience in spark.Experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;Experience with containerization and related technologies (e.g. Docker,...

  • Big Data Engineer

    2 months ago


    Hyderabad, India Anblicks Full time

    Experience- 4+ yearsExperience in hadoop/big data technologies.Experience in spark.Experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required;Experience with containerization and related technologies (e.g. Docker,...

  • Big Data Engineer

    2 months ago


    Hyderabad, India Anblicks Full time

    Experience- 4+ years Experience in hadoop/big data technologies. Experience in spark. Experience working on Google or AWS cloud developing data solutions. Certifications preferred. Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required; Experience with containerization and related technologies (e.g. Docker,...


  • Hyderabad, Telangana, India Madhees Techno Full time

    **Job Description:** We are seeking an experienced Azure Big Data Engineer Leader to join our team at Madhees Techno. As a key member of our big data engineering team, you will be responsible for designing, implementing, and managing data solutions on the Microsoft Azure cloud platform. With 5+ years of experience in data engineering, you will oversee the...


  • Hyderabad, Telangana, India Tata Consultancy Services Full time

    Job Title: Senior Big Data EngineerAbout Us: Tata Consultancy Services, a global leader in IT services and consulting, is seeking an experienced Senior Big Data Engineer to join our team.Estimated Salary: ₹1,200,000 - ₹1,500,000 per annum (dependent on location and experience).Job Description:The successful candidate will have hands-on experience with...


  • Hyderabad, India data.ai Remote Work Freelance Full time

    What can you tell your friends when they ask you what you do? We’re looking for an experienced Big Data Engineer who can create innovative new products in the analytics and data space. You will participate in the development that creates the world's #1 mobile app analytics service. Together with the team, you will build out new product features and...


  • Hyderabad, Telangana, India People Tech Group Inc Full time

    We are seeking a highly skilled Senior Big Data Engineer to join our team at People Tech Group Inc. This is a fantastic opportunity for someone looking to leverage their expertise in big data processing and analytics to drive business growth.Job DescriptionIn this role, you will be responsible for designing, developing, and implementing large-scale data...


  • Hyderabad, India iitjobs, Inc. Full time

    Hello Folks,Are you a tech enthusiast or a sales ace looking for a new opportunity? Look no further! We're hiring for various positions.We are hiring Data Engineer/Data Scientist who enjoys working to join an innovative and growing organisation with experience on Python, SQL, Building data pipelines in big data environments, Tableau, Data Visualization and...