Big data lead engineer
2 weeks ago
Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same
Integrate CDMS framework with Lake and Data bridge API
Data migration from S3 Managed to Hadoop On prem
Jobs for Daily and Bulk loads
Test support for AVRO to test lake features
Test support for Compression types like LZO, .ENC to test lake features
ABINITIO integration: Build feature to create operation trigger for ABI pipeline
Movement to new datacenter -SQL server migration
Carlstadt to Ashburn (DR switchover)
Develop and maintain data platforms using Python.
Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity.
Collaborate with cross functional teams to understand data requirements and design solutions that meet business needs .
Implement and manage agents for monitoring, logging, and automation within AWS environments.
Handling migration from PySpark to AWS.
(Secondary) Resource must have hands on development experience with various Ab Initio components such as Rollup Scan join Partition by key Partition by Round Robin Gather Merge Interleave Lookup etc.
Must have experience with SQL database programming SQL performance tuning relational model analysis.
Good knowledge in developing UNIX scripts Oracle SQLPLSQL.
Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging.
Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management.
Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Primary Location Hyderabad, Andhra Pradesh, India Job Type Experienced Primary Skills Big Data, Java, Ab Initio, PySpark, Spark, Python, AWS Native Data Services Years of Experience 12 Qualification Education: Any degree or equivalent Experience: 8 years Travel No
-
Big data lead engineer
2 weeks ago
hyderabad, India Virtusa Full timeBig data lead engineer - CREQ188245 Description Position: Big data engineer Primary skills: Big data, Pyspark, AWS Location: HYD Create Trigger based automation framework for Data MigrationIdentify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the sameIntegrate CDMS framework with Lake and Data...
-
Big data lead engineer
7 days ago
Hyderabad, India Virtusa Full timeBig data lead engineer - CREQ188245 DescriptionPosition: Big data engineerPrimary skills: Big data, Pyspark, AWSLocation: HYDCreate Trigger based automation framework for Data MigrationIdentify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the sameIntegrate CDMS framework with Lake and Data bridge APIData...
-
Lead Big Data Software Engineer
1 month ago
Hyderabad, India Epam Full timeDescription EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that...
-
Big Data Platform Engineer
1 month ago
hyderabad, India Apple Full timeSummary:The people here at Apple don’t just build products — we craft the kind of wonder that’s revolutionized entire industries. It’s the diversity of those people and their ideas that supports the innovation that runs through everything we do, from amazing technology to industry-leading environmental efforts. Join Apple, and help us leave the world...
-
Big Data Engineer
4 weeks ago
Hyderabad, India TechSophy Full timeJob Description:Roles&Responsibilities :1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map.2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into the data...
-
Big Data Engineer
1 month ago
Hyderabad, India TechSophy Full timeJob Description: Roles&Responsibilities : 1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map. 2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into...
-
Big Data Engineer
1 month ago
hyderabad, India TechSophy Full timeJob Description: Roles&Responsibilities : 1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map.2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into...
-
Senior Big Data Software Engineer
4 weeks ago
Hyderabad, India Epam Full timeDescriptionEPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver...
-
Big Data Engineer
4 weeks ago
Hyderabad, India Anicalls (Pty) Ltd Full timeBig Data EngineerSpecification-• 2 – 7 years of recent experience indata engineering.• Bachelor's Degree or more in ComputerScience or a related field.• A solid track record of data management showing your flawless executionand attention to detail.• Strong knowledge of and experience with statistics.• Programming experience, ideally in , Spark,...
-
Senior Big Data Software Engineer
1 month ago
Hyderabad, India Epam Full timeDescription EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that...
-
Big Data Engineer
1 month ago
hyderabad, India Anicalls (Pty) Ltd Full timeBig Data Engineer Specification- • 2 – 7 years of recent experience indata engineering. • Bachelor's Degree or more in ComputerScience or a related field. • A solid track record of data management showing your flawless executionand attention to detail.• Strong knowledge of and experience with statistics.• Programming experience, ideally in ,...
-
Principal Consultant-Big Data Engineer
3 weeks ago
Hyderabad, India Genpact Full timeGenpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people –...
-
Principal Consultant-Big Data Engineer
3 weeks ago
Hyderabad, India Genpact Full timeGenpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people – we...
-
Principal Consultant-Big Data Engineer
3 weeks ago
hyderabad, India Genpact Full timeGenpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people –...
-
Senior Azure Data Engineer
4 weeks ago
Hyderabad, India Enable Data Incorporated Full timeEnable Data Incorporated is seeking a highly skilled Senior Spark Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem. As a Senior Spark Engineer, you will be responsible for overseeing all aspects...
-
Senior Azure Data Engineer
1 month ago
Hyderabad, India Enable Data Incorporated Full timeEnable Data Incorporated is seeking a highly skilled Senior Spark Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem. As a Senior Spark Engineer, you will be responsible for overseeing all aspects...
-
Senior Azure Data Engineer
1 month ago
hyderabad, India Enable Data Incorporated Full timeEnable Data Incorporated is seeking a highly skilled Senior Spark Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem. As a Senior Spark Engineer, you will be responsible for overseeing all aspects...
-
SRE Manager Big Data
3 days ago
Hyderabad/ Secunderabad, India timesjobs Full timeRole & ResponsibilitiesWe are seeking a hands-on Manager who has experience leading large Big Data environments spread across thousands of nodes and petabytes of data.We look forward to a dynamic manager with a background & experience that looks like this:Grown into leadership roles after proving technical skills in individual contributor roles but still...
-
Big Data with Data Modeling
3 weeks ago
Hyderabad, India Virtusa Full timeBig Data with Data Modeling - CREQ186896 Description EIS DATA JD Tier 2Cloud Data Lead with very strong in Experience in Bigdata (Kafka, Elastic search,Logstash,Kibana)Required experience with Data Engineering and ETL/EDW Design Process and Practices.Experience in Data ModelingExperience in CDC process (Added Advantage if knowledge in Debezium or any CDC...
-
Big Data Engineer, Data Analysis
4 days ago
Hyderabad, India data.ai Full timeWhat can you tell your friendswhen they ask you what you do?“As a Data Analyst at data.ai, I help analyze, monitor, and deliver comprehensive and high-quality app market data to app professionals. I am empowered to work independently with the support of team members spread across APAC, AMER, and EMEA. I am the key contributor to the new and existing...