Big data engineer
2 weeks ago
Identify roles/access needed for data migration from federated bucket to managed bucket and Build APIs for the same
Integrate CDMS framework with Lake and Data bridge API
Data migration from S3 Managed to Hadoop On prem
Jobs for Daily and Bulk loads
Test support for AVRO to test lake features
Test support for Compression types like LZO, .ENC to test lake features
ABINITIO integration: Build feature to create operation trigger for ABI pipeline
Movement to new datacenter -SQL server migration
Carlstadt to Ashburn (DR switchover)
Develop and maintain data platforms using Python.
Work with AWS and Big Data, design and implement data pipelines, and ensure data quality and integrity.
Collaborate with cross functional teams to understand data requirements and design solutions that meet business needs .
Implement and manage agents for monitoring, logging, and automation within AWS environments.
Handling migration from PySpark to AWS.
(Secondary) Resource must have hands on development experience with various Ab Initio components such as Rollup Scan join Partition by key Partition by Round Robin Gather Merge Interleave Lookup etc.
Must have experience with SQL database programming SQL performance tuning relational model analysis.
Good knowledge in developing UNIX scripts Oracle SQLPLSQL.
Leverage internal tools and SDKs, utilize AWS services such as S3, Athena, and Glue, and integrate with our internal Archival Service Platform for efficient data purging.
Lead the integration efforts with the internal Archival Service Platform for seamless data purging and lifecycle management.
Collaborate with the data engineering team to continuously improve data integration pipelines, ensuring adaptability to evolving business needs. Primary Location Hyderabad, Andhra Pradesh, India Job Type Experienced Primary Skills Big Data, Python, Spark Years of Experience 12 Qualification Education: Any degree or equivalent Experience: 6+ years Travel No
-
Big Data Engineer, Data Modeling
2 weeks ago
Hyderabad, Telangana, India data Remote Work Freelance Full timeWhat can you tell your friends when they ask you what you do? We're looking for an experienced Big Data Engineer who can create innovative new products in the analytics and data space. You will participate in the development that creates the world's #1 mobile app analytics service. Together with the team, you will build out new product features and...
-
Big Data Engineer, Data Analysis
2 weeks ago
Hyderabad, Telangana, India data Remote Work Freelance Full timeWhat can you tell your friends when they ask you what you do? "As a Data Analyst at data.ai, I help analyze, monitor, and deliver comprehensive and high-quality app market data to app professionals. I am empowered to work independently with the support of team members spread across APAC, AMER, and EMEA. I am the key contributor to the new and existing...
-
Big Data Engineering Lead
7 days ago
Hyderabad, Telangana, India Enable Data Incorporated Full timeKey ResponsibilitiesDesign and develop scalable and robust data solutions in the cloud using Apache Spark and Databricks.Gather and analyze data requirements from business stakeholders and identify opportunities for data-driven insights.Build and optimize data pipelines for data ingestion, processing, and integration using Spark and Databricks.Ensure data...
-
Big Data Engineer
2 days ago
Hyderabad, Telangana, India SMARTWORK IT SERVICES Full timeJob Title : Big Data Engineer (Dynatrace)Location : Chennai, Bangalore, HyderabadExperience : 6-10 YearsRequired skills : Big Data, DynatraceJob Description : We are looking for an experienced Big Data Engineer (Dynatrace) with building scalable data pipelines and ensuring performance optimization through Dynatrace monitoring. This role requires a seasoned...
-
Big Data Engineer
1 week ago
Hyderabad, Telangana, India Questsphere Private ltd. Full timeOur client is looking to deploy big data engineer at one of the worlds top product company R&D centre based out of Hyderabad. As a Big Data Engineer, you are going to work with cross-functional collaboration and communication with members from other teams.As an Engineer, you are responsible to analyze large data sets to identify errors . You are responsible...
-
Big Data Engineer
3 weeks ago
Hyderabad, Telangana, India Folks Palette LLP Full timeJob Description : We are seeking an experienced Big Data Engineer with PySpark Developer to join our dynamic team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining scalable data solutions in a big data ecosystem. The ideal candidate will have a strong background in big data technologies,...
-
Data Engineer for Big Data
6 days ago
Hyderabad, Telangana, India Incedo Inc. Full timeIncedo Inc. is seeking a highly skilled Data Engineer for Big Data to join our team. The ideal candidate will have strong technical, analytical, and problem-solving skills.We are looking for someone with experience in designing and building big data pipelines using CDH On-premise, Spark, PySpark, Scala, Python, NiFi, Hive, and NoSql DBs.The candidate should...
-
Big Data Engineer
2 weeks ago
Hyderabad, Telangana, India TechSophy Full timeJob Description: Roles&Responsibilities : 1. As an Engineer, you are responsible to analyze large data sets to identify errors in the map. 2. As an Engineer, you are responsible to design and implement a complex algorithm for resolving the issue, review the solution with a team of engineers and analysts, and integrate the resulting solution into the...
-
Big Data Python Engineer
2 weeks ago
Hyderabad, Telangana, India Virtusa Full timeBig Data Python Engineer - CREQ191701 Description Role GDT WS Big Data Python Engineer Exp 5+ Location Pune/Hyderabad MUST Have Skills Design and implement data processing pipelines using Python + SCALA programming and big data technologies such as Hadoop, MapReduce, Hive, and Spark. Airflow Jenkins Understand business requirements to code up data assets...
-
AWS Big Data Engineer
2 weeks ago
Hyderabad, Telangana, India Tiger Analytics Full timeJob Description We are looking for a Senior Data Engineer to be based out of our Chennai, Bangalore & Hyderabad offices. This role involves a combination of hands-on contribution, customer engagement, and technical team management. As a Senior Data Engineer, you will Design and build solutions for near real-time stream processing as well as batch...
-
Big Data Engineer
4 days ago
Hyderabad, Telangana, India Apollo HealthAxis Full timeJob DescriptionAs a Big Data Engineer at Apollo HealthAxis, you will be responsible for designing and implementing large-scale data processing systems using distributed computing technologies. You will work closely with cross-functional teams to understand business requirements and translate them into effective data architecture and...
-
Python Big Data Engineer
6 days ago
Hyderabad, Telangana, India Coforge Full timeCoforge is looking for an experienced Python Big Data Engineer to join our team.The ideal candidate will have expertise in designing and developing big data processing systems using PySpark.Key Responsibilities:Design and implement big data processing pipelines using PySpark.Analyze complex data sets to identify trends and insights.Collaborate with...
-
Senior Big Data Engineer
1 week ago
Hyderabad, Telangana, India PRI Global Full timeAbout the RolePRI Global is looking for an experienced Senior Big Data Engineer to lead our big data initiatives. In this role, you will be responsible for designing and implementing large-scale data architectures and systems.Develop and Implement Data ArchitecturesCollaborate with Engineering TeamsEvaluate System PerformanceTroubleshoot Complex IssuesKey...
-
Big Data Engineer
2 weeks ago
Hyderabad, Telangana, India Anicalls (Pty) Ltd Full timeBig Data Engineer Specification- • 2 – 7 years of recent experience in data engineering.• Bachelor's Degree or more in Computer Science or a related field. • A solid track record of data management showing your flawless execution and attention to detail.• Strong knowledge of and experience with statistics.• Programming experience, ideally...
-
Senior Big Data Engineer
6 days ago
Hyderabad, Telangana, India Enable Data Incorporated Full timeWe are looking for a Senior Data Engineer to join our team at Enable Data Incorporated. Our company is a leading provider of advanced application, data, and cloud engineering services that leverage modern solutions to drive increased value across our clients' business ecosystem.In this role, you will design, develop, and maintain scalable and efficient data...
-
Big Data Engineer for ValueLabs
24 hours ago
Hyderabad, Telangana, India ValueLabs Full timeCompany OverviewValueLabs is a leading data engineering and analytics company that delivers innovative solutions to its clients. We are passionate about leveraging big data technologies to drive business growth and improvement.
-
Big Data Engineer Specialist
5 days ago
Hyderabad, Telangana, India Zensar Technologies Full timeJob Description:We are seeking a highly skilled Data Engineer to join our team at Zensar Technologies. This role involves working with big data technologies, optimizing spark jobs, and developing pySpark applications using Spark Dataframes in Python.About the Role:We require candidates with 5+ years of overall IT experience, including hands-on experience in...
-
Big Data Solutions Developer
1 day ago
Hyderabad, Telangana, India Cognisol Full timeAbout the Job:Cognisol is looking for a highly skilled Databricks engineer to join our team. As a Databricks engineer, you will be responsible for designing, developing and implementing big data solutions using Databricks and Apache Spark.Key Responsibilities:Design and develop big data solutions using Databricks and Apache SparkWork closely with...
-
Senior Big Data Engineer Lead
2 days ago
Hyderabad, Telangana, India Micron Semiconductor Asia Operations Pte. Ltd Full timeWe are looking for leaders with strong technical experience in Big Data and Cloud Data warehouse technologies. This role will work primarily in Cloud data warehouse like Snowflake and GCP platforms, monitoring solutions such as Splunk and automation and machine learning using Python.Key Skills and Qualifications:10+ years developing, delivering, and/or...
-
GCP Big Data Engineer
2 weeks ago
Hyderabad, Telangana, India Tiger Analytics Full timeJob Description .css-ylb{color:var(--chakra-colors-black-);}.css-l0deym h1{margin:0;padding:0;font-size:26px;}.css-l0deym h2{margin:0;padding:0;font-size:20px;}.css-l0deym h3{margin:0;padding:0;font-size:15px;}.css-l0deym h4{margin:0;padding:0;font-size:13px;}.css-l0deym ul{list-style-type:disc;margin:0.5rem;margin-bottom:1.5rem;}.css-l0deym ul...