Data Engineer
4 weeks ago
Key Responsibilities :
- Design and develop data pipelines using AWS Glue, EMR, Spark Scala, and S3 to support both batch and real-time data processing needs.
- Implement ETL processes to extract, transform, and load data from various sources (structured and unstructured) into the data lake.
- Leverage Apache Spark on EMR for big data processing and transformations, using Spark Scala
- Manage and optimize data storage on S3, ensuring proper data partitioning, file formats (Parquet, ORC, Avro), and lifecycle policies for cost-effective storage solutions.
- Monitor, troubleshoot, and optimize EMR clusters for performance, scalability, and cost efficiency.
- Collaborate with data architect and analysts to ensure seamless data integration and support advanced analytics and machine learning models.
- Automate data workflows using AWS Step Functions, Lambda, and other orchestration tools.
Required Qualifications :
- Bachelor's or master's degree in computer science, Data Engineering, or a related technical field.
- 3-5 years of experience in data engineering, particularly using AWS services (EMR, Glue, S3, Lambda).
- Strong expertise in Apache Spark for distributed data processing, with hands-on coding experience in Scala and Python.
- Experience with building ETL pipelines and working with big data in a cloud-based Lakehouse environment.
- Deep understanding of data formats (Parquet, Avro, ORC) and file optimization techniques.
- Familiarity with data modeling principles, including partitioning, bucketing, and schema management in AWS Glue Data Catalog.
- Strong knowledge of SQL and query optimization for working with large datasets.
- Experience with AWS security services such as IAM, KMS (Key Management Service), and encryption best practices.
- Proficiency in troubleshooting and performance tuning of Spark and EMR clusters for large-scale data processing.
- Familiarity with CI/CD pipelines and infrastructure-as-code (Terraform, CloudFormation) for managing AWS environments.
Preferred Qualifications :
- AWS Certified Data Analytics, Developer, or Solutions Architect certification.
- Experience with streaming data technologies such as Kinesis or Kafka for real-time data ingestion.
- Knowledge of serverless computing and experience with AWS Lambda, Step Functions, and DynamoDB.
- Familiarity with DevOps and automation tools (e.g., Jenkins, Git, Docker).
Soft Skills :
- Strong problem-solving and analytical thinking skills.
- Ability to work collaboratively in a fast-paced, cross-functional environment.
- Excellent communication skills to explain complex technical issues to both technical and non-technical stakeholders.
-
Solution Engineer
2 months ago
Bangalore, Karnataka, India HEVO DATA Full timeOverview :. We're seeking a talented Solutions Engineer with strong background in Data Domain to join our growing Sales team. You'll work one-on-one with the sales team to help discover our customers' business and technical needs and demonstrate how Hevo's platform can solve their real-time data replication needs. In this role, you will have...
-
Software Development Engineer II
2 months ago
Bangalore, Karnataka, India HEVO DATA Full timeAbout Hevo : Ever wondered, what's common between Zepto, Napster, Curefit, Groww, Jumio, Cars24 ? They all super-charge their business growth with Hevo Data Pipelines. How? 2 of every 3 modern day companies are unable to take data driven decision making due to fragmented data across 100s business software. Hevo is a simple, intuitive, and powerful...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Prydan Consultancy Full timeCompany Description : PRYDAN Consultancy Services is a global staffing solutions provider committed to excellence in recruitment services across various industries. With a focus on matching the right talent with the right opportunities, we empower clients and candidates to thrive in today's competitive landscape. Our comprehensive recruitment services...
-
Data Engineer
1 month ago
Bangalore, Karnataka, India Consero Global Solutions Full timeWe are seeking experienced, motivated and enthusiastic Data Engineering Specialists to join our team. As part of the team, you will be involved in the implementation of the ongoing and new initiatives for our company and contribute to our mission of delivering high-quality data-driven solutions to our clients. Role Description : The data engineering role...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Creencia Technologies Pvt Ltd Full timeJob Description:We are seeking a skilled Data Engineer to join our team. The ideal candidate will have a strong background in Data Engineering, with expertise in building and managing data pipelines. Proficiency in Scala programming, as well as experience with Java, GCP, and Spark, is essential for this role.Salary : 14-18 LPA Responsibilities :- Design,...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Coders Brain Technology Private Limited Full timeRole : Data Engineer (no one over 10 years of experience will be considered)Qualifications :- Bachelor's degree in Computer Science/Engineering or similar- 5 to 7 years of data engineering experience with focus on designing and implementing scalable big data transformations and pipelines- Proficiency with Cloud Platforms like Azure(preferred), AWS,...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Spigot Software Full timeResponsibility :- Design, develop, and maintain scalable data pipelines and analytics solutions using DBT, Snowflake, and related technologies- Collaborate with stakeholders to gather requirements and translate business needs into technical solutions- Develop efficient code with unit testing and code documentation- Ensuring accuracy and integrity of data and...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India BlueByte Technologies Full time1. Design and Develop ETL Pipelines :- Extract, transform, and load data from various sources into the data warehouse.- Ensure data integrity and quality throughout the ETL process.2. Data Warehousing :- Design and maintain data models, schemas, and database structures.- Optimize data storage and retrieval for performance and scalability.3. Complex SQL...
-
Lead Data Engineer
2 months ago
Bangalore, Karnataka, India Sentienz Full timeJob Description : As a Lead Data Engineer, you will play a pivotal role in driving our data-driven initiatives. You will be responsible for designing, developing, and maintaining robust data engineering solutions that support our analytics and decision-making processes.Responsibilities :- Lead and mentor a team of data engineers, providing technical guidance...
-
Head - Data Engineering
4 weeks ago
Bangalore, Karnataka, India Merck Group Full timeJob Description : Join us to contribute to the org's enterprise priority of driving innovation through data & digital. In R&D, the promise of bringing more medicines to more patients faster is closely linked with the ability to maximize the use of our data for improved decision making using data analytics & AI. For this, having high-quality data readily...
-
Data Engineer
4 weeks ago
Bangalore, Karnataka, India HRwing Full timeJob Description :Role & Responsibilities :- Design, develop, and maintain scalable data pipelines to extract, transform, and load (ETL) data from various sources into our data warehouse- Optimize and tune SQL queries for improved performance and efficiency- Collaborate with data scientists, analysts, and product managers to understand data requirements and...
-
Senior Data Engineer
2 months ago
Bangalore, Karnataka, India Newbies Staffing Full timeRole : Senior Data EngineerExperience : 8 - 10 YearsEducation : B.E/Master's (CS/IT)Location : BangaloreJob Overview :Data Engineers are responsible for developing, optimizing, and standardizing data assets/new projects which add value to our company. Our Data Engineer carry a unique position whereby they are involved in all aspects of the data assets...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Easy Recruitz Full timeData engineer : Hadoop. Location : Bangalore (Hybrid). We are looking for an Engineer II to join our growing team of data experts. This role is responsible for developing, maintaining and optimizing data pipelines, delivering data timely to our cross-functional teams and support reporting and analytics. The Engineer will work closely with product partner,...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Coders Brain Technology Private Limited Full timeQualifications : - Bachelor's degree in Computer Science/Engineering or similar - 5-7 years of data engineering experience with focus on designing and implementing scalable big data transformations and pipelines - Proficiency with Cloud Platforms like Azure(preferred), AWS, and/or Google cloud, including knowledge of core services like Azure: Data...
-
Data Engineer
4 weeks ago
Bangalore, Karnataka, India JoulestoWatts Business Solutions Pvt Ltd Full timeCompany Description : JoulestoWatts Business Solutions is a global IT services and consulting enterprise headquartered in Texas, USA. With over 4,000 professionals serving 200+ clients across 5 countries, we are recognized as a certified Great Place to Work (GPTW). Our focus is on delivering state-of-the-art solutions across diverse domains. Role Description...
-
Bangalore, Karnataka, India Cloudcover Consultancy Private Limited Full timeJob brief : We are seeking a highly skilled and experienced Engineering Manager Data Insights to join our team who has at least 10 years of experience in data engineering and data science, with significant experience in managing teams and delivering large scale projects. This role will be pivotal in driving the strategy, design, and implementation of our...
-
Senior Data Engineer
2 months ago
Bangalore, Karnataka, India MonoSpear Technologies Pvt Ltd Full timeSenior Data Engineer :- 9+ years' work experience and bachelor's degree in Information Science, Computer Science, Mathematics, Statistics or a quantitative discipline in science, business, or social science.- Hands-on engineer who is curious about technology, should be able to quickly adopt to change and one who understands the technologies...
-
Data Pipeline Engineer
1 month ago
Bangalore, Karnataka, India TekPillar Full timeJob Title : Data Pipeline EngineerLocation : BangaloreExperience : 4-5 YearsAbout the Role :Are you a skilled Data Pipeline Engineer looking to take your career to the next level? Join our dynamic team and work on cutting-edge data engineering solutions in a fast-paced environment! You'll have the opportunity to design, implement, and optimize data...
-
Data Engineer
2 months ago
Bangalore, Karnataka, India Arting Digital Full timePosition : Data Engineer Experience : 2+ YrsLocation : Jaipur, BangaloreNotice Period : 1 Month (Max)Primary Skills : SQL, Python, Databricks, Spark. Good to have : MongoDB, Dataiku DSS, PysparkResponsibilities of the role include : - Understand business requirements to develop and implement solutions that address business needs and adhere to big data...
-
MetroMax - Data Engineer
2 months ago
Bangalore, Karnataka, India The Metromax Group Full timeResponsibilities : 1. Data for AI :- Design and implement data pipelines specifically tailored for AI training and model- development.- Partner with AI data scientists and engineers to understand their data needs and ensure- efficient data access.- Implement data pre-processing and feature engineering techniques for optimal model- performance.2. AWS...