Data Engineer
4 weeks ago
Azure Data Engineer :
We are seeking a highly skilled Azure Data Engineer to join the data engineering team and play a crucial role in an optimization initiative. The current problem is that read speed is compromised and faster responses from delta are required.
The ideal candidate should have extensive experience in Python programming and be proficient in PySpark and Databricks. The selected candidate will be responsible for designing and implementing efficient data pipelines and solutions for data-driven projects in Azure Data Factory and Azure Databricks
Responsibilities :
- Design, build, and maintain scalable data pipelines using PySpark and Databricks
- Optimize data processing and storage for maximum performance and efficiency
- Troubleshoot and debug data-related issues, and implement solutions to prevent reoccurrence
- Collaborate with data scientists, software engineers, and other stakeholders to ensure that data solutions are aligned with business goals
Requirements :
- Strong experience in Python programming and PySpark, and SparkSQL
- Clear understanding of Spark Data structures, RDD, Dataframe, dataset
- Expertise in Databricks and ADLS
- Expertise handling data type, from dictionaries, lists, tuples, sets, arrays, pandas dataframes, and spark dataframes
- Expertise working with complex data types such as, structs, and JSON strings.
- Clear understanding of Spark Broadcast, Repartition, Bloom index filters
- Experience with ADLS optimization, partitioning, shuffling and shrinking
- Ideal experience with disk caching
- Ideal Experience with cost based optimizer
- Experience with data modeling, data warehousing, data-lake, delta-lake and ETL/ELT processes in ADF
- Strong analytical and problem-solving skills
- Excellent documentation, communication and collaboration skills
- Experience of Leading team of engineers and have mentored them
-
AMAZATIC - Data Engineer/Senior Engineer - Azure
4 weeks ago
Hyderabad/Pune, IN AMAZATIC SOLUTIONS LLP Full timeSenior AZURE DE Our Values : Passion, Continuous Learning, Adaptability, Teamwork, Customer Centricity, ReliabilityPosition : Data Engineer / Senior Data EngineerExperience : 8+ YearsKey Skills :- Minimum 5Y+ on Azure cloud services.- Proficiency in using Azure Synapse, Azure Storage, and Azure Data Factory for building data solutions.- Strong understanding...
-
Data Engineer
2 weeks ago
Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full timeRole : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...
-
Data Engineer
24 hours ago
Hyderabad/Chennai/Pune, IN Freelancer HR Full timeDescription :Job Description :- Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Data bricks and/or HDInsights- Experience with other Open Source big data products Hadoop (incl. Hive, Pig, Impala)- Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,...
-
Data Engineer
4 weeks ago
Pune/Hyderabad, IN Huquo Full timeMust-Have :- 4+ years of experience in Data Engineering experience on Azure and Databricks- Strong knowledge of Databricks, Azure Data Factory, Azure Blob, and Azure SQL is required- Experience Developing and tuning Spark Application and SparkSQL queries-- Expertise to Implementing Data warehousing Solutions - Good understanding of Azure Databricks platform...
-
Data Engineer
4 weeks ago
Pune/Hyderabad/Remote, IN HARP Technologies and Services Full timeJob Location : Pune & Hyderabad (Initial 1 -2 months hybrid and then complete remote job. Client will offer accommodation(stay), food & travel exp. )Exp range : 6+ years Shift timings : General 9am - 6pm IST or 10am - 7pm IST Mandatory skills : Data engineering (4+years), Databricks (3.5+ years), Pyspark (3+ years), Python (preferred) or bash, Data pipeline,...
-
Azure Data Engineer
2 weeks ago
Bangalore/Hyderabad/Pune, IN Kezan Consulting Full timeWork Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...
-
Python Data Engineer
4 weeks ago
Bangalore/Hyderabad/Pune/Mumbai/Chennai, IN Swift Strategic Staff Solutions INC Full timeJob Description :We are seeking a talented and experienced Python Data Engineer to join our data team and play a crucial role in building, maintaining, and optimizing data pipelines for our cloud-based data warehouse.Responsibilities :- Design, develop, and implement data pipelines for ingesting, transforming, and loading data into Snowflake using Python...
-
Full Stack Data Engineer
4 weeks ago
Pune/Hyderabad, IN Idyllic Services Pvt Ltd Full timeAbout the job :Job Description :Job Title : Fullstack Data EngineerExperience : 7+ Yrs,Location : Pune, Hyderabad.Full Stack Data Engineer role is expected to own and manage end to end activities within the team. Person would be entrusted to developed solutions/design ideas related to Data Engineering, service upkeep, developing new capabilities and identify...
-
AWS Data Engineer
4 weeks ago
Pune/Hyderabad, IN Huquo Full timeRole : Data EngineerMust-Have : - 4 to 6 years of Relevant experience in Data Engineering and delivery.- 3 to 4 years of relevant work experience in large-scale Data applications- AWS Glue is mandatory.- Strong 4-6 years of working experience in SQL Spark Python, Snowflake, and AWS cloud knowledge.- Good aptitude, strong problem-solving abilities, and...
-
Kanerika Software
3 weeks ago
Hyderabad/Ahmedabad, IN KANERIKA SOFTWARE PRIVATE LIMITED Full timeRole :As a Data Engineer Lead / Power BI Lead, your role will involve designing, developing, and implementing interactive dashboards and reports using Power BI. You will work closely with stakeholders to gather requirements and translate them into effective data visualizations that provide valuable insights. Additionally, you will be responsible for...
-
Data Engineer
4 weeks ago
Pune/Mumbai/Hyderabad/Chennai/Bangalore, IN RapidBraiins Full timePosition Overview :As a Talend - Data Engineering Specialist, you will play a key role in designing, developing, and maintaining data pipelines using Talend, AWS, and SQL. You will work closely with cross-functional teams to gather requirements, design data solutions, and implement scalable and efficient ETL processes. Additionally, you will leverage your...
-
CGI - Data Engineer - Data Pipeline
4 weeks ago
Bangalore/Hyderabad/Chennai/Pune, IN CGI Information Systems and Management Consultants Full timeRequired Skills :- 5+ years of experience in database administration duties- Bachelor's degree (or equivalent) in computer science, information technology, engineering, or related discipline- Experience in building or maintaining ETL processes.- Experienced in working with big data analytics tools.- Need to be proficient in various database types and...
-
Lead Data Engineer
4 weeks ago
Bangalore/Hyderabad/Pune, IN iMind Your Business Solutions Private Limited Full timeJob Description :We are hiring Lead Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune)Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...
-
Senior Data Engineer
4 weeks ago
Hyderabad/Pune/Bangalore, IN iMind Your Business Solutions Private Limited Full timeJob Description :We are hiring Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune).Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...
-
AWS Data Engineer
2 weeks ago
Bangalore/Pune/Hyderabad, IN Kezan Consulting Full timeCompany Overview : Kezan Consulting is a leading provider of Human Resources Services in the industry. We specialize in providing comprehensive HR solutions to our clients, helping them achieve their goals and drive business success. With a team of highly skilled professionals and a commitment to delivering excellence, we have established ourselves as a...
-
Lead Data Engineer
3 weeks ago
Pune/Hyderabad/Gurgaon/Gurugram, IN LnS Infotech Full timePosition : Lead Data EngineerLocation : : 8+ YearsRoles & Responsibilities : This role is a hybrid across Data Engineering skill sets and DBA. Ideally someone with a background in Oracle DBA roles who has branched into data engineering.- 8+ years of hands-on experience in SQL, Py Spark & Python, should be experienced in shell scriptingJob Description :-...
-
Data Engineer
1 week ago
Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Mumbai/Ahmedabad, IN Sterlite Technologies Limited Full timeRole as a Data engineer :- Develop and build AI/ML model ready data ponds using Azure Databricks Delta lake- Improve upon the data ingestion models, ETL jobs, and alarm to maintain data integrity and data availability.- Partner with AI/ML product teams proactively identify the data needs for rapid experimentation, stable production deployment and ensure the...
-
Principal Salesforce Engineer
24 hours ago
Pune/Mumbai/Hyderabad, IN Zallery Full timeRole : Principal Salesforce Engineer (Data Migration)Location : Pune, Hyderabad, or Mumbai - IndiaRole Overview :We are seeking a Principal Salesforce Engineer with a deep specialization in data migration to architect and lead our Salesforce data migration initiatives. This strategic role focuses on ensuring that our data migration processes are seamlessly...
-
AI/ML Engineer
4 weeks ago
Bangalore/Pune/Hyderabad/Chennai, IN Nilasu consulting Full timeSummary :We are seeking a talented and passionate AI/Machine Learning Engineer to join our team. You will play a key role in the entire lifecycle of machine learning projects, from data engineering and classical machine learning to potentially exploring deep learning and natural language processing techniques. Your strong foundation in Python coding,...
-
Senior Data Engineer
24 hours ago
Pune/Mysore, IN IT Full timeWe are seeking a talented Senior Data Engineer with a strong background in developing ETLs and data pipelines using Apache Spark with the PySpark library. The ideal candidate will have proven experience in data modeling for data warehousing use cases, designing patterns, and building highly scalable and secure solutions. Additionally, experience with Azure...