Big Data Engineer
2 months ago
Job Description :
As a Big Data Engineer (Azure), you will build and learn about a variety of analytics solutions & platforms, data lakes, modern data platforms, data fabric solutions, etc. using different Open Source, Big Data, and Cloud technologies on Microsoft Azure.
On a typical day, you might :
- Design and build scalable & metadata-driven data ingestion pipelines (For Batch and Streaming Datasets)
- Conceptualize and execute high-performance data processing for structured and unstructured data, and data harmonization
- Schedule, orchestrate, and validate pipelines
- Design exception handling and log monitoring for debugging
- Ideate with your peers to make tech stack and tools-related decisions
- Interact and collaborate with multiple teams (Consulting/Data Science & App Dev) and various stakeholders to meet deadlines, to bring Analytical Solutions to life.
What do we expect?
- 4 to 9 years of total IT experience with 2+ years in big data engineering and Microsoft Azure
- Experience in implementing Data Lake with technologies like Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database
- A comprehensive foundation with working knowledge of Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB, and Purview
- A passion for writing high-quality code and the code should be modular, scalable, and free of bugs (debugging skills in SQL, Python, or Scala/Java).
- Enthuse to collaborate with various stakeholders across the organization and take complete ownership of deliverables.
- Experience in using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search
- Adept understanding of different file formats like Delta Lake, Avro, Parquet, JSON, and CSV
- Good knowledge of building and designing REST APIs with real-time experience working on Data Lake or Lakehouse projects
- Experience in supporting BI and Data Science teams in consuming the data in a secure and governed manner
- Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE) are valuable addition
- Mandatory : Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database
- Optional : Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB and Purview.
- Strong programming, unit testing & debugging skills in SQL, Python or Scala/Java. Some experience of using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search.
- Good Understanding of different file formats like Delta Lake, Avro, Parquet, JSON and CSV. Experience of working in Agile projects and following DevOps processes with technologies like Git, Jenkins & Azure DevOps.
Good to have :
- Experience of working on Data Lake & Lakehouse projects
- Experience of building REST services and implementing service-oriented architectures.
- Experience of supporting BI and Data Science teams in consuming the data in a secure and governed manner.
- Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE)
-
Big Data Engineer
2 months ago
Chennai/Hyderabad/Bangalore/Coimbatore/Mumbai/Delhi/Pune, IN GAIUS HYPERLOCAL PRIVATE LIMITED Full timeJob Description : Big Data Engineer - Scala, SparkYears of exp: 6 to 13 yearsJob location: Chennai / Hyderabad / Bangalore / Pune / Mumbai / Delhi / CoimbatoreAbout the Role :We are seeking a highly skilled Big Data Engineer to join our team and help us leverage the power of big data to drive innovation and business value. As a Big Data Engineer, you will be...
-
Senior Big Data Engineer
4 weeks ago
Mumbai/Pune/Kolkata/Delhi/Nagpur/Ahmedabad, IN AEXONIC TECHNOLOGIES PRIVATE LIMITED Full timePosition - Senior Big Data Engineer (GCP)Experience - 5 to 8 YrsLocation - RemoteShift - EST Shift 5:30 PM to 2:30 AMWe are seeking a highly skilled Senior Big Data Engineer with extensive experience in Big Data technologies, including Hadoop, Hive, PySpark, and expertise in Google Cloud Platform (GCP) and BigQuery. The successful candidate will be...
-
Big Data Engineer
2 months ago
Chennai/Pune, IN Pan Asia Man power Consultants Recruiting agency Full timeJob Title : Big Data EngineerExperience Range : 6+ YearsLocation : Chennai/Pune (Hybrid - 2 Days Work From Office in a Week)Job Description :We are seeking a skilled Big Data Engineer to join our Agile team and contribute to building backend data applications and services that support our analytics platforms. This role requires a hands-on developer with...
-
Big Data Engineer
2 months ago
Bangalore/Pune/Mumbai/Chennai, IN a client of change leaders Full timeJob Description : - 5+ years of experience developing scalable Big Data applications or solutions on distributed platforms- Design and implement data pipelines using Snowflake, PostgreSQL, and Python.- Experience in AWS/ Azure/ GCP and good to have other cloud platform tools- Experience working with Big Data platforms, including Hadoop- Hive, Data Bricks...
-
Data Engineer
1 month ago
Delhi NCR/Bangalore/Hyderabad/Chennai, IN ANLAGE Full timeJob Overview : We are seeking a highly skilled and motivated Scala Data Engineer to join our team. In this role, you will design, build, and maintain large-scale data processing systems. The ideal candidate will have a strong background in Scala programming, distributed data processing frameworks, and a deep understanding of data architecture and pipelines....
-
Big Data Developer
2 months ago
Hyderabad/Bangalore/Chennai/Bhubaneshwar, IN Rigel networks Full timeJob Title : Big Data/Spark Scala DeveloperLocation : Type : Full-TimeJob Overview : We are looking for an experienced Big Data/Spark Scala Developer to join our team. The ideal candidate will have a deep understanding of big data technologies and strong programming skills in Scala. In this role, you will be responsible for designing, developing, and...
-
Data Engineer II
2 months ago
Bangalore/Pune/Mumbai/Coimbatore/Delhi NCR/Delhi/Navi Mumbai, IN Concept Management Full timeRole - Data Engineer 2Job Type - Full-time, Permanent Qualification - BE or BtechNumber Of Roles - 15Location - Bangalore, Pune, Mumbai, Delhi, Gurugram, CoimbatorePackage - 15 to 20 LPAExperience - 3.5 to 6 YearsShift - UK Shift15 to 30 Days JoinerSkills : Data Engineering, Big data, ETL, Python, PySpark and SparkSQ, SQL, JIRA, Git, and GitHub.Job Summery...
-
Program Lead/Technology Consultant
1 month ago
Chennai/Bangalore/Hyderabad, IN Resourcetree Full timeAbout the role :As Program Lead - Technology Consulting, you will be key member of the teams working on end-to-end AI and analytics solutions being built for our clients. While your primary focus will be on the data engineering and application engineering components of the overall solution, you will collaborate with internal data science teams for the ML...
-
PySpark/Databricks Engineer
6 months ago
Anywhere in India/Multiple Locations/Hyderabad/Srinagar/Jaipur, IN Aricent Full timeJob : PySpark/Databricks EngineerOpen for Multiple Locations with WFO and WFHJob Description :We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims to build a data standardized and curation-based Hadoop clusterThis high visibility, fast-paced key...
-
Big Data Developer
2 months ago
Pune/Gurgaon/Gurugram, IN Talent Toppers LLP Full timeResponsibilities :- Undertake complete ownership in accomplishing activities and assigned responsibilities across all phases of project lifecycle to solve business problems across one or more client engagements;- Apply appropriate development methodologies (e.g.: agile, waterfall) and best practices (e.g.: mid-development client reviews, embedded QA...
-
Data Engineer
2 months ago
Delhi NCR/Bangalore/Hyderabad/Pune, IN GENPACT India Private Limited Full timeJob Description :- Experience in IT industry- Working experience with building productionized data ingestion and processing data pipelines in Snowflake- Fully well-versed with data warehousing concepts.- Expertise and excellent understanding of Snowflake features and integration of Snowflake with other data processing.- Able to create the data pipeline for...
-
Data Engineer
6 months ago
Pune/Hyderabad, IN EDGESOFT Full timeJob Description :The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.Responsibilities :- Design,...
-
Senior Data Engineer
1 month ago
Delhi NCR/Gurgaon/Gurugram, IN AJ Consulting Full timeTitle : Senior Data EngineerJob Location : GurgaonEducation : B.TECH / MTECH Experience : 5 + Payroll : Nippon Data Systems LtdNIPPON DATA SYSTEMS LIMITED is a provider of IT enabled business solutions which provide our clients competitive business advantage. We partner with our customers to deliver business solutions to their satisfaction and help them...
-
Azure Data Engineer
2 months ago
Pune/Hyderabad/Bangalore/Chennai, IN BSRI Solutions Full timeJob Description :We are seeking a talented and experienced Azure Data Engineer to join our team. The ideal candidate will have a strong background in data engineering with a focus on Microsoft Azure services. The candidate should have a minimum of 5-10 years of hands-on experience with Azure Data Factory (ADF), Azure Synapse Analytics, Synapse Pipelines,...
-
Big Data Team Lead
2 months ago
Bangalore/Chennai/Pune, IN Rigel networks Full timeJob Title : Senior Software Developer / Applications Development ManagerLocation : Pune, Chennai, Bangalore Shift Timings : 10 AM - 7 PMPosition Overview : We are seeking an experienced Senior Software Developer / Applications Development Manager with a strong background in software development and project management. The ideal candidate will possess...
-
Data Engineer
4 weeks ago
Noida/Bangalore/Chennai/Hyderabad/Coimbatore, IN Huntsmen & Barons Full timeRole : Data Engineer - Apache spark/Spark/Scala/pysparkJob Description :- Relevant Experience 5 years.- The indent is created for Apache spark. But the actual demand is for Spark/ Scala.- Expertise in Data Engineering.- Hands-on experience in design and development of big data platform.- Deep understanding of modern data processing technology stacks: Spark,...
-
Data Modeler
2 months ago
Bangalore/Hyderabad/Chennai/Pune, IN Hashone Careers Private Limited Full timeRole Overview :We are seeking an experienced Data Modeller with a strong foundation in data architecture, modelling, and analysis. The ideal candidate will be proficient in Erwin Data Modelling and have a deep understanding of data quality, data lineage, and data cataloging. You will play a crucial role in designing and optimizing data models for both OLTP...
-
KiE Square Analytics
2 months ago
Delhi NCR/Bangalore/Chennai, IN KiE Square Analytics Full timeJob Description : KiE Square is a leading AIML Company that provides advanced Data Science/Data Engineering Solutions to its clients and drives their Business Intelligence Transformation.We invite you to join our Core Team of Data Scientists, Data Engineers and AIML Developers who have a sharp analytical bent, have delivered multiple high value projects...
-
ZS - Engineering Lead - Big Data
2 months ago
Pune/Gurgaon/Gurugram/Bangalore, IN ZS Full timeComapany Description :ZS is a place where passion changes lives. As a management consulting and technology firm focused on transforming global healthcare and beyond, our most valuable asset is our people. Here you'll work side-by-side with a powerful collective of thinkers and experts shZS is a place where passion changes lives. As a management...
-
Hero FinCorp
4 weeks ago
Delhi NCR/Bangalore, IN Hero Fincorp Ltd Full timeResponsibilities for the job :Key Responsibilities :- Architectural Leadership: Design and implement scalable data architectures using Databricks, ensuring they meet the specific needs of financial services such as risk management, compliance, and customer analytics.- Conduct assessment of existing data architectures, identify areas of improvement, and...