Data Engineering Azure databricks
3 weeks ago
Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.
Responsibilities:
- Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
- Create the data integration and data diagram documentation.
- Lead the data validation, UAT and regression test for new data asset creation.
- Create and maintain data models, including schema design and optimization.
- Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.
Qualifications and Skills:
- Strong knowledge on Python and Pyspark
- Expectation is to have ability to write Pyspark scripts for developing data workflows.
- Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum
- Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.
- Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
- Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.
- Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.
- Expectation is to have strong problem-solving and troubleshooting skills.
- Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
- Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
- 4-7 years of experience in Data Engineer.
- Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
- Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
- Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.
-
Databricks Data Engineer
2 weeks ago
india, IN Insight Global Full timeInsight Global is seeking a Databricks Data Engineer in India with with 3–5 years of experience to support data engineering initiatives in the pharmaceutical domain. The ideal candidate will have hands-on expertise in Databricks, SQL, and Python, and a strong understanding of pharma/life sciences data. This role involves building and optimizing data...
-
Databricks Data Engineer
2 weeks ago
india, IN Insight Global Full time** Immediate Joiner **Insight Global is seeking a Databricks Data Engineer in India with with 3–5 years of experience to support data engineering initiatives in the pharmaceutical domain. The ideal candidate will have hands-on expertise in Databricks, SQL, and Python, and a strong understanding of pharma/life sciences data. This role involves building and...
-
Data Engineer with Databricks
2 weeks ago
india, IN KPG99 INC Full timeJob Title: Databricks Engineer Location: RemoteDuration: 12 months Contract with extensions Job Description: REQUIRED SKILLS AND EXPERIENCE- 3–5 years of experience in data engineering roles - Strong hands-on experience with Databricks for data processing and pipeline development. - Proficiency in SQL for data querying, transformation, and troubleshooting....
-
Engineering Manager – Big Data
3 weeks ago
india, IN DataE2E Technologies Full timeJob Title: Engineering Manager – Big Data, Azure & DatabricksLocation: RemoteEmployment Type: Full-timeDepartment: Data Engineering / TechnologyExperience: 8+ years (3+ years in a managerial role)Salary - 30 LPAAbout the Role:We are seeking a highly experienced and strategic Engineering Manager with deep expertise in Big Data platforms, Microsoft Azure,...
-
Azure Data Engineer
2 weeks ago
Bangalore Urban, Karnataka, India, IN Tata Consultancy Services Full timeTCS present an excellent opportunity for Azure Data EngineerJob Location: Pan IndiaExperience required : 6-12 yrsInterview date: 18-Oct-25 (Saturday)Skillset:Strong SQL/PL SQL knowledge (Development/Analytical)Good at Azure Data Engineering( Databricks/Data Factory)Must have Airline domain expertise ( Revenue Accounting/Booking/Inventory/Pricing)Knowledge of...
-
Azure data Engineer
3 weeks ago
Bhubaneswar, Odisha, India, IN Tata Consultancy Services Full timeRole: Azure data EngineerExperience: 4Location: BhubaneswarMust-Have4+ years of hands-on experience in development.Azure – cloud technology – Function Apps, API, Azure DevOps, Databricks, Data Factory, SynapseStrong knowledge Incident, problem, Change management, service management, service request.Must have hands on experience code versioning tool...
-
Data Engineer
3 weeks ago
india, IN Vriba Solutions Full timeRole: Data Engineering Remote Looking for 5-10 years of Exp• Design, develop & maintain ETL/ELT pipelines• Ingest & transform data from APIs, DBs, files, streams• Build real-time & batch processing solutions• Data validation, quality & cleansing• Translate business needs into data models• Ensure data security, access control & compliance•...
-
Data Engineer
2 weeks ago
india, IN Keasis Full timeWe are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS/Hive/Impala) into Apache Iceberg tables, with downstream integration into Snowflake and Databricks. The ideal candidate will have hands-on experience with modern data lakehouse architectures and will play a...
-
Databricks Engineer Remote Job Offshore
2 weeks ago
india, IN KPG99 INC Full timePosition : Databricks EngineerLocation: REMOTEDuration: 12 months with extensions REQUIRED SKILLS AND EXPERIENCE- 3–5 years of experience in data engineering roles - Strong hands-on experience with Databricks for data processing and pipeline development. - Proficiency in SQL for data querying, transformation, and troubleshooting. - Solid programming skills...
-
Contract role
2 weeks ago
india, IN KPG99 INC Full timeRole: Databricks EngineerLocation: REMOTEDuration: 12 months with extensionsREQUIRED SKILLS AND EXPERIENCE- 3–5 years of experience in data engineering roles- Strong hands-on experience with Databricks for data processing and pipeline development.- Proficiency in SQL for data querying, transformation, and troubleshooting.- Solid programming skills in...