Senior Data Engineer-10+years, ADF, Snowflake, Databricks
3 weeks ago
At Streamline, we are experts in Enterprise Mobility, Product Engineering, and IT Transformation. We help organizations navigate the constantly evolving landscape of IT. Our sole focus is ensuring that our client’s organization is armed with the strategies, products and solutions that are transformative to their business. Streamline works closely with our clients, takes pride in developing genuine relationships and embraces open communication and collaboration with our clients. We become a part of our client’s team, working together to achieve short-term goals and enable long-term success. Our team is comprised of world-class strategists, architects, engineers, and developers.
Role Summary
The Data Engineer will work with a team of other software developers to define, design, develop, integrate, and re-engineer the Enterprise Data warehouse. Data Marts, Data Virtualization and Data Visualization components in different environments which meet customers’ analytical and business intelligence requirements, scales easily and supports deployment in highly available environments. The Senior engineer acts as the development and technical lead and individual contributor on complex projects, contributing to strategic vision and technical decisions, participating in vendor analysis and selection projects, introducing process improvements, completing a proof-of-concept projects for the introduction of changes to our architecture, and providing oversight on the work of other technical staff.
Role Responsibilities
- Build data-intensive solutions that are highly available, scalable, reliable, secure, and cost-effective- Create and maintain highly scalable data pipelines using Databricks, Azure, AWS, Kafka.- Design and build ETL/ELT data pipelines to extract and process data from variety of external/internal data sources.- Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery- Build data insights, data analytics, ML models, fraud and anomaly detection using Snowflake- Build and deploy modern data solutions.- Define, implement and build jobs to populate data models.- Relational and NoSQL Database management- DevOps building CI/CD pipelines
Technical System Expertise: Understands data warehouse development practices and processes including ETL design, Dimensional models, slowly changing dimensions, Data Security components, Data quality methods, how MPP databases operate and data flows. Aware of current technology benefits. Expected to independently develop full stack ETL solution from source to staging to presentation layers. Understands the building blocks, interactions, dependencies, and tools required to complete BI Data warehousing software and automation work. An Independent study of current technology is expected. Interact with system engineers to define continuous delivery and/or DevOps solutions, data security and/or necessary requirements for automation.
Technical Engineering Services: Supports BI Data Warehousing and Enterprise Data Solutions projects by analyzing, designing, and developing ETL solutions; conducting tests and inspections; creating BI reports and virtualization components. Create interface jobs/APIs for data sharing with internal and external stakeholders Ensure we use accurate and secure methods to extract data. Analyze Big Data and MPP Databases to discover trends and patterns. Participates in reviews (walkthroughs) of technical specifications and program code with other members of the DevOps team. Expected to supervise associate engineers on occasion.
Innovation: Presents new ideas which improve existing BI Data Warehousing systems/processes/services. Presents new ideas which utilize new frameworks and reusable components to improve existing ETL jobs/processes/services. Express new perspectives based on an independent study of the industry. Review current company processes to highlight questions that may drive process refinement and optimization.
Technical Writing: Maintains knowledge of existing technology documents. Writes basic documentation on how technology works. Contributes clear documentation for new code and systems used. Documenting systems designs, presentations, and business requirements for consumption at the engineer level. Develop application support documentation as required by the application support teams for acceptance of systems changes into production.
Technical Leadership: Collaborates with other engineering, development teams and utilizes data engineering/analyst expertise to deliver technical solutions. Continuously learn and mentor on new technologies.
Qualifications & Skills
Bachelor’s degree in CS, Statistics, Information systems or equivalent experience.
Strong expertise working with relational databases. Exceptional ability to author and optimize complex SQL queries/workflows Strong analytical skills to work with unstructured data.
Experience building and optimizing highly scalable data pipelines, architectures, and data sets.
A successful history of manipulating, processing, and extracting value from large, disconnected datasets and provide valuable insights.
Proficiency in workflow orchestration (Databricks, Azure data factory).
Experience working with streaming data solutions such as Kafka, IoT hub and Spark Streaming.
Working experience building insights, ML models, fraud and anomaly detection using Snowflake/Databricks.
Experience working with NoSQL databases like MongoDB, Cassandra, Cosmos DB
Proficiency in Java/Python/Scala, pandas, pySpark, NumPy
8+ years of ETL Development experience using Azure Databricks, Azure data factory, SSIS, Talend.
8+ years Professional experience in designing and building cost-effective large scale data marts, data warehousing, distributed big data processing MPP Systems (Databricks, Spark, Snowflake)
5+ years of Expertise in Logical and Physical Data Model design using various modelling Tools like Erwin 7.3 and Power Designer.
5+ years of experience with Azure and AWS cloud stack
3+ years BI Visualization experience developing reports using Snowflake, PowerBI, Grafana, Qlik and analytic cubes utilizing SQL Server Analysis Services (SSAS).
Preferred Skills
- Certifications- DevOps experience- Bash, Golang scripting experience- Docker/Kubernetes experience- CI/CD pipelines
-
delhi, India Streamline Full timeWho We AreAt Streamline, we are experts in Enterprise Mobility, Product Engineering, and IT Transformation. We help organizations navigate the constantly evolving landscape of IT. Our sole focus is ensuring that our client’s organization is armed with the strategies, products and solutions that are transformative to their business. Streamline works closely...
-
Delhi, India Streamline Full timeWho We AreAt Streamline, we are experts in Enterprise Mobility, Product Engineering, and IT Transformation. We help organizations navigate the constantly evolving landscape of IT. Our sole focus is ensuring that our client’s organization is armed with the strategies, products and solutions that are transformative to their business. Streamline works closely...
-
Data Engineer
2 months ago
delhi, India VSquare Systems Pvt. Ltd. Full timeJob Title: Data Engineer (Azure Databricks)Location: Bangalore / HybridJob Type: Full-timeJob Description:Experienced Data engineer & Solutions designer with a deep understanding of data solution design and implementation, seeking a challenging role to leverage my expertise in data architecture, cloud migrations, and solution design, aiming to drive...
-
Data Engineer
1 week ago
Delhi, India VSquare Systems Pvt. Ltd. Full timeJob Title: Data Engineer (Azure Databricks)Location: Bangalore / HybridJob Type: Full-timeJob Description:Experienced Data engineer & Solutions designer with a deep understanding of data solution design and implementation, seeking a challenging role to leverage my expertise in data architecture, cloud migrations, and solution design, aiming to drive...
-
Data Engineer
3 weeks ago
delhi, India The Ksquare Group Full timeRequirementsBachelor's degree in computer science, information technology, or a related field.Prior experience with data lakes and data warehouses, working knowledge with dimensional modeling is must.Demonstrate advanced understanding in data engineering practices, Including ETL/ELT, data integrations, reusable data pipelines,data management methods.Advanced...
-
Data Engineer
1 week ago
Delhi, India tsworks Full timeWho We Are tsworks Technologies India Private Limited (subsidiary of The Software Works, Inc, USA) is a technology product and services company. Our mission is to provide domain expertise, innovative solutions and thought leadership to empower businesses to thrive in a digital world. We value our employees, take pride in providing best value in customer...
-
Delhi, India VSquare Systems Pvt. Ltd. Full timeJob Title: Data Engineer (Azure Databricks)Location: Bangalore / HybridJob Type: Full-timeJob Description:Experienced Data engineer & Solutions designer with a deep understanding of data solution design and implementation, seeking a challenging role to leverage my expertise in data architecture, cloud migrations, and solution design, aiming to drive...
-
Data Engineer II
1 month ago
Delhi, India tsworks Full timeWho We Aretsworks Technologies India Private Limited (subsidiary of The Software Works, Inc, USA) is a technology product and services company. Our mission is to provide domain expertise, innovative solutions and thought leadership to empower businesses to thrive in a digital world. We value our employees, take pride in providing best value in customer...
-
Data Engineer
2 days ago
new delhi, India Capleo Global Full timeLead Data EngineerExperience: 8-10 yearsLocation: BengaluruAvailability: Immediate to 30 daysMandatory Skills:SparkDatabricksPySpark or ScalaSQL & PythonGithubDWH (Snowflake or Presto or Hive or Hadoop)Data Pipeline & Data ModelWe are seeking an experienced Senior Data Engineer to join our dynamic team in Bengaluru. If you are passionate about shaping the...
-
Databricks Engineer
2 weeks ago
Delhi, India KniTT Full timeJob Responsibilities:Write pyspark, sql queries on databricksImplement data flows to load dataWrite transformations based on the business logic for a given usecasePerformance optimizationDesigning ETL frameworksSkill set:Essential-Overall Data Engineer ExpSound with Databricks concepts with all the latest features of it (should be the recent one).Proficient...
-
Snowflake Data Engineer
3 days ago
delhi, India Lapis Data Analytics Full timeCompany DescriptionLapis Data Analytics is a dynamic and innovative data engineering company founded in 2020 and located in Chennai. Our primary focus is on designing, building, and maintaining robust data pipelines that convert raw data into actionable insights. With a team of seasoned experts and cutting-edge technology, we deliver customized data...
-
Senior Partner Sales Engineer
5 days ago
New Delhi, India Snowflake Full timeBuild the future of data. Join the Snowflake team. We’re at the forefront of the data revolution, committed to building the world’s greatest data and applications platform. Our ‘get it done’ culture allows everyone at Snowflake to have an equal opportunity to innovate on new ideas, create work with a lasting impact, and excel in a culture of...
-
Data Engineer
1 week ago
Delhi, India VSquare Systems Pvt. Ltd. Full timeJob Title: Data Engineer (Azure Databricks) Location: Bangalore / Hybrid Job Type: Full-time Job Description: Experienced Data engineer & Solutions designer with a deep understanding of data solution design and implementation, seeking a challenging role to leverage my expertise in data architecture, cloud migrations, and solution design, aiming to drive...
-
Data Engineer
1 month ago
Delhi, India ValueLabs Full timeExperience -6+YearsPrimary Skills -ADF, SQL Server, Databricks NoteBooks, Data Pipelines, SPROCs, Data Warehouse, Databricks DataLakeSecondary Skills -PowerBl, Data modelingNotice Period - Immediate - 15 daysJob Description:The Data Engineer is responsible for designing, developing, and maintaining data pipelines and data warehouses to support the...
-
Data Engineer II
3 weeks ago
Delhi, India tsworks Full timeWho We Aretsworks Technologies India Private Limited (subsidiary of The Software Works, Inc, USA) is a technology product and services company. Our mission is to provide domain expertise, innovative solutions and thought leadership to empower businesses to thrive in a digital world. We value our employees, take pride in providing best value in customer...
-
Data Engineer
1 month ago
delhi, India ValueLabs Full timeExperience - 6+YearsPrimary Skills - ADF, SQL Server, Databricks NoteBooks, Data Pipelines, SPROCs, Data Warehouse, Databricks DataLakeSecondary Skills - PowerBl, Data modelingNotice Period - Immediate - 15 daysJob Description: The Data Engineer is responsible for designing, developing, and maintaining data pipelines and data warehouses to support the...
-
Senior Data Engineer
2 months ago
Delhi NCR, India Marktine Technology Solutions Full timeExperience : 5+ Years. Location : Remote. Timings : 3 PM to 12 AM IST. Technical Skills :- 5+ years working with SQL, Python and other programming languages.- 3+ years of building analytical solutions leveraging SQL and Python programming.- Strong problem-solving and communication skills.- Experience testing datasets for accuracy, implementing automated...
-
Architect : Big Data
2 weeks ago
Delhi, India Databricks Full timeAs a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...
-
Architect : Big Data / Ml
1 month ago
Delhi, India Databricks Full timeAs a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...
-
Azure Data Engineer
3 weeks ago
delhi, India Direct Client Full timeAzure Data Engineer - Fivetran or Snowflake SpecialistBengaluru, Data & AnalyticsJOB DESCRIPTION1. Design, develop, and deploy data pipelines and workflows using Fivetran or Snowflake on the Azure platform.2. Collaborate with business stakeholders to gather data requirements and translate them into technical specifications and Configure and customize...
