Senior Data Engineer

The Data Engineer should have strong data infrastructure and data architecture skills, a proven track record of administration for data engineering teams, strong operational skills to drive efficiency and speed, strong project management skills, and a core value base for how analytics can proactively improve organizations. You will be part of the Data & Analytics team and will partner closely with a team of data scientists, business analysts & business intelligence leading a cloud-based strategy for our clients.About the RoleYou'll work on our Azure based data pipeline projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data to turn information into insights using multiple technology platforms. Therefore, this role requires an understanding of how a secure data cloud environment is architected to gain real insights faster, with less friction and complexity. Ideal candidate should be a champion of building a distributed and scalable architecture that is metadata driven. You're also a detail-oriented person that believes data quality should never be sacrificed.Mandatory Skills: Python, SQL, Snowflake, CloudExperience - 5 to 7 yrsLocation - BangaloreWork Mode - HybridResponsibilities:- Master the use of technology & modern tools to optimize solutions and continuous process improvements - Facilitate team to design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation - Help build out and support an Enterprise Data Marketplace - Administer the Big data platform on the cloud - Implement automation of jobs and testing - Optimize the data pipeline to support ML workloads and use cases - Support mission critical applications and near real time data needs from the data platform - Build and support reusable framework to ingest, integration and provision data - Capture and publish metadata and new data to subscribed users - Work collaboratively with product managers, data scientists as well as business partners and actively participate in design thinking session - Motivate, coach, and serve as a role model and mentor for other development team associates/members that leverage the platform - Establish relationship with key business lines, Infosec and infrastructure team - Ensure security is integrated into all cloud services. - Build, monitor, and manage the cloud infrastructure. - Monitor and optimize cloud workloads. - Review current and future Azure implementations and suggest ways to improve and standardize the cloud offerings.Qualifications- Bachelor’s degree in Computer Science, Physics, Mathematics, Statistics or other Engineering disciplinesRequired Skills:- 5+ years' experience as Data engineer - Experience with big data tools: Spark, Databricks - Experience with building relational SQL Data Warehouses - Good Hands on experience with data pipeline and workflow management tools: Azure Data Factory, SSIS - Experience with object-oriented/object function scripting languages: Python, C#, Scala, VBA, etc. - Experience with systems scripting languages: Batch Scripts, Powershell - Experience with Git/GitHub and automating builds - Strong interpersonal, verbal and written communication skills and ability to present complex technical/analytical concepts to executive audience - Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management and deliveryPreferred Skills:- Experience building distributed architecture and IoT in a Dev/Ops environment - Experience in both Agile and Waterfall methodologies - Experience building and optimizing claims data pipelines, architectures/data models and data sets. - Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement. - Experience with the Agile workflow, Test Driven Development, and continuous improvement/continuous deployment. - Strong analytic skills related to working with structured and unstructured datasets. - Build processes supporting data transformation, data structures, metadata, dependency and workload management using Azure Automation, Azure Function Apps, and Azure Data Factory. - A successful history of manipulating, processing and extracting value from large disconnected datasets. - Working knowledge of Azure Automation, Azure Data Factory, and highly scalable 'big data' data stores such as Azure Blob Storage, Azure Data Lake - Experience supporting and working with cross-functional teams in a dynamic environment. - Excellent planning, prioritization, problem solving skills. - Experience with any of the following scripting languages, R, JSON, Parquet, JavaScript, Regex, PowerShell, shell scripting. - Experience with Data Bricks, HDInsight, Python.