Senior Data Engineer

The Data Engineer should have strong data infrastructure and data architecture skills, a proven track record of administration for data engineering teams, strong operational skills to drive efficiency and speed, strong project management skills, and a core value base for how analytics can proactively improve organizations. You will be part of the Data & Analytics team and will partner closely with a team of data scientists, business analysts & business intelligence leading a cloud-based strategy for our clients.

About the Role

You'll work on our Azure based data pipeline projects with a focus on collecting, parsing, managing, analysing and visualizing large sets of data to turn information into insights using multiple technology platforms. Therefore, this role requires an understanding of how a secure data cloud environment is architected to gain real insights faster, with less friction and complexity. Ideal candidate should be a champion of building a distributed and scalable architecture that is metadata driven. You're also a detail-oriented person that believes data quality should never be sacrificed.

Mandatory Skills: Python, SQL, Snowflake, Cloud

Experience - 5 to 7 yrs

Location - Bangalore

Work Mode - Hybrid

Responsibilities:

• Master the use of technology & modern tools to optimize solutions and continuous process improvements
• Facilitate team to design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation
• Help build out and support an Enterprise Data Marketplace
• Administer the Big data platform on the cloud
• Implement automation of jobs and testing
• Optimize the data pipeline to support ML workloads and use cases
• Support mission critical applications and near real time data needs from the data platform
• Build and support reusable framework to ingest, integration and provision data
• Capture and publish metadata and new data to subscribed users
• Work collaboratively with product managers, data scientists as well as business partners and actively participate in design thinking session
• Motivate, coach, and serve as a role model and mentor for other development team associates/members that leverage the platform
• Establish relationship with key business lines, Infosec and infrastructure team
• Ensure security is integrated into all cloud services.
• Build, monitor, and manage the cloud infrastructure.
• Monitor and optimize cloud workloads.
• Review current and future Azure implementations and suggest ways to improve and standardize the cloud offerings.

Qualifications

• Bachelor’s degree in Computer Science, Physics, Mathematics, Statistics or other Engineering disciplines

Required Skills:

• 5+ years' experience as Data engineer
• Experience with big data tools: Spark, Databricks
• Experience with building relational SQL Data Warehouses
• Good Hands on experience with data pipeline and workflow management tools: Azure Data Factory, SSIS
• Experience with object-oriented/object function scripting languages: Python, C#, Scala, VBA, etc.
• Experience with systems scripting languages: Batch Scripts, Powershell
• Experience with Git/GitHub and automating builds
• Strong interpersonal, verbal and written communication skills and ability to present complex technical/analytical concepts to executive audience
• Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management and delivery

Preferred Skills:

• Experience building distributed architecture and IoT in a Dev/Ops environment
• Experience in both Agile and Waterfall methodologies
• Experience building and optimizing claims data pipelines, architectures/data models and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Experience with the Agile workflow, Test Driven Development, and continuous improvement/continuous deployment.
• Strong analytic skills related to working with structured and unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management using Azure Automation, Azure Function Apps, and Azure Data Factory.
• A successful history of manipulating, processing and extracting value from large disconnected datasets.
• Working knowledge of Azure Automation, Azure Data Factory, and highly scalable 'big data' data stores such as Azure Blob Storage, Azure Data Lake
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Excellent planning, prioritization, problem solving skills.
• Experience with any of the following scripting languages, R, JSON, Parquet, JavaScript, Regex, PowerShell, shell scripting.
• Experience with Data Bricks, HDInsight, Python.