Senior Data Engineer

JOB DESCRIPTION :Responsibilities : - Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights.- Optimize PySpark jobs and data pipelines for performance, scalability, and reliability.- Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows.- Ensure data quality and integrity throughout all stages of data processing.- Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation.- Troubleshoot and debug issues related to data pipelines and processing.- Ensure compliance with regulatory requirements and industry standards in all data processing activities.- Implement best practices for data security, compliance, and privacy within Azure environment.- Document technical specifications, data flows, and solution architecture- Design and build reusable components, frameworks and libraries at scale to support analytics products- Design and implement product features in collaboration with business and Technology stakeholders- Design and develop scalable data pipelines using Azure Databricks and PySpark. - Transform raw data into actionable insights through advanced data engineering techniques. - Build, deploy, and maintain machine learning models using MLlib, TensorFlow, and MLflow. - Optimize data integration workflows from Azure Blob Storage, Data Lake, and SQL/NoSQL sources. - Execute large-scale data processing using Spark Pools, fine-tuning configurations for performance and cost-efficiency. - Collaborate with data scientists, analysts, and business stakeholders to deliver robust data solutions.- Maintain and enhance Databricks notebooks and Delta Lake architectures.- Architect, build, and maintain scalable and reliable data pipelines from diverse data sources. - Design effective data storage, retrieval mechanisms, and data models to support analytics and business needs. - Implement data validation, transformation, and quality monitoring processes. - Collaborate with cross-functional teams to deliver impactful, data-driven solutions. - Proactively identify bottlenecks and optimize existing workflows and processes. - Provide guidance and mentorship to junior engineers in the team.- Anticipate, identify and solve issues concerning data management to improve data quality- Clean, prepare and optimize data at scale for ingestion and consumption- Drive the implementation of new data management projects and re-structure of the current data architecture- Implement complex automated workflows and routines using workflow scheduling tools- Build continuous integration, test-driven development and production deployment frameworks- Drive collaborative reviews of design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards- Analyze and profile data for the purpose of designing scalable solutions- Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues- Mentor and develop other data engineers in adopting best practicesQualifications : - 3+ years experiencing developing scalable Big Data applications or solutions on distributed platforms- Able to partner with others in solving complex problems by taking a broad perspective to identify innovative solutions- Strong skills building positive relationships across Product and Engineering.- Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders- Able to quickly pick up new programming languages, technologies, and frameworks- Experience working in Agile and Scrum development process- Experience working in a fast-paced, results-oriented environment- Experience in Amazon Web Services (AWS) or other cloud platform tools- Experience working with Data warehousing tools, including Dynamo DB, SQL, Amazon Redshift, and Snowflake- Experience architecting data product in Streaming, Serverless and Microservices Architecture and platform.- Experience working with Data platforms, including EMR, Data Bricks etc- Experience working with distributed technology tools, including Spark, Presto, Scala, Python, Databricks, Airflow- Working knowledge of Data warehousing, Data modelling, Governance and Data Architecture- Working knowledge of Reporting & Analytical tools such as Tableau, Quick site etc.- Demonstrated experience in learning new technologies and skills- Bachelor's degree in Computer Science, Information Systems, Business, or other relevant subject area (ref:hirist.tech)