Senior Data Engineer

Data Engineer RoleLocation: India RemoteAre you a passionate Data Engineer ready to design and build cutting-edge, scalable, and reliable ML/AI solutions? Join our dynamic cross-functional feature team in India and drive innovation from concept to industrialization. We're looking for an expert to integrate and optimize algorithms across the entire product lifecycle, ensuring robust and performant data pipelines.What You'll Do:● Architect and Implement: Collaborate with AI Product Managers and DevOps to co-realize data and tech due diligence, setting up pristine exploratory environments. You'll design and build efficient data pipelines on demand, meticulously measuring cost-effectiveness across transformation scenarios.● Innovate and Optimize: Embed critical monitoring metrics into products, design future-proof MVP target architectures, and meticulously package and document your code and infrastructure. Your expertise will optimize ML/AI algorithm performance in distributed systems, enhancing model response times and parallelism.● Drive Future-Forward Initiatives: Stay at the forefront of technology through continuous technological watch, test new innovations in real-world use cases (MVMs & MVPs), and champion market-shifting technologies that add tangible value. You'll even contribute to selecting new service providers.Your Technical Prowess Will Shine In:● Programming Mastery: Excellent command of Python (mandatory) and experience with Scala. A solid understanding of both Object-Oriented Programming (OOP) and functional programming is key. Java, C#, and R are a plus.● Big Data & Distributed Systems:○ Big Data: Spark is a must, coupled with knowledge of message-based systems like Kafka and Stream. Experience with Azure Databricks is highly valued.○ Containerization: Proficient in Docker and Kubernetes.○ Architecture & Cloud Patterns: Expertise in Microservices (Azure Functions), SOA architectures, and API design practices (HATEOAS, GraphQL).○ Databases: Strong with SQL (Azure SQL Server) and NoSQL (ElasticSearch,CosmosDB, MongoDB). Proven experience with Snowflake is a significant plus.○ Deep understanding of data structures and big data-oriented file formats (Parquet/Delta, ORC, AVRO).● Machine Learning and Analytics:○ Familiarity with main Machine Learning algorithms (Reinforcement, Supervised, Unsupervised).○ Hands-on coding experience with analytics and ML frameworks: NumPy, Scikit-learn, Pandas.○ Experience with experimental environments: Jupyter Notebook, Zeppelin, MLFlow.○ Proficiency in deep learning frameworks: TensorFlow, Keras, Caffe.○ Experience with dashboarding and visualization tools (PowerBI in particular) is a plus.○ Experience with Azure Machine Learning Services is a plus.● Cloud Expertise (Azure is Key):○ Architect-level knowledge of Azure Services, with a sharp focus on data services like Azure Data Factory, Data Lake, Databricks.○ Solid understanding of governance, networking, and security in the cloud (subscriptions, resource groups, SSOs, VPC, NSGs).○ Knowledge of other cloud offerings (AWS, GCP) is a plus, but Azure Cloud experience is mandatory.● Software Engineering & DevOps:○ Comfortable with Git for Source Code Management.○ Adept at creating build and release pipelines in CI/CD tools like Azure DevOps (preferred) or Gitlab.○ Experience with workflows and ETL/ELT processes.○ Familiarity with ADLS Gen2, Azure Functions, and Logic Apps.○ Understanding of Medallion architecture (Bronze, Silver, Gold).○ Experience with Jira for project management.○ Familiarity with Azure OpenAI and working with APIs/webapps and other relevant Azure services.Qualifications:● MSc in Computer Sciences● Typically 3-5 years of relevant experience (5-8 for Senior Data Engineer role).Your Core Competencies:● System Design: Expert in specification creation, user stories & standards writing, and comprehensive application & solution design.● Data Management & Analysis: Proficient in Data Modeling & Design, Meta Data Management, Data Warehouse, Business Intelligence & Data Visualization, and Master Data Management.● Application Development, Support & Maintenance: Strong in application performancemeasurement, DevOps methodology, coding & scripting, and thorough validation & testing.● Processes & Methodologies: A master of Agile methodology.Personal Traits:● Collaborates effectively and contributes independently.● Ensures accountability and drives tangible results.● Demonstrates nimble learning capabilities.If you're ready to make a significant impact on our AI/ML journey and thrive in a fast-paced, innovative environment, apply now