Data Engineer

Data Engineer Job Description About Sirius AI Sirius AI is a US headquartered AI Consulting services and products company with operations in India. Sirius AI focuses on Financial Services enterprises and solutions / services delivered across multiple geographies. We are an innovation-driven AI and data driven consulting services firm with a high focus on delivering business outcomes for our clients. The Sirius AI team is a mix of talent from a diverse background of Consulting, Industry and Product companies which bring together their Data, Platform, Ecosystem, Technology, Project management and AI skills to develop new solutions and deliver client engagements. Key Responsibilities: • Data Architecture & Management: Design and implement scalable, cloud agnostic Data Lake, Data LakeHouse, Data Mesh and Data Fabric architectures to efficiently store, process, and manage structured and unstructured data from various sources. • Data Pipeline Development: Design, develop, and maintain robust data pipelines to ingest, process, and transform data from multiple sources into usable formats for analytics and reporting using services like AWS Glue, Azure Data Factory, GCP Dataflow, Apache Spark, or Apache Airflow. • Data Integration and ETL: Develop and optimize Extract, Transform, Load (ETL) and ELT processes to integrate disparate data sources into the data lake, ensuring high data quality, consistency, and reliability across multiple cloud platforms. • Cloud-Agnostic Data Engineering: Develop data solutions that are cloud agnostic, leveraging open-source technologies like Apache Spark, Delta Lake, Presto, and Kubernetes, ensuring compatibility across AWS, Azure, and GCP. • Big Data Processing & Analytics: Utilize big data technologies such as Apache Spark, Hive, and Presto for distributed computing, enabling large-scale data transformations and analytics. • Data Governance and Security: Implement robust data governance policies, security frameworks, and compliance controls, including role-based access control (RBAC), encryption, and monitoring to meet industry standards (GDPR, HIPAA, PCI-DSS). • DevOps Integration for Data Platforms: Leverage cloud-agnostic DevOps tools and practices for source control, build automation, release management, and Infrastructure as Code (IaC) to streamline the development, deployment, and management of data lake and data architecture solutions across multiple cloud providers. Solutions should support CI/CD pipelines, automated testing, and scalable data workflows. • Continuous Integration and Deployment (CI/CD): Establish automated CI/CD pipelines to streamline deployment, testing, and monitoring of data infrastructure and workflows. • Performance Optimization: Optimize data workflows and query performance using indexing, caching, and partitioning strategies to improve efficiency and cost-effectiveness. • Monitoring and Troubleshooting: Implement observability solutions using tools like Prometheus, Grafana, or cloud-native monitoring services to proactively detect and resolve data pipeline issues. • Collaboration and Documentation: Work with cross-functional teams, including data scientists, analysts, and business stakeholders, to design and implement scalable data solutions. Maintain comprehensive documentation of data architectures, processes, and best practices. Job Qualifications: • Bachelor's degree in Computer Science, Engineering, or a related field. • 6+ years of experience as a Data Engineer, specializing in cloud-agnostic data solutions and data lake architectures. • Strong expertise in cloud data platforms such as AWS, Azure, and Google Cloud, with hands-on experience in services like AWS S3, Azure Data Lake, Google Cloud Storage, and related data processing tools. • Proficiency in big data technologies such as Apache Spark, Hadoop, Kafka, Delta Lake, or Presto. • Experience with SQL and NoSQL databases, including PostgreSQL, MySQL, and DynamoDB. • Expertise in containerization and orchestration platforms such as Docker and Kubernetes. • Experience implementing DevOps and CI/CD practices using Terraform, CloudFormation, or other Infrastructure as Code (IaC) tools. • Knowledge of data visualization tools such as Power BI, Tableau, or Looker for presenting insights and reports. • Strong problem-solving and troubleshooting skills with a proactive approach to identifying and resolving issues. • Experience leading teams of 5+ cloud engineers. • Preferred certifications in AWS, Azure, or Google Cloud data engineering.