Cloud-Based Data Engineering Specialist

Boston Insights is transforming the pharmaceutical industry by unlocking the power of data-driven supply chain management.

This role requires a highly skilled Data Engineer to lead the design and implementation of cloud-based data integration solutions, leveraging expertise in Azure and AWS technologies. The ideal candidate will have a proven track record of integrating data from external sources, including AWS services and SAP systems.

The successful applicant will be responsible for designing, building, and maintaining scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake. They will also lead the integration strategy for ingesting and harmonizing data from external sources, automate ETL processes, and build real-time and batch data integration workflows between Azure, AWS, and on-premises sources.

Key Responsibilities:

• Data Integration & Pipeline Development
  • Design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake.
  • Lead the integration strategy for ingesting and harmonizing data from external sources, including AWS-based data lakes/warehouses (such as S3, Redshift) and SAP systems.
  • Automate ETL processes for data extraction, transformation, and loading across hybrid and multi-cloud environments.
  • Build and maintain real-time and batch data integration workflows between Azure, AWS, and on-premises sources.
• Data Architecture & Infrastructure
  • Design and implement data lake and data warehouse solutions on Azure platform.
  • Establish data governance frameworks and ensure data quality across all pipelines.
  • Implement security best practices for handling sensitive pharmaceutical data.
  • Create and maintain data documentation and lineage tracking.
• Data Governance & Quality
  • Define and enforce data governance frameworks: data cataloging, lineage, quality, privacy, and compliance.
  • Implement robust data validation, cleansing, and monitoring systems to ensure accuracy and reliability.
  • Support security standards through effective data management practices.
• Automation & Tooling
  • Develop data automation tools and reusable components using Python, PySpark (and other relevant frameworks/languages).
  • Enable end-to-end process automation for data ingestion, processing, and reporting.
  • Implement CI/CD processes for data solutions, including testing, monitoring, and alerting.
• Analytics & Reporting Support
  • Collaborate with data scientists and analysts to support advanced analytics.
  • Build data models that enable risk assessment and supply chain optimization.
  • Develop APIs and data services to support front-end applications.
  • Create monitoring and alerting systems for data pipeline health.
• Collaboration & Support
  • Partner with supply chain, analytics, and business stakeholders to understand business requirements and translate them into scalable technical solutions.
  • Collaborate with SAP functional and technical teams to optimize data extraction and synchronization.

Required Technical Qualifications:

• 5+ years of professional data engineering experience.
• Proven track record integrating data from AWS services (S3, Redshift, Glue, etc.) into Azure or other cloud environments.
• Expert-level knowledge of Azure Data Services (Azure Data Factory, Azure Synapse Analytics, Data Bricks, Azure Data Lake Storage, and Azure SQL Database, Apache Spark).
• Strong knowledge of both relational (SQL Server, PostgreSQL) and NoSQL (Cosmos DB) databases.
• Proficiency in Python, SQL, PySpark, and PowerShell for data automation and wrangling.
• Version Control: Proficiency with Git and Azure DevOps.
• Hands-on experience with SAP data models and integrating SAP data with Azure data lake.

Preferred Technical Skills:

• Experience with API-based data integration for cloud and enterprise applications.
• Experience with Infrastructure as Code (ARM templates, Terraform).
• Familiarity with data quality tools, metadata management, and automated data lineage tracking.
• Knowledge of containerization (Docker, Kubernetes) for data automation workflows.
• Knowledge of machine learning pipelines and MLOps practices.
• Experience with data visualization tool, Power BI.

Professional Skills:

• Strong problem-solving and analytical thinking abilities.
• Excellent communication skills with ability to explain technical concepts to non-technical stakeholders.
• Experience with Agile development methodologies.
• Attention to detail and commitment to data quality.

What We Offer:

• Competitive salary commensurate with experience.
• Professional development training and certifications.

Work Environment:

• Remote work arrangement with flexible hours.
• State-of-the-art technology and tools.
• Collaborative, innovation-driven culture.
• Access to cutting-edge pharmaceutical industry data and challenges.
• Opportunity to shape an innovative pharma analytics platform.

Application Instructions:

• Please submit your resume and a cover letter highlighting:
• Experience in Azure, AWS data integration, and SAP data extraction.
• Examples of data automation tools or frameworks you have developed.