Data Engineer

The Core Responsibilities For The Job Include The Following
Palantir Data Engineering and Analytics;

• Design, develop, and maintain scalable, modular data pipelines using Foundry Pipeline Builder (visual and code-based).
• Ingest, integrate, and process data from diverse sources (S3 RDBMS, REST APIs, flat files, etc. ).
• Implement advanced data processing techniques: incremental processing, anomaly detection, geospatial transformations, and time series analysis using PySpark, Python, and Foundry's no-code tools.
• Parse and process various data formats (CSV, JSON, XML, Parquet, Avro) within Foundry.
• Reuse and modularize functions and parameters for efficient, maintainable pipeline management.
• Leverage LLMs for translation, classification, and data enrichment in pipelines via Palantir AIP Logic.

Ontology And Schema Management

• Create and manage ontologies using Ontology Manager and Object Explorer to model business entities, relationships, and data lineage.
• Implement property-level and object-level access controls for secure data modeling and compliance.

Data Quality, Validation, And Monitoring

• Design and implement Master Data Management (MDM) and Reference Data Management solutions to ensure consistency and accuracy of key business entities across the organization.
• Lead efforts in entity resolution, de-duplication, and golden record creation within Palantir or integrated MDM platforms.
• Implement data validation, health checks, and monitoring for production-grade reliability.
• Ensure data integrity, quality, and consistency across all stages of the data lifecycle.
• Set up automated alerts and dashboards for pipeline health and anomaly detection.

Data Security And Governance

• Enforce data privacy, security, and compliance standards (RBAC, audit logs, data masking) within Palantir and cloud environments.
• Document data lineage, transformations, and access controls for auditability and governance.

Collaboration And Best Practices

• Work closely with business analysts, data scientists, and product owners to translate requirements into robust data solutions.
• Mentor junior engineers and analysts, contribute to code reviews, and champion best practices.
• Document technical designs, workflows, and user guides for maintainability and knowledge transfer.

Data Analysis And Visualization

• Perform data profiling, cleaning, joining, and enrichment to support business decision-making.
• Conduct statistical and exploratory analysis to uncover trends, patterns, and actionable insights.

Dashboarding And Reporting

• Develop and manage interactive dashboards and reports using Palantir Contour, Quiver, and other BI tools (Tableau, Power BI, Looker).
• Build pivot tables, advanced reporting, and custom visualizations tailored to business needs.
• Leverage Palantir's visualization modules for real-time and historical data analysis.

Cloud Platform Integration

• Integrate AWS, Azure, or GCP data engineering and analytics services (Glue, Data Factory, BigQuery, Redshift, Synapse, etc. ) with Palantir workflows.
• Design and implement end-to-end data pipelines that bridge Palantir and cloud-native ecosystems.

API And Microservices Integration

• Develop and consume RESTful APIs, GraphQL endpoints, and microservices for scalable, modular data architectures.

DevOps And Best Practices

• Implement CI/CD pipelines for data pipeline deployment and updates (Foundry, GitHub Actions, Jenkins, etc. ).
• Apply containerization (Docker) and orchestration (Kubernetes) for scalable data processing.

Agile Collaboration

• Work in Agile/Scrum teams, participate in sprint planning, and contribute to continuous improvement.

Requirements

• Experience: 7-12 years in data engineering, data analysis, or related roles; 1-2 years on Palantir Foundry, Pipeline Builder, Contour, Quiver, or strong experience with AWS/Azure/GCP data engineering and analytics services.
• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, Mathematics, or a related field.

Certifications (Preferred But Not Mandatory)

• Palantir Foundry Data Engineer/Analyst Certification.
• AWS/Azure/GCP Data Engineering or Analytics Certifications.
• Relevant BI/Visualization tool certifications.

Palantir Platform

• Data pipeline development, transformation, and cleaning (Pipeline Builder, Code Workspaces).
• Ontology creation, management, and data lineage (Ontology Manager, Object Explorer).
• Data validation, health checks, and monitoring in production pipelines.
• Data security, RBAC, audit logging, and compliance within Palantir.
• Dashboarding and visualization (Contour, Quiver).
• LLM integration for data enrichment (AIP Logic).

Data Engineering

• Proficiency in SQL and Python; experience with PySpark is highly desirable.
• Experience with data ingestion, integration, aggregation, and transformation from multiple sources.
• Geospatial data processing, time series analysis, and anomaly detection.
• Parsing and processing structured, semi-structured, and unstructured data.

Data Analysis

• Data profiling, cleaning, joining, and enrichment.
• Exploratory and statistical analysis.
• Dashboarding, reporting, and advanced visualization (Contour, Quiver, Tableau, Power BI).

Cloud Platforms

• Hands-on experience with AWS, Azure, or GCP data engineering and analytics services.
• Integration of cloud services with Palantir workflows.

General Skills

• Strong analytical, problem-solving, and communication skills.
• Experience working in Agile/Scrum environments.
• Ability to mentor and guide junior engineers and analysts.

Preferred Skills

• Experience with Palantir's Pipeline Builder, Code Workspaces, and advanced data transformation modules.
• Exposure to LLMs for data translation and enrichment.
• Familiarity with data governance, security, and compliance best practices.
• Prior experience in industries such as finance, healthcare, manufacturing, or government.
• Experience with open-source data engineering tools (dbt, Great Expectations, Delta Lake, Iceberg).
• Knowledge of CI/CD, DevOps, and automation tools.

This job was posted by Veronica A from Hashedin by Deloitte.