Senior Data Engineer Palantir

About the Role:

We are hiring a Senior Data Engineer with strong expertise in Palantir Foundry or Gotham to lead the design, development, and optimization of data engineering solutions across large-scale enterprise environments. The ideal candidate will have deep experience building secure, high-performance data pipelines using Python, ETL/ELT practices, distributed computing frameworks, and cloud platforms ( AWS ,Azure ,GCP)while working closely with DevOps, analytics, and business teams to enable data-driven decisions.

Key Responsibilities:

• Architect, build, and optimize large-scale ETL/ELT pipelines using Palantir Foundry/Gotham.
• Integrate diverse data sources including SQL, NoSQL, APIs, and real-time streaming platforms (e.g., Kafka) into Palantirs ontology-driven models.
• Develop and customize ontologies, transforms, and operational workflows within Palantir to support business intelligence and analytics.
• Implement scalable, distributed processing using frameworks like Apache Spark, Hadoop, and other big data technologies for handling petabyte-scale datasets.
• Write efficient, production-grade Python scripts for data transformation, automation, workflow orchestration, and custom business logic.
• Optimize data workflows and platform performance through query tuning, caching, partitioning, and incremental data updates.
• Ensure robust data governance, lineage tracking, and enterprise-grade security (e.g., RBAC, encryption).
• Collaborate with DevOps and platform engineering teams to implement and maintain CI/CD pipelines and automated, scalable deployment processes.
• Maintain and version-control data assets and transformations using Git and DevOps best practices.
• Partner with data scientists, analysts, and stakeholders to translate complex business requirements into scalable data engineering solutions.
• Create and maintain comprehensive technical documentation, including data models, architecture diagrams, deployment workflows, and operational guides.

Required Skills:

• Proven expertise in Palantir Foundry or Gotham, with deep knowledge of its ontology framework, code workbooks, and data pipeline architecture.
• Strong programming experience in Python and advanced SQL.
• Hands-on experience with ETL/ELT design, data transformation, and workflow automation.
• Proficiency with Apache Spark, Hadoop, or similar distributed data processing frameworks.
• Experience integrating data from relational and non-relational databases, streaming sources, and external APIs.
• Understanding of data modeling, ontology-driven design, and semantic layers.
• Solid grasp of CI/CD, DevOps practices, and deployment automation tools.
• Familiarity with cloud platforms (AWS, Azure, or GCP), containerization (Docker), and orchestration tools (Kubernetes, Airflow).

Nice to Have:

• Experience with Foundry Ontology SDK, Object Explorer, and Python transforms in Palantir.
• Exposure to data cataloging, metadata management, and monitoring frameworks.
• Understanding of ML/AI pipeline integration within data platforms.
• Certifications in Palantir, cloud platforms, or big data technologies.

Educational Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Minimum 7-10 years of experience in data engineering, ETL, and data integration.
• Strong programming experience in Python (including Pandas, PySpark, NumPy, etc.).
• Hands-on expertise with Palantir Foundry and its ecosystem.
• Familiarity with big data technologies such as Hadoop, Spark, Kafka, etc.
• Excellent problem-solving and analytical skills.
• Effective communication skills and a collaborative mindset.
• Proven leadership or mentorship experience is a strong plus.

Why Join Us?

• Be part of transformative data engineering projects using one of the most cutting-edge data platforms in the industry.
• Work in a fast-paced, collaborative environment with access to global teams and complex use cases.
• Hybrid work model that supports flexibility and productivity.
• Competitive salary and benefits with opportunities for growth and learning.