Data Engineer

About the Role :

We are seeking a highly skilled and experienced Data Engineer with deep expertise in Python, PySpark, and Palantir Foundry to join our growing engineering team. You will be instrumental in building scalable, secure, and high-performance data solutions that power mission-critical analytics and front-end applications across our organization.

This role will involve designing and developing robust data models, implementing efficient ETL pipelines, collaborating with subject matter experts, and delivering production-ready Palantir Foundry applications. The ideal candidate will have strong software engineering fundamentals, hands-on experience with DevOps tooling, and a deep understanding of data transformation in distributed environments.

Key Responsibilities :

Data Engineering & Pipeline Development :

- Design and implement scalable, fault-tolerant ETL pipelines using Python and PySpark within the Palantir Foundry platform.

- Transform raw data from multiple sources into curated, enriched, and ontology-compliant datasets for analytical and operational use cases.

- Develop and manage Foundry Ontology Objects, ensuring consistency and accuracy across interconnected datasets.

Application Development in Palantir Foundry :

- Build and maintain operational and analytical applications using Foundrys internal tools: Workshop, Map, and Slate.

- Work closely with SMEs to translate business logic into ontology models and intuitive user interfaces.

- Continuously improve application performance, data accuracy, and UX design as requirements evolve.

Platform Engineering & DevOps Integration :

- Use modern CI/CD pipelines including Git/Bitbucket, Jenkins/CodeBuild, and CodePipeline for automated deployment and testing.

- Set up and monitor platform alerts, health checks, and observability dashboards to ensure application reliability and uptime.

Collaboration & Agile Execution :

- Actively participate in all Agile ceremonies including daily stand-ups, sprint planning, backlog refinement, and retrospectives.

- Collaborate with cross-functional teams including data scientists, business analysts, platform engineers, and product owners.

- Contribute to post-implementation reviews and foster a culture of continuous improvement through actionable insights and & Governance :

- Develop and maintain comprehensive documentation for data models, transformation logic, application components, and user guides.

- Ensure all data engineering solutions are compliant with governance, security, and audit standards.

- Maintain a well-organized data catalog with lineage and metadata annotations.

Stakeholder Communication :

- Translate complex technical concepts into actionable insights for non-technical stakeholders.

- Work closely with business leaders to align data solutions with strategic goals.

Required Skills & Qualifications :

- 5+ years of hands-on experience in data engineering, software development, or data platform operations.

- Strong programming skills in Python and PySpark, with demonstrated ability to write modular, testable, and reusable code.

- Advanced knowledge of SQL for data manipulation, analytics, and performance tuning.

- Proven experience working with Palantir Foundry, including building ontologies, creating datasets, developing workflows, and deploying front-end applications using Workshop, Map, and Slate.

- Proficiency with version control and CI/CD systems: Git/Bitbucket, Jenkins/CodeBuild, and CodePipeline.

- Hands-on experience with data orchestration, transformation frameworks, and working with large-scale distributed data systems.

- Experience with platform observability tools for alerting, logging, and monitoring (e.g., CloudWatch, Grafana, or Datadog).

- Solid understanding of data modeling, metadata management, and data lineage.

- Strong debugging and performance profiling skills across the data stack.

- Excellent communication and collaboration skills; ability to convey complex data problems and solutions to business and technical audiences alike.