DEV Community: Kavi Writes

Building External Applications on Palantir Foundry with the Ontology SDK and TypeScript

Kavi Writes — Thu, 29 Oct 2026 13:05:50 +0000

Introduction

Many enterprise applications run into the same problem: the data is somewhere, the business logic is somewhere else, and access rules are re-implemented in every new app. Building on a governed data platform moves that work to one place. The application asks for business objects, and the platform decides what the caller is allowed to see and change.

Palantir Foundry approaches this through its Ontology, a model of business entities (orders, customers, assets, shipments), the relationships between them, and the operations people are allowed to perform on them. The Ontology SDK (OSDK) exposes that model to code. With the TypeScript OSDK, an application that runs outside Foundry can query objects, follow relationships, and apply actions with typed code instead of hand-built HTTP calls.

This article covers:

What the Ontology is and which of its parts an external app works with.
When an external application makes sense, and when a Foundry-native app is the better choice.
How the TypeScript OSDK packages fit together and how authentication works.
A small server-side example that reads Ontology objects with filtering and pagination.
The security, governance, and production concerns that matter once the demo works.

All code follows the patterns in Palantir's public documentation for TypeScript OSDK 2.x. Object type names, property names, and package names are placeholders. Your Foundry environment will generate its own.

Scope note: Foundry enrollments differ in enabled features, network restrictions, and administrator settings. Treat this article as a guide to the documented model, and confirm details in your stack's Developer Console documentation.

1. Understanding Foundry and the Ontology

Foundry is Palantir's data platform. It integrates datasets, virtual tables, and models from source systems. Palantir's documentation describes the Ontology as an operational layer on top of those assets: it maps raw data to real-world things such as plants, equipment, customer orders, or financial transactions.

The Ontology has two kinds of building blocks. The semantic elements describe what exists. The kinetic elements describe what can change and who can change it.

Concept	What it is	Supply-chain example
Object type	A schema for a real-world entity, backed by one or more data sources	`Order`, `Customer`, `Shipment`
Property	A typed field on an object type	`order.status`, `order.promisedDate`
Link type	A relationship between object types	An `Order` belongs to one `Customer`; it has many `Shipment` objects
Action type	A governed operation that edits objects, with parameters, rules, and submission criteria	`Reschedule Shipment`, `Flag Order At Risk`
Function	Business logic authored in code and published to the Ontology	A function that scores late-delivery risk
Interface	A shared shape that several object types can implement	A `Trackable` interface implemented by `Shipment` and `Container`

For an application developer, the important point is this: you do not query the warehouse tables behind Order. You query the Order object type. Its properties, links, and permissions are defined once in Foundry and apply to every consumer.

2. Why Build an External Application?

Build an external application when the users, the runtime, or the integration needs sit outside what Foundry's own app builders cover. Typical cases:

Internal business portals that combine Ontology data with systems Foundry does not own, such as an ERP screen or a ticketing tool.
Operational dashboards with a custom UX or design system that a low-code builder cannot reproduce.
Workflow interfaces where planners or field staff apply governed actions ("reschedule this shipment") from a tool they already use.
Backend services such as schedulers or integration jobs that read Ontology objects and push results to other systems.

It is not always the right choice. Foundry's own tools, such as Workshop, already sit on the Ontology and need no separate auth, hosting, or release pipeline. If the requirement is a standard operational screen for Foundry users, start there.

Option	Where code runs	Server-side logic	Good fit
Foundry-native app (for example, Workshop)	Inside Foundry	Uses Foundry functions and actions	Standard operational UIs for Foundry users
OSDK app hosted on Foundry	Static assets served by Foundry; code runs in the browser	Not supported by the hosting feature	Custom single-page apps for Foundry users, no extra infrastructure
OSDK app on your own infrastructure	Your servers, cloud, or containers	Yes, under your control	Integration with other systems, backend jobs, custom runtimes

Palantir's hosting documentation is explicit that hosted sites serve static files only (HTML, CSS, JavaScript, and assets). Node.js backends, Python servers, and server-side rendering are not supported there. If you need any of those, you are building an external application in the full sense of this article.

3. Understanding the Ontology SDK and TypeScript

The Ontology SDK is generated per application, not downloaded as one universal library. In Foundry's Developer Console, you create an application, select the object types, action types, and functions it may use, and generate an SDK version. Palantir distributes the TypeScript output through npm, Python through pip or Conda, Java through Maven, and other languages through an OpenAPI spec.

A TypeScript OSDK project normally combines these packages:

Package	Source	Role
Generated package (for example, `@your-app/sdk`)	Your Foundry stack's npm registry	Typed definitions for the object types, actions, and functions you selected
`@osdk/client`	Public npm	Runtime client: `createClient`, queries, actions, the `Osdk` types
`@osdk/oauth`	Public npm	Token providers: `createPublicOauthClient` and `createConfidentialOauthClient`
`@osdk/foundry.*` (optional)	Public npm	Platform SDK bindings for non-Ontology Foundry APIs

Since TypeScript OSDK 2.0, the runtime client is separate from the generated code. Palantir's migration guide says this lets you update the client library without regenerating the SDK, and that newer features such as interfaces and media sets require 2.0.

TypeScript adds real value here. Because types come from your Ontology's metadata, a misspelled property name or an unsupported aggregation metric fails at compile time, not in production. The TypeScript OSDK reference notes limits to this: some checks use a property's base type, so a few invalid queries on array properties still compile and fail at runtime. Types reduce errors; they do not replace tests.

Two rules keep you out of trouble:

Use the docs for your SDK version. Developer Console generates documentation for your specific Ontology and has a version picker. Public examples target 2.x.
Treat each SDK version as an API contract. Regenerating after an Ontology change can rename or remove types your code uses.

4. Architecture and Integration Flow

For enterprise integrations, a server-mediated design is a sound default. That is a recommendation, not a platform requirement. The diagram shows that pattern, with the browser-only alternative as a dotted line.

flowchart LR
  U["End user"] --> UI["Web or mobile UI"]
  UI -->|"app session"| BE["Your backend service"]
  BE -->|"validated, authorized request"| SDK["OSDK client<br/>@osdk/client + generated SDK"]
  SDK -->|"token request"| AUTH["Foundry OAuth"]
  SDK -->|"typed queries and actions"| API["Foundry Ontology APIs"]
  API --> ONT["Object types, links,<br/>action types, functions"]
  ONT --> DATA["Backing datasets"]
  UI -. "browser-only option:<br/>public client, user permissions" .-> API

Each component has one job:

Component	Responsibility
External TypeScript UI	Presentation and user interaction. Holds no secrets.
Backend service	Authenticates your users, validates input, applies app-level authorization, holds the OAuth client secret, shapes responses.
OAuth (Foundry)	Issues tokens: client credentials for a service user, or authorization code for a signed-in Foundry user.
OSDK client	Builds typed requests, handles paging, surfaces structured errors.
Ontology	Enforces resource restrictions, object permissions, and action rules, then reads or writes the backing data.

A typical read follows six steps:

The user opens your app and authenticates with your identity provider.
The UI calls your backend with an app session.
The backend validates the input and checks that this user may request this data.
The OSDK client obtains or reuses a Foundry token.
Foundry checks the token's resource scope and the caller's permissions, then runs the query.
The backend maps the result to a response shape and returns it.

The alternatives are legitimate. A browser-only app using a public OAuth client calls Foundry directly under the user's own permissions, with no backend to run. An application can also combine a client-facing and a backend type. Palantir also documents running OSDK code inside Foundry through compute modules. Choose based on where secrets can live and whose permissions should apply.

5. Setting Up the Development Environment

The example in this article is a server-side Node.js service that uses application permissions. The setup follows Palantir's guide to bootstrapping a back-end TypeScript application.

Prerequisites

Access to a Foundry enrollment with Developer Console, and permission to create applications. Palantir's permissions page lists the roles involved, including the third-party application administrator workflow your organization may control.
An object type to read. This article assumes an Order object type with region, status, and promisedDate properties. Substitute your own.
Node.js. Palantir's guides state Node 18 or higher. Node 18 and 20 have reached end of life, so use a currently supported LTS release.

Create the Developer Console application

Create a new application. Choose Backend service as the application type and Application's permissions as the permission type.
On the resources page, choose to generate an Ontology SDK, select your Ontology, and add only the object types and action types this service needs. The Ontology cannot be changed after generation.
Copy the client secret when it is shown. Developer Console displays it once; losing it means rotating it, which breaks running deployments.
Grant the generated service user read access to the selected object types. Without this, queries fail or return nothing.
Generate the first SDK version and note the package name and registry URL.

Install packages

Configure npm to resolve your generated package from Foundry's registry. Keep the token in an environment variable, never in the file:

# .npmrc
//<REGISTRY-URL-FROM-OVERVIEW-PAGE>:_authToken=${FOUNDRY_TOKEN}
<PACKAGE-NAME>:registry=https://<REGISTRY-URL-FROM-OVERVIEW-PAGE>

export FOUNDRY_TOKEN=<token-from-developer-console>   # not committed
npm install <PACKAGE-NAME> @osdk/client @osdk/oauth
npm install --save-dev typescript @types/node

Palantir's guides use @latest tags. For anything beyond a prototype, commit package-lock.json and upgrade deliberately.

Browser apps differ

A browser application cannot keep a secret. Palantir's docs require client-facing applications to use user permissions with the authorization code grant, through createPublicOauthClient and a registered redirect URL. For a browser project, start with npm create @osdk/app@latest as shown in the TypeScript bootstrap guide.

6. Building a Practical TypeScript Example

Use case: a logistics team's integration service needs the list of at-risk orders for one region, oldest promise date first, so it can push them into an external ticketing tool. The service reads through a service user, pages through results, and never exposes raw Ontology objects to callers.

The API calls below (createConfidentialOauthClient, createClient, where, fetchPage, PalantirApiError) come from Palantir's documentation. Order, its properties, the "AT_RISK" value, and @your-app/sdk are placeholders for your generated SDK.

Project structure

order-risk-service/
├── .env            # secrets, git-ignored
├── .npmrc          # registry config, token via env var
├── package.json    # "type": "module"
├── tsconfig.json
└── src/
    ├── config.ts   # validated environment
    ├── foundry.ts  # auth + OSDK client
    ├── orders.ts   # query logic
    └── index.ts    # entry point

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noUncheckedIndexedAccess": true,
    "outDir": "dist",
    "rootDir": "src"
  },
  "include": ["src"]
}

Configuration: fail fast, never hard-code

// src/config.ts
function required(name: string): string {
  const value = process.env[name]?.trim();
  if (!value) {
    throw new Error(`Missing required environment variable: ${name}`);
  }
  return value;
}

const foundryUrl = new URL(required("FOUNDRY_URL"));
if (foundryUrl.protocol !== "https:") {
  throw new Error("FOUNDRY_URL must use https");
}

export const config = {
  foundryUrl: foundryUrl.origin,
  ontologyRid: required("ONTOLOGY_RID"),
  clientId: required("FOUNDRY_CLIENT_ID"),
  clientSecret: required("FOUNDRY_CLIENT_SECRET"),
} as const;

Auth and client

// src/foundry.ts
import { createClient } from "@osdk/client";
import { createConfidentialOauthClient } from "@osdk/oauth";
import { config } from "./config.js";

export const auth = createConfidentialOauthClient(
  config.clientId,
  config.clientSecret,
  config.foundryUrl,
);

export const client = createClient(config.foundryUrl, config.ontologyRid, auth);

The client takes three things: the stack URL, the Ontology RID (a resource identifier, shown in Developer Console), and a token provider. Create it once per process and reuse it.

Query with filtering and pagination

// src/orders.ts
import { Order } from "@your-app/sdk"; // replace with your generated package
import { client } from "./foundry.js";

const REGION_PATTERN = /^[A-Z]{2,8}$/;
const MAX_PAGE_SIZE = 100;

export interface AtRiskOrder {
  orderId: string;
  status: string | undefined;
  promisedDate: string | undefined;
}

export interface AtRiskOrderPage {
  orders: AtRiskOrder[];
  nextPageToken: string | undefined;
}

export async function getAtRiskOrders(
  region: string,
  pageSize: number,
  pageToken?: string,
): Promise<AtRiskOrderPage> {
  if (!REGION_PATTERN.test(region)) {
    throw new RangeError("Invalid region code");
  }
  const size = Math.min(Math.max(1, Math.trunc(pageSize)), MAX_PAGE_SIZE);

  const page = await client(Order)
    .where({
      $and: [
        { region: { $eq: region } },
        { status: { $eq: "AT_RISK" } },
      ],
    })
    .fetchPage({
      $select: ["status", "promisedDate"],
      $orderBy: { promisedDate: "asc" },
      $pageSize: size,
      $nextPageToken: pageToken,
    });

  return {
    orders: page.data.map((order) => ({
      orderId: String(order.$primaryKey),
      status: order.status,
      promisedDate: order.promisedDate,
    })),
    nextPageToken: page.nextPageToken,
  };
}

Entry point and error handling

// src/index.ts
import { PalantirApiError } from "@osdk/client";
import { auth } from "./foundry.js";
import { getAtRiskOrders } from "./orders.js";

const MAX_PAGES = 5; // demo guard against unbounded loops

async function main(): Promise<void> {
  const region = process.argv[2] ?? "EMEA";
  await auth.signIn(); // as in Palantir's back-end guide

  let pageToken: string | undefined;
  let pages = 0;
  do {
    const page = await getAtRiskOrders(region, 50, pageToken);
    for (const order of page.orders) {
      console.log(`${order.orderId}\t${order.status ?? "-"}\t${order.promisedDate ?? "-"}`);
    }
    pageToken = page.nextPageToken;
    pages += 1;
  } while (pageToken !== undefined && pages < MAX_PAGES);
}

main().catch((error: unknown) => {
  if (error instanceof PalantirApiError) {
    // Log identifiers, not parameters: parameters can contain business data.
    console.error(
      `Foundry API error ${error.errorName ?? "unknown"} ` +
        `(status ${error.statusCode ?? "?"}, instance ${error.errorInstanceId ?? "n/a"})`,
    );
  } else if (error instanceof Error) {
    console.error(`Error: ${error.message}`);
  } else {
    console.error("Unknown error");
  }
  process.exitCode = 1;
});

What the important parts do

where + $and compiles to a server-side filter. Only matching objects cross the network.
$select limits returned properties. The primary key and object metadata still come back.
$pageSize and $nextPageToken implement cursor pagination. nextPageToken is undefined on the last page.
The DTO mapping decouples callers from generated types and avoids passing through properties nobody asked for.
PalantirApiError carries errorName, statusCode, and errorInstanceId, per the osdk-ts docs. The instance ID is what Palantir support asks for.

For long-running services, follow the token-refresh guidance in your Developer Console documentation rather than relying on a single signIn() at startup.

Run and test

# package.json scripts: "build": "tsc", "start": "node --env-file=.env dist/index.js"
npm run build
npm start -- EMEA

Expected behavior: one tab-separated line per at-risk order, oldest promise date first, at most 250 rows. Then test the failure paths on purpose: a wrong client secret should produce an auth error, a lowercase region a RangeError, and an object type the service user cannot read should produce an error or an empty result. Do not treat "no rows" as proof that no data exists.

Extending to actions

Writes go through action types, not direct edits. If you add an action type (here, a placeholder flagOrderAtRisk) to the SDK, the documented pattern is:

const result = await client(flagOrderAtRisk).applyAction(
  { orderId: "ORD-1001", reason: "Carrier delay" },
  { $returnEdits: true },
);
if (result.type !== "edits") {
  // handle a rejected submission
}

The action's submission criteria still run on the server, for the service user.

7. Authentication, Security, and Governance

Authentication proves who is calling. Authorization decides what that caller may read or change. A valid OAuth token settles the first question only.

Pick the permission model first

Developer Console offers two permission types, and the choice shapes your whole design:

	User permissions	Application permissions
OAuth flow	Authorization code grant	Client credentials grant
Whose access applies	The signed-in Foundry user	A generated service user
OSDK helper	`createPublicOauthClient`	`createConfidentialOauthClient`
Allowed in browser apps	Yes (required for client-facing apps)	No: needs a secret, so server-side only

An application can use both, for example user permissions for most data plus a service user for a reference data set every user may see.

What Foundry enforces, and what it does not

Foundry applies two layers on every OSDK request. First, the token is limited to the resources selected for the application, called resource access restrictions. Second, the caller's own permissions apply, including object security policies and restricted views.

Palantir's OSDK overview states the limit plainly: these controls filter what can be read. They do not follow the response payload or anything your application does with it. This matters most with application permissions. If a service user can read every Order and your backend serves those orders to any logged-in portal user, Foundry is no longer deciding who sees what. Your backend is.

Practical rules:

Prefer user permissions when end users see the data, so Foundry's policies apply per person.
If you use a service user, implement your own authorization layer, and do not share caches across users with different entitlements.
Grant least privilege twice: select only the object types, actions, and functions the app needs in its SDK, and give the service user only the read and action rights it needs.
Keep secrets server-side in a secret manager or a git-ignored .env for local work. Rotating a secret in Developer Console breaks running deployments, so treat rotation as a planned release.
Validate input before it reaches a query or action. The OSDK builds structured filters rather than query strings, but an unvalidated parameter can still widen a filter or submit a bad action.
Mind downstream copies. Data exported to logs, caches, or other systems leaves Foundry's controls. Palantir points to markings and classification-based access controls for data that must stay protected as it flows on.

8. Common Challenges and Best Practices

Each row below is either documented by Palantir or follows directly from documented behavior. Check configuration and permissions before debugging code.

Symptom	Likely cause	What to do
`npm install` returns 404 for the generated package	npm resolves from another registry	Run `npm config delete registry`, check every `.npmrc`, confirm the registry URL and `FOUNDRY_TOKEN` (guide)
Generated code imports `FoundryClient` or `PublicClientAuth`	SDK was generated with the 1.x generator	Generate a new SDK version with the TypeScript OSDK 2.x generator
Auth works but queries fail or return nothing	Service user or end user lacks read access, or a resource is missing from the app's access restrictions	Check grants; act on Developer Console's "outdated dependencies" warning
Old clients break after a new SDK version	A removed resource was also dropped from access restrictions	Keep removed resources in restrictions until every client has upgraded
`ObjectsExceededLimit` error	Object type backed by Object Storage v1 (10,000-result cap), or an aggregation over 10,000 results	Filter on the server, aggregate instead of listing, review the object type's backing store
Duplicate or missing rows across pages	Data changed between page requests	Use the documented `$snapshot` fetch option when consistency matters
TLS errors behind a corporate proxy	Node does not trust the proxy's CA	Set `NODE_EXTRA_CA_CERTS`; never disable certificate verification

Beyond the table:

Avoid N+1 link traversal. Calling $link per object in a loop multiplies requests. pivotTo() traverses links on a whole object set, and aggregate() computes counts and averages on the server.
Retry reads, not writes. Retry idempotent queries on transient errors with backoff and a cap. Do not blindly retry applyAction; a retry can apply an edit twice unless the action is designed to be idempotent.
Log identifiers, not data. errorInstanceId is enough for support. Never log tokens, secrets, or object payloads.
Respect the type system's edges. The TypeScript OSDK reference notes that aggregation results such as avg, min, and max are undefined when nothing matches. Handle that branch.
Test at two levels. Unit-test your mapping and validation with stubbed data, and keep a small integration suite against a non-production Ontology. Palantir publishes guides for unit testing and integration testing TypeScript OSDK code.

9. How This Fits into Enterprise Data and AI Transformation

An Ontology-backed external application connects three things that usually live apart: business context (what an Order is and how it relates to a Shipment), operational workflow (who may reschedule it and under what rules), and the tools people already use. The OSDK is the contract between them. When the Ontology changes, you regenerate the SDK and the compiler shows where your application is affected.

The same governed layer serves AI-enabled processes where your enrollment supports them. The TypeScript OSDK documents vector similarity search through nearestNeighbors() for object types with embedding properties. Palantir's Ontology MCP exposes an application's object types, actions, and query functions as Model Context Protocol tools for external AI agents, with application restrictions limiting what an agent can do. Palantir's documentation also notes that sending Ontology data to LLMs hosted outside its AI Platform must comply with your organization's policies. That decision belongs to governance, not to the developer alone.

None of this makes transformation automatic. An SDK cannot fix an Ontology that models the business poorly, an unclear ownership model for action types, or a permission design nobody reviewed. Those decisions need as much attention as the code.

This is also where Palantir consulting work typically contributes, whether delivered by Palantir's own teams or by independent implementation partners. Useful contributions are concrete: Ontology design reviews before SDK generation, choosing between user and application permissions, integration architecture with existing systems, permission and marking design, and production-readiness reviews covering secrets, observability, and SDK upgrade policy. Evaluate any provider on those specifics and on evidence from comparable work.

10. Conclusion

An external TypeScript application talks to Foundry through a generated, typed SDK scoped to the Ontology resources you select. The code is the small part: createClient, a token provider, and object set queries such as where and fetchPage. The decisions that matter come earlier. Choose user or application permissions, select the minimum resources, decide who is responsible for authorization once data leaves Foundry, and decide whether a Foundry-native tool would serve the need with less to operate.

A realistic next step: in a non-production Foundry environment, create a Backend service application with one read-only object type, run the example above against it, and then test the failure paths on purpose. Once that works, decide between user and application permissions for your real use case before writing more code.

This article was drafted with AI assistance. The author reviewed the technical content against the linked documentation before publishing.

References

Official Palantir documentation consulted for this article:

Grounding Palantir AIP Workflows in the Ontology: Tool Design, Guardrails, and Evaluation Patterns

Kavi Writes — Wed, 14 Oct 2026 10:24:46 +0000

Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation

Replacing a legacy ETL platform is rarely a simple exercise in rewriting SQL or moving transformation code.

The difficult part is preserving data correctness while changing how data is ingested, transformed, validated, and consumed. Existing pipelines may support dashboards, operational reports, financial processes, APIs, machine learning workloads, and applications that nobody wants to interrupt.

For a Palantir Foundry migration, three concerns deserve particular attention:

Incremental data processing
Schema evolution
Data reconciliation

A fourth concern connects all three: how to operate the legacy and Foundry pipelines safely during the transition.

The objective should not be to move every pipeline as quickly as possible. A better objective is to create a controlled migration in which differences can be detected before they affect consumers.

This article describes one practical migration pattern. It is an architectural approach, not a mandatory Palantir methodology.

Why Legacy ETL Migrations Are Difficult

Older ETL estates often contain much more business logic than their diagrams suggest.

A nightly job might appear to perform four steps:

Extract -> Transform -> Load -> Report

In practice, the process may depend on:

Stored procedures maintained by another team
File drops from external systems
Hard-coded business mappings
Historical correction rules
Scheduled dependencies
Manual restart procedures
Temporary tables
Reporting-specific transformations
Downstream jobs that assume a particular schema

Some dependencies are documented. Others exist only because one job happens to finish before another begins.

This is why a direct "lift and shift" can be dangerous.

The technical transformation may be reproduced correctly while an operational dependency is missed.

A big-bang migration increases that risk. If extraction, transformation, storage, schedules, downstream interfaces, and reporting consumers all move together, diagnosing a discrepancy becomes harder.

A safer approach reduces the migration boundary.

A Safer Migration Pattern

Instead of migrating an entire ETL estate at once, divide it into data products or logical pipeline groups.

A possible migration lifecycle is:

Inventory existing pipelines.
Identify source and downstream dependencies.
Select a manageable migration boundary.
Rebuild the required pipeline in Foundry.
Keep the existing ETL process operating.
Execute the new pipeline in parallel.
Reconcile legacy and Foundry outputs.
Investigate unexplained differences.
Move consumers gradually.
Retire the corresponding legacy jobs.

For example, suppose a reporting warehouse receives customer, product, order, invoice, and payment data.

Migrating all five domains simultaneously makes reconciliation difficult. Instead, the team might begin with the order domain while maintaining its existing interfaces to customer and product data.

The boundary should be large enough to deliver something useful but small enough to validate thoroughly.

This staged approach is an implementation strategy. Foundry provides capabilities that can support pipeline construction, data validation, and incremental processing, but organizations still need to design the migration sequence around their own dependencies.

Incremental Transforms in Foundry

Full-refresh pipelines are common in legacy environments.

Consider a source table containing 500 million transaction records. A nightly pipeline might reread and transform the entire table even though only a fraction changed during the day.

A simplified process could look like this:

Read all transactions
        |
Apply transformation rules
        |
Recalculate entire output
        |
Replace reporting table

That design may be acceptable for smaller datasets, but it becomes increasingly expensive and operationally awkward as data grows.

Foundry supports incremental transforms that use build history to avoid recomputing the entire output when appropriate. Palantir documents its incremental transform model as using previous build information to determine the state of inputs and process eligible changes incrementally.

Conceptually, the migration could instead become:

Identify new or changed records
        |
Process affected data
        |
Merge or append results
        |
Publish updated dataset

Change detection

A migration team first needs to understand how the source exposes changes.

Possible mechanisms include:

Append-only transactions
Modification timestamps
Change-data-capture feeds
Source sequence numbers
Partition dates
Snapshot comparisons

The correct choice depends on the source architecture.

A timestamp such as last_modified_at can sometimes act as a watermark:

previous_watermark = last_successful_timestamp

read records where:
last_modified_at > previous_watermark

This is an architectural concept rather than a universal Foundry requirement.

Partitioning

Partitioning can help narrow processing boundaries.

A transaction pipeline might partition data by:

transaction_date
region
source_system

If the source changes only recent partitions, the pipeline may avoid unnecessary work on older data.

Partition design should follow access and update patterns rather than being added only for migration purposes.

Late-arriving data

Watermarks introduce another challenge.

Suppose the pipeline processed data through 02:00, but an upstream application later sends a transaction created at 01:55.

A strict timestamp boundary could miss it.

Teams may address late arrivals with strategies such as:

Reprocessing a recent time window
Tracking source sequence identifiers
Maintaining correction feeds
Comparing recent partitions
Designing source ingestion around append semantics where possible

The correct approach depends on the source system.

Idempotency matters

Incremental processing should also consider retries.

If yesterday's batch runs twice, the output should not unintentionally duplicate transactions.

A useful design question is:

Can this transform safely process the same source change more than once?

Keys, merge rules, deduplication logic, and transaction semantics should be considered before an incremental design is promoted.

Palantir's documentation also notes that certain input changes can cause an incremental transform to fall back to non-incremental execution, so teams should understand how their selected transform implementation behaves rather than assuming every run will always remain incremental.

Managing Schema Evolution

Schema changes are one of the easiest ways to break a migration unexpectedly.

Typical changes include:

Adding a column
Renaming a field
Removing a field
Changing a data type
Modifying nullability
Changing nested structures
Introducing a new source-system version

Foundry datasets can have schemas describing fields and their types, and schemas can change as the structure of a dataset changes over time.

That does not mean every schema change should automatically propagate through a production pipeline.

Use explicit schema contracts

Assume a customer source currently provides:

customer_id: string
customer_name: string
credit_limit: decimal

A source upgrade changes it to:

customer_id: string
customer_name: string
credit_limit: string
customer_segment: string

An automatic mapping could hide a significant problem.

credit_limit may now contain values such as:

"50000"
"25K"
"NOT_AVAILABLE"

The field name stayed the same, but its meaning and validation requirements changed.

A safer transformation explicitly handles the new schema:

validate source type
        |
normalize credit_limit
        |
reject or quarantine invalid values
        |
map customer_segment
        |
publish stable consumer schema

Separate source schemas from consumer contracts

A useful pattern is to avoid forcing every upstream schema change directly onto downstream consumers.

For example:

Raw source schema
      |
Normalization layer
      |
Canonical customer schema
      |
Reporting / analytics / applications

This creates a controlled boundary.

Source systems can evolve while the canonical output remains stable until consumers are ready for a new version.

Treat incremental schema changes carefully

Incremental pipelines deserve additional attention because historical and newly written records may have been produced under different assumptions.

Palantir's incremental transform documentation describes schema compatibility considerations when reading previous output. It also documents patterns such as changing a transform's semantic version when a logic or schema change requires existing data to be recomputed.

The implementation decision should therefore answer:

Can old and new records coexist safely?
Does historical data require recomputation?
Will downstream consumers understand the new field?
Does a type change require migration logic?
Should the new schema be introduced through a versioned dataset?

Schema evolution should be treated as an interface change, not merely a column change.

Reconciliation Between Legacy and Foundry Pipelines

Building the new pipeline is only half the migration.

The next question is more important:

Does it produce the correct result?

A common mistake is checking only row counts.

Suppose both systems produce exactly:

10,000,000 rows

That does not prove equivalence.

The pipelines could contain different records, incorrect values, duplicate keys, or different revenue totals while still having identical row counts.

Reconciliation should happen at several levels.

1. Row counts

Compare overall counts and meaningful subsets.

legacy_count = 10,000,000
foundry_count = 10,000,000

Then compare by business dimensions:

count by date
count by region
count by product
count by status

2. Key coverage

Compare business keys.

legacy_only_keys
foundry_only_keys
matching_keys
duplicate_keys

Unexpected missing keys usually deserve investigation.

3. Record-level comparison

For shared keys, compare significant fields.

Conceptually:

SELECT
    customer_id,
    legacy.amount,
    foundry.amount
FROM legacy
JOIN foundry
  ON legacy.customer_id = foundry.customer_id
WHERE legacy.amount <> foundry.amount;

Real implementations may require null handling, rounding rules, timestamps, or tolerance thresholds.

4. Aggregate reconciliation

Business aggregates often expose problems faster than row-level inspection.

Compare:

SUM(invoice_amount)
COUNT(DISTINCT customer_id)
AVG(order_value)
SUM(quantity)

Then calculate them across meaningful dimensions.

For example:

Metric	Legacy	Foundry
Order count	1,004,218	1,004,218
Revenue	87,412,552.10	87,412,552.10
Unique customers	124,911	124,911

The numbers above are purely illustrative.

5. Data quality indicators

Compare:

Null percentages
Duplicate rates
Invalid codes
Referential integrity
Date ranges
Unexpected values
Key uniqueness

Foundry provides mechanisms for defining data expectations and health checks. Depending on the implementation, these can validate requirements such as row counts, non-null values, permitted values, primary-key conditions, and custom row-level conditions.

These platform capabilities can support validation, but migration-specific reconciliation logic still has to reflect the organization's business rules.

Running Both Pipelines Without Downtime

Parallel execution gives teams time to establish confidence before changing consumers.

The pattern may look like this:

                 +-> Legacy ETL -> Legacy output -> Current consumers
Source systems --|
                 +-> Foundry pipeline -> New output -> Validation

Users continue reading from the existing production path while the Foundry output is evaluated.

This is often called shadow processing or parallel validation.

Move consumers gradually

Once reconciliation is stable, consumers can be migrated separately.

For example:

Week 1:
Finance dashboard -> Legacy
Operations dashboard -> Legacy
Risk model -> Legacy

Week 2:
Finance dashboard -> Foundry
Operations dashboard -> Legacy
Risk model -> Legacy

Week 3:
Finance dashboard -> Foundry
Operations dashboard -> Foundry
Risk model -> Legacy

This creates smaller cutover events.

If a problem appears in one consumer, the entire migration does not necessarily need to be reversed.

Define rollback before cutover

A rollback plan should answer:

What conditions trigger rollback?
Can the consumer reconnect to the legacy output?
How long will the legacy pipeline remain operational?
Which team makes the rollback decision?
How will data generated during the transition be handled?

Do not define these rules during an incident.

The goal is not to promise literal zero downtime. The goal is to minimize disruption and preserve a controlled path back to a known state.

A Practical Migration Example

Consider a hypothetical distributor with a nightly ETL pipeline.

Three operational systems provide:

Customer records
Orders
Payment transactions

Every night, the legacy platform reloads the data warehouse. Finance and sales dashboards depend on the output by 7:00 AM.

The migration could proceed in six stages.

Stage 1: Understand the legacy pipeline

The team documents:

CRM -----------+
               |
ERP -> Orders -+-> Legacy ETL -> Warehouse -> BI
               |
Payments ------+

Transformation rules, schedules, keys, error handling, and downstream dependencies are identified.

Stage 2: Build the Foundry version

The equivalent processing path is implemented in Foundry.

The objective is not yet optimization. The first priority is producing a result that can be compared with the existing pipeline.

Stage 3: Introduce incremental processing

Instead of refreshing ten years of orders nightly, the new design processes changed data where the source and pipeline architecture allow it.

Historical data is loaded during initial construction.

Subsequent processing focuses on eligible new or changed records.

Stage 4: Run both pipelines

For several production cycles:

Legacy output  ----+
                   +-> Reconciliation process
Foundry output ----+

The migration team compares counts, keys, aggregates, duplicates, nulls, and important business rules.

Stage 5: Explain every material difference

Imagine the Foundry result reports 214 more orders.

The difference should not immediately be classified as an error.

Investigation might find that the legacy job silently excluded late-arriving orders after its extraction window.

Now the team must decide what the correct business behavior should be.

Reconciliation is therefore not simply about making the new system match old defects. It is about identifying and explaining differences.

Stage 6: Move consumers and retire legacy jobs

After agreed validation thresholds are met, consumers are migrated individually.

The legacy process remains available during the defined rollback period.

Only after downstream dependencies, schedules, reconciliations, and support responsibilities have been confirmed should the corresponding legacy jobs be retired.

Common Migration Mistakes

Migrating everything at once

Large migration boundaries increase the number of possible causes when something fails.

Start with well-understood domains.

Ignoring downstream dependencies

A dataset is not isolated simply because no other ETL job reads it.

Reports, spreadsheets, APIs, applications, data science workflows, and manual processes may depend on it.

Treating schema changes as trivial

Changing integer to string is not merely metadata if consumers perform arithmetic on the value.

Evaluate semantic impact.

Validating only row counts

Matching counts can hide incorrect records and aggregates.

Use multiple reconciliation dimensions.

Switching consumers too early

A successful pipeline run does not prove production readiness.

Reconciliation should reach an agreed confidence level first.

Failing to define rollback criteria

Teams need measurable cutover and rollback conditions.

"Rollback if something looks wrong" is not sufficient.

Leaving ownership unclear

During parallel operation, determine who owns:

Legacy failures
Foundry failures
Reconciliation issues
Source-system changes
Consumer migration
Final cutover approval

Retiring legacy pipelines too early

Removing the previous processing path immediately after cutover eliminates a useful recovery option.

Retirement should follow validated operational stability and dependency confirmation.

Where Palantir Consulting Can Fit

A Palantir consulting team or experienced internal implementation group can be useful when the main challenge is architectural rather than simply writing transforms.

Typical areas of support may include:

Legacy dependency analysis
Migration boundary design
Foundry pipeline architecture
Incremental processing strategy
Schema compatibility planning
Reconciliation design
Data quality controls
Cutover planning
Operational handover

The important qualification is domain knowledge.

A technically elegant Foundry implementation can still fail a migration if the team does not understand the meaning of the legacy data, hidden consumer dependencies, or operational recovery requirements.

For that reason, Palantir Consulting Services should be evaluated partly on their ability to reason about the existing environment, not only their familiarity with the destination platform.

Migration Checklist

Before moving a pipeline into production, confirm the following.

Source and dependencies

[ ] Source systems are documented.
[ ] Source update behavior is understood.
[ ] Downstream consumers are identified.
[ ] Scheduling dependencies are known.
[ ] Hidden manual processes have been investigated.

Transformation design

[ ] Existing transformation rules are understood.
[ ] Incremental versus full processing has been decided.
[ ] Late-arriving data is handled.
[ ] Retry behavior is understood.
[ ] Idempotency has been considered.

Schema

[ ] Source schemas are documented.
[ ] Consumer schema contracts are defined.
[ ] Type changes are handled explicitly.
[ ] Compatibility risks are identified.
[ ] Historical recomputation requirements are understood.

Validation

[ ] Row counts are compared.
[ ] Key coverage is compared.
[ ] Important fields are reconciled.
[ ] Business aggregates are validated.
[ ] Null and duplicate rates are checked.
[ ] Business rules are tested.
[ ] Material differences are explained.

Cutover

[ ] Parallel execution has completed successfully.
[ ] Monitoring is in place.
[ ] Cutover criteria are documented.
[ ] Rollback criteria are documented.
[ ] Ownership is assigned.
[ ] Consumers are migrated deliberately.
[ ] Legacy retirement occurs only after validation.

Final Takeaway

A successful legacy ETL migration to Palantir Foundry is not primarily a code-conversion project.

It is a controlled change to the way enterprise data is produced and trusted.

Incremental processing can reduce unnecessary recomputation, but it requires careful handling of source changes, retries, historical state, and late-arriving data.

Schema evolution can support changing business systems, but only when compatibility and consumer impact are managed deliberately.

Reconciliation provides the evidence needed to determine whether the new pipeline is actually correct.

Finally, parallel execution creates room to validate those decisions before production consumers are fully committed to the new path.

The strongest migration plans therefore focus on three questions:

Is the data correct?

Can we explain every important difference?

Can we recover safely if the cutover fails?

If those questions can be answered confidently, the organization is doing more than moving ETL workloads. It is building a safer foundation for long-term enterprise data transformation.

Migrating Legacy ETL to Palantir Foundry: Incremental Transforms, Schema Evolution, and Reconciliation Without Downtime

Kavi Writes — Tue, 06 Oct 2026 07:13:40 +0000

Legacy ETL replacements rarely fail because the new platform can't run the transformations. They fail because nobody can prove the new outputs are correct, or because a finance report three teams away breaks on the first Monday after cutover.

A legacy pipeline has usually been around long enough to collect undocumented fixes, silent type coercions, and consumers nobody remembers onboarding. Rewriting its SQL in a new tool is the easy part. Keeping the behavior that the business actually depends on is the hard part.

This article covers three concerns that decide whether a migration to Palantir Foundry goes smoothly:

Incremental transformation: processing only what changed, instead of rebuilding everything nightly.
Schema evolution: handling source and output schema changes without breaking consumers.
Reconciliation: proving, with evidence, that the new pipeline produces equivalent results.

Running the legacy and Foundry pipelines side by side ties these together. Parallel operation turns cutover from a leap of faith into a decision backed by data.

Why Legacy ETL Migrations Are Difficult

Most legacy ETL estates share a familiar set of problems:

Monolithic jobs. A single workflow extracts, cleans, joins, aggregates, and loads, so you can't migrate one piece without understanding all of it.
Hidden dependencies. A downstream job reads an intermediate staging table that was never meant to be public.
Hard-coded logic. Currency rates, region mappings, and exclusion lists live inside stored procedures.
Fragile schedules. Job B starts at 02:30 because job A "usually finishes by then."
Inconsistent schemas. The same customer ID is a string in one system and an integer in another.
Limited observability. Failures surface as a stale dashboard, not as an alert.
Locked-in consumers. Reports, extracts, and applications depend on exact table names, column orders, and even rounding behavior.

A big bang migration bets that every one of these has been found and handled before go-live. In practice, some will only be found after go-live, when the old system is gone and there's nothing to compare against.

A Safer Migration Pattern

An incremental approach reduces that risk by migrating in bounded slices. One common lifecycle:

Inventory existing jobs, sources, targets, and schedules.
Identify critical data products and who consumes them.
Select a migration boundary small enough to validate fully, such as one subject area or one fact table and its dimensions.
Build the corresponding Foundry pipeline.
Run both pipelines in parallel on the same inputs.
Compare outputs systematically.
Resolve differences, classifying each as a new-pipeline bug, a legacy bug, or an accepted change.
Shift consumers gradually.
Retire the legacy pipeline once nothing reads from it.

This is an architectural pattern, not a mandatory Palantir methodology. Its value is that every step produces evidence, and the blast radius of a mistake stays limited to one slice.

Incremental Transforms in Foundry

Many legacy jobs follow a truncate-and-reload pattern: delete the target table, rebuild it from full source extracts. It's simple, but cost and runtime grow with total data volume rather than with the volume of change.

What Foundry provides

Foundry datasets are versioned through transactions. A build either commits a transaction or it doesn't, so a failed run doesn't leave a half-written output visible to consumers. Python transforms in Code Repositories support an @incremental decorator. When a transform runs incrementally, reading an input returns only the rows added since the last successful build, and the output can append rather than replace. Bumping the decorator's semantic_version forces a full recompute, which is useful when logic changes. Data Connection also supports incremental syncs for some source types, typically by tracking a monotonically increasing column such as an update timestamp or sequence ID. Pipeline Builder has its own incremental options; what's available can depend on your enrollment and pipeline type.

An important constraint: incremental transforms are built around appended data. If an upstream dataset is fully replaced by a snapshot, a downstream incremental transform generally can't run incrementally. Foundry datasets also aren't key-based tables with native upserts, so "latest state per key" has to be modeled explicitly.

Design concerns

Change detection. Prefer reliable change signals (CDC feeds, sequence numbers) over updated_at columns that applications forget to maintain.
Partitioning. Partition outputs by a column consumers filter on, such as business date, so reprocessing stays targeted.
Watermarks. Ingestion-side watermarks decide what gets pulled. Transform-side incrementality decides what gets processed. Keep the two concepts separate in your design.
Late-arriving data. An order updated today for last month's date lands in today's increment. Any aggregate by business date must recompute the affected dates, not just append.
Idempotency. Transactions make commits atomic, but your logic must still be deterministic. Avoid current_timestamp() inside business logic and deduplicate on stable keys.

Example: from full refresh to changelog plus current state

Legacy: a nightly job truncates fact_orders and reloads from a full ERP extract.

Redesign: ingest only changed rows into an append-only changelog, then derive the current state per order.

from pyspark.sql import functions as F, Window
from transforms.api import transform, incremental, Input, Output

@incremental(semantic_version=1)
@transform(
    out=Output("/Sales/clean/orders_current"),
    changes=Input("/Sales/clean/orders_changelog"),
)
def compute(changes, out):
    new_rows = changes.dataframe()  # only rows appended since last build
    previous = out.dataframe("previous", schema=new_rows.schema)

    w = Window.partitionBy("order_id").orderBy(
        F.col("updated_at").desc(), F.col("ingested_at").desc()
    )
    latest = (
        previous.unionByName(new_rows)
        .withColumn("rn", F.row_number().over(w))
        .filter("rn = 1")
        .drop("rn")
    )
    out.set_mode("replace")
    out.write_dataframe(latest)

The trade-off is visible: the extraction and changelog are incremental, but this current-state dataset is still rewritten each run. For very large tables, teams may partition the current-state output, or have consumers read from the changelog through a view. The decorator style above is the classic API; check your repository template for the syntax your Foundry version uses.

Managing Schema Evolution

Schema changes cause more migration incidents than transformation logic does, because they often fail silently.

Change	Typical risk
Column added	Low, unless consumers use `SELECT *` or positional reads
Field renamed	Breaks consumers; can look like a dropped column plus a new null column
Type changed	Silent nulls on failed casts, precision loss, timezone shifts
Field removed	Breaks consumers and business rules that depend on it
Nested structure changed	Breaks flattening logic, often deep inside the pipeline
Source version upgrade	Several of the above at once, often undocumented

Practices that help:

Schema contracts. Define the canonical output schema explicitly and treat changes to it as reviewed changes.
Explicit mappings. Map source fields to canonical fields by name, never by position.
Version-aware transforms. Handle known source versions deliberately instead of hoping the shape stays stable.
Validation before promotion. Foundry Data Expectations let a transform fail a build when checks don't pass, for example Check(E.primary_key("order_id"), "orders pk", on_error="FAIL"). Foundry also supports branching, so schema changes can be built and checked on a branch before merging.
Consumer impact analysis. Use Foundry's data lineage to see which downstream datasets and applications read a column before you change it.

Example: a source rename and type change

The ERP upgrade renames cust_id to customer_id and changes amount from string to decimal.

from pyspark.sql import functions as F, types as T

CANONICAL = {
    "order_id": T.StringType(),
    "customer_id": T.StringType(),
    "amount": T.DecimalType(18, 2),
    "updated_at": T.TimestampType(),
}
RENAMES = {"cust_id": "customer_id"}  # known source v1 -> v2 mapping

def to_canonical(df):
    for old, new in RENAMES.items():
        if old in df.columns and new not in df.columns:
            df = df.withColumnRenamed(old, new)
    missing = [c for c in CANONICAL if c not in df.columns]
    if missing:
        raise ValueError(f"Schema contract violated, missing: {missing}")
    return df.select([F.col(c).cast(t).alias(c) for c, t in CANONICAL.items()])

One catch: with Spark's default (non-ANSI) settings, a failed cast returns null instead of raising an error. Pair the mapping with a null-rate check on amount. Also, if the output schema of an incremental transform changes, appending to the existing output will typically conflict. Bumping semantic_version to force a full rebuild is the cleaner path.

Reconciliation Between Legacy and Foundry Pipelines

Matching row counts proves very little. Two tables can have identical counts while one has duplicated keys, swapped values, or nulls where the other has data.

Reconcile at several levels:

Level	What it catches
Row counts	Gross load failures, filter differences
Key coverage	Records present in only one output
Duplicate detection	Join fan-out, broken dedup logic
Aggregate totals	Sums by date, region, product; rounding drift
Null rates per column	Failed casts, broken mappings
Record-level comparison	Value differences on matching keys
Business-rule validation	Rules that must hold regardless of either pipeline
Sample-based review	Human inspection where full comparison is impractical

A common approach is to ingest the legacy output into Foundry via Data Connection, then compare in a dedicated reconciliation transform:

COMPARE_COLS = ["customer_id", "amount", "status"]

def row_hash(df):
    normalized = [F.coalesce(F.trim(F.col(c).cast("string")), F.lit("<null>"))
                  for c in COMPARE_COLS]
    return df.select("order_id", F.sha2(F.concat_ws("||", *normalized), 256).alias("h"))

legacy, foundry = row_hash(legacy_df).alias("l"), row_hash(foundry_df).alias("f")
result = legacy.join(foundry, "order_id", "full_outer").select(
    "order_id",
    F.when(F.col("f.h").isNull(), "only_legacy")
     .when(F.col("l.h").isNull(), "only_foundry")
     .when(F.col("l.h") != F.col("f.h"), "value_mismatch")
     .otherwise("match").alias("status"),
)

Normalization rules matter more than the comparison itself: decimal scale, timestamp timezones, trailing whitespace, empty strings versus nulls. Agree on documented tolerances up front, and record every accepted difference with a reason. Some mismatches turn out to be legacy bugs, and the business has to decide whether to fix or preserve them.

Running Both Pipelines Without Downtime

"Without downtime" here means consumers keep getting correct data on schedule throughout the migration. Whether cutover itself is invisible depends on your architecture.

Dual processing. Both pipelines run on the same inputs and schedule.
Shadow validation. Foundry outputs are reconciled every run, but nobody consumes them yet.
Consumer-by-consumer migration. Move low-risk consumers first, critical ones last.
Read-path switching. Give consumers a stable interface, such as a warehouse view, so switching means redirecting the view, not editing every report. Where Foundry needs to feed an external system, Data Connection exports may handle this, depending on the target.
Rollback planning. Define measurable rollback triggers, like mismatch rate above threshold or a missed freshness SLA, and keep the legacy path warm until the window closes.
Monitoring. Foundry health checks can cover freshness, build status, and schema. Monitor the reconciliation results as well.
Ownership. Name who fixes a broken build on each side during the transition. Ambiguity here causes most parallel-run incidents.

A Practical Migration Example

A hypothetical scenario. A retailer runs a nightly ETL job that pulls customers from a CRM, orders from an ERP, and payments from a payment gateway, then loads a reporting warehouse star schema used by dozens of reports.

Legacy pipeline. Full extracts land in staging, stored procedures build dimensions and fact_orders, and reports read the warehouse directly.

Foundry pipeline. Data Connection syncs land raw data, incremental where the source exposes a reliable change column. Python transforms build changelogs, apply the schema contracts, and produce current-state customer, order, and payment datasets. Data Expectations enforce primary keys and non-negative amounts.

Parallel validation. Both pipelines run nightly through at least one month-end close, since period-end adjustments and late payments often surface edge cases that daily runs don't.

Reconciliation. Early runs show value mismatches on amount. The cause is a legacy procedure that rounds at line level while the Foundry transform rounds at order level. Finance chooses line-level rounding, and the Foundry logic is adjusted to match.

Consumer migration. Operational dashboards move first through view redirection, then finance reports after a clean month-end.

Legacy retirement. Once query logs show no reads from the legacy tables for an agreed period, the jobs are disabled, archived, and eventually removed.

Common Migration Mistakes

Migrating everything at once, which removes the ability to compare and roll back.
Ignoring downstream dependencies, especially consumers of intermediate tables.
Treating schema changes as column edits instead of contract changes with consumers.
Validating only row counts.
Switching consumers before reconciliation is stable across a full business cycle.
No rollback criteria, so rollback becomes a debate during an incident.
Unclear ownership during the parallel period.
Retiring legacy jobs too early, before quarterly or annual processes have run.

Where Palantir Consulting Fits

Some organizations bring in Palantir consulting or implementation partners for specific parts of the program: migration architecture, legacy dependency analysis, incremental pipeline design, schema strategy, reconciliation frameworks, deployment planning, and operational handover. The most useful engagements leave the internal team able to own and extend the pipelines afterward. If you use outside help, make knowledge transfer an explicit deliverable.

Migration Checklist

Source inventory: systems, extraction methods, volumes, change signals
Dependencies: all consumers, including intermediate-table readers
Transformation logic: documented, including undocumented fixes in legacy code
Incremental strategy: append versus snapshot, late data, recompute triggers
Schema compatibility: contracts, explicit mappings, versioning rules
Data quality: expectations enforced at build time
Reconciliation: multi-level checks with agreed tolerances
Parallel execution: same inputs, same schedule, through a full business cycle
Monitoring: freshness, build health, reconciliation trends
Cutover: consumer order, read-path switching, communication plan
Rollback: measurable triggers and a warm legacy path
Legacy retirement: read-log evidence, archival, decommission date

Final Takeaway

Moving ETL to Palantir Foundry is not mainly a code-translation exercise. The real work is controlling correctness, dependencies, and operational risk while the business keeps running. Incremental processing keeps the new pipeline efficient, schema contracts keep it stable, and reconciliation gives you the evidence to cut over. Migrate in slices, compare relentlessly, and retire the old system only when the data says it's safe.

Designing a Palantir Foundry Ontology That Holds Up in Production: Object Types, Link Types, and Action Governance

Kavi Writes — Tue, 29 Sep 2026 10:40:04 +0000

A Foundry pilot usually works. A small team maps a few datasets into the Ontology, builds an operational app, and a planner can finally see open orders next to supplier delays in one place.

Then the second and third use cases arrive. A Customer object means "billing account" to one team and "legal entity" to another. A link that looked like obvious double-counts shipments. An Action that updates a status field has no clear owner, and nobody can say who changed what last Tuesday, or why.

These are rarely platform limitations. They are design decisions made quickly during a proof of concept and then inherited by production. This article covers how to design a Palantir Foundry Ontology that survives that transition: Object Types, Link Types, properties, and the Action governance that connects the model to real workflows.

What a Production-Ready Foundry Ontology Actually Needs

The Foundry Ontology is a semantic layer over your datasets. It maps rows and columns to business concepts through four main building blocks:

Object Types: the entities people work with, such as a Work Order, Supplier, or Aircraft.
Properties: the attributes of those entities.
Link Types: the relationships between Object Types.
Action Types: defined, permissioned ways to change objects and links.

Applications built on Foundry, from operational apps to SDK-based custom software to AI workflows, read and write through this layer. That is what makes it powerful, and it is also why design mistakes spread quickly.

In a pilot, the Ontology is a convenience. In production, it behaves like a public API with many consumers. It needs stable meaning, named owners, controlled change, secure access, and writes you can audit. The rest of this article is about building those properties in from the start.

Start from what people act on

A useful test for a candidate Object Type: would someone open it, assign it, approve it, or track it? If the answer is no, it is probably a property, a lookup, or a pipeline detail.

Avoid creating one Object Type per source table. ERP systems often split one business entity across header, item, and status tables. The Ontology should present the entity users recognize, with the joining done upstream in pipelines.

Granularity takes judgment. Should a Work Order and its individual operations be one Object Type or two? Split them when they have different lifecycles, owners, or access rules. Keep them together when users only ever treat them as one unit.

Choose primary keys that never move

The primary key is the most consequential decision on an Object Type. Links, user edits, and downstream references all depend on it, so changing it later is expensive.

A good key is unique, deterministic, and stable across pipeline rebuilds. Avoid row numbers, surrogate IDs that are regenerated with each run, and natural keys that are reused, e.g., reissued badge numbers. When objects come from multiple source systems, prefix the key with the system and site (for example, SAP-DE01-4500012345) to prevent collisions after a merge.

Be selective with properties

Include the properties people filter on, decide with, or need an AI workflow to reason about. Leave the other 180 source columns in the dataset. Use business names, explicit units, UTC timestamps, and a clear title property so objects are readable in search results and apps.

Getting Link Types Right

A Link Type connects two Object Types with a defined cardinality. One-to-many links are typically resolved through a foreign key property, while many-to-many links are backed by a join dataset.

The goal is to model relationships the business talks about, not every foreign key in the source schema. "Supplier supplies Part" is meaningful. "Table A references column B in a staging table" is not. Name both directions clearly, such as Supplied Parts from the Supplier side and Approved Suppliers from the Part side.

When a link should become an object

Some relationships carry their own data: effective dates, contract prices, approval status. If you need that, consider modeling the relationship as its own Object Type, like a Supplier Part Approval between Supplier and Part. You pay for an extra hop in queries, but you gain history, a lifecycle, and separate permissions. For a simple association with no attributes, a plain link is the better choice.

Verify cardinality against real data

A link declared one-to-many that is actually many-to-many in the data will not always fail loudly. It can produce misleading counts and incomplete traversals instead. Test cardinality assumptions in the backing pipelines before you publish the link.

From Data Model to Operational Model

Data quality belongs upstream

The Ontology reflects whatever its backing datasets contain. Duplicate keys, null foreign keys, and orphaned link targets show up as confusing objects in front of users. Put checks where the data is produced: key uniqueness, required fields, and referential integrity for every link, using Foundry's data health checks and pipeline expectations.

Ownership and lineage

Give every Object Type a named business owner who decides what it means and a technical owner who maintains its pipeline. Foundry's lineage shows how data flows from source to backing dataset, which helps with impact analysis. Lineage does not explain meaning, though, so write real descriptions for Object Types and properties.

Change over time

Decide early whether an Object Type represents current state, history, or both. A common pattern keeps the entity as current state and adds a separate event Object Type, such as Status Change, for history.

Treat schema changes like API versioning. Adding a property is usually safe. Renaming or removing properties, changing types, or altering a primary key can break apps and SDK consumers. Deprecate first, notify consumers, and use review workflows such as ontology branching or proposals where your environment supports them.

Designing for AI applications

AI workflows that reason over the Ontology depend on the same clarity humans do: precise names, useful descriptions, and unambiguous links. When an AI workflow is allowed to propose or make changes, the Action becomes the safety boundary. It should get the same permissions, validation, and audit trail as any human user, not a shortcut around them.

Designing Actions With Governance in Mind

An Action Type is a parameterized, permissioned transaction against the Ontology. It can create, modify, or delete objects and links, and it can trigger side effects such as notifications or webhooks. Simple Actions are configured with rules; complex ones can be backed by functions. Actions are how the Ontology stops being a read-only view and starts supporting decisions.

They are also where most production risk lives. Ontology governance that controls reads but not writes is only half done.

Keep Actions narrow

A single "Update Work Order" Action with 30 optional fields is hard to secure and harder to audit. Separate Actions like Reassign Work Order and Close Work Order each have one purpose, one set of eligible users, and one clear validation rule.

The five governance questions

Concern	What to define
Permissions	Who can submit, separate from who can view. Grant through groups, not individuals.
Validation	Submission criteria on the user and parameters, enforced server-side, not just in the UI.
Ownership	A named owner, a documented purpose, and the expected callers: people, automations, or AI.
Auditability	Who changed what, when, and why. Add a required justification parameter for sensitive changes.
Approval	Whether a second person must confirm before the change takes effect.

Approvals need explicit design

For changes that need sign-off, a portable pattern is a request object with a status, plus a separate approval Action limited to approvers. Submission criteria can block self-approval by checking that the approver is not the requester. If your environment offers a native approval capability, evaluate it against this pattern, but make the approval path visible in the model either way.

A Practical Enterprise Example

Consider a hypothetical manufacturer that wants to manage supplier quality problems across several plants. Today, incidents live in spreadsheets, and putting a supplier on hold means an email thread with procurement.

Object Types and Link Types

Supplier, keyed by a global vendor ID, not a per-plant ERP code.
Part, linked many-to-many to Supplier through an approved supplier list.
Plant, the site where incidents are raised.
Quality Incident, linked to one Part, one Plant, and one Supplier.
Supplier Hold Request, linked to one Supplier and to the incidents that justify it.

Actions

Action	Who can submit	Key validation	Effect
Log Quality Incident	Plant quality engineers	Part is active at the selected Plant; severity required	Creates an incident and its links
Request Supplier Hold	Quality leads	At least one linked incident; justification required	Creates a request with status Pending
Approve Supplier Hold	Procurement leads	Approver is not the requester; request is Pending	Sets the Supplier's hold status and notifies buyers
Release Supplier Hold	Procurement leads	Supplier is currently on hold; reason required	Clears the hold and records the release

An AI assistant could summarize open incidents for a supplier and draft a hold request. Only a procurement lead can submit the approval, so the most consequential step stays with a human.

One decision remains outside the Ontology: which system owns the hold status. If the ERP blocks purchase orders, the approved hold must reach it, for example through a webhook side effect or an outbound sync. Define the system of record explicitly, or two systems will disagree about whether a supplier is blocked.

Common Ontology Design Mistakes

Mirroring the source schema. The Ontology becomes a harder-to-query copy of the ERP instead of a model of the business.
Unstable primary keys. Keys that shift on rebuild break links and orphan user edits.
Catch-all Object Types. A single Asset type covering vehicles, buildings, and software licenses forces dozens of mostly-null properties and muddled permissions.
Generic write Actions. "Edit Object" style Actions make validation and audit nearly meaningless.
Governing reads but not writes. Row-level access is carefully designed, while any editor can submit any Action.
No owners or descriptions. Six months later, nobody knows whether status means ERP status or operational status.
Using the Ontology as a reporting layer. Heavy aggregations usually belong in datasets and analytics tools, with the Ontology focused on entities people act on.
Breaking changes without a consumer inventory. Renaming a property can quietly break apps, automations, and SDK clients you did not know existed.

Production Readiness Checklist

Use this before promoting an Object Type, Link Type, or Action to production.

Model

Each Object Type represents something users open, assign, approve, or track.
Primary keys are unique, deterministic, and stable across rebuilds and source systems.
Properties include explicit units and business names. Timezones are all the same.
The link cardinality is tested on real data and both directions are explicitly stated.

Data and ownership

Backing datasets have health checks for key uniqueness, required fields, and link integrity.
Every Object Type has a business owner, a technical owner, and a written description.
History requirements are decided: current state, event objects, or both.
Consumers are known, and there is a deprecation process for breaking changes.

Actions

Each Action has a single purpose and a named owner.
Submission permissions are group-based and distinct from read access.
Validation is performed server-side via submission criteria or function logic.
Sensitive Actions require a justification and, where needed, a second approver.
AI-initiated changes go through the same Actions and controls as human changes.
The system of record is defined for any value that is written back to external systems.

Final Takeaways

A production Foundry Ontology is a contract, not a diagram. Object Types should reflect the entities people act on, Link Types should capture relationships the business actually uses, and primary keys should be treated as permanent from day one.

Actions deserve the most care, because they are where the model changes the business. Keep them narrow, permission them deliberately, validate server-side, and make approvals and audit trails explicit.

None of these rules is absolute. Granularity, history, and the choice between a link and an intermediate object depend on your workflows. Whether the work is done by an internal platform team or with a Palantir consulting partner, the useful test is the same: can a new team understand, trust, and safely change this Ontology without asking the people who built it?