How do you keep consistent tokens across warehouses like Snowflake and Databricks?
You apply one keyed transform. Use one shared key. Do this before or outside the warehouse. That habit alone gives you the join-preserving trait.
Deterministic tokenization means one thing. Same plaintext, same key, same token, every time. Google Cloud puts it plainly. With a deterministic transform, "a table of data will be replaced with the same obfuscated form each time it is transformed, which ensures that connections between values (and, with structured data, records) are preserved, even across tables". That preserved link is referential integrity. A join needs just that. It is the trait a consistent service guarantees. Joins still run on that data. So does aggregation.
The join-preserving trait needs just one thing. Same input, same key, same output. The compute engine does not matter here. Snowflake does not track how a token got minted in Databricks. It does not need to. Say one vault drives one HMAC over the customer id. Both warehouses get the output of that one transform. The token in Snowflake then matches the token in Databricks. Byte for byte. A join on the token column returns real matches.
This is the core of how DataShield's Ontology works. One customer-held key drives an HMAC. A value tokenizes the same way everywhere it lands. One key, one algorithm, one token. Run two tokenizers on their own. You get two other tokens for the same person. The join fails before it starts.
Why do native Snowflake and Databricks masking policies break the join?
Each platform's native control is scoped to that one platform. Each was set up alone. They were never built to agree with each other. Expecting them to agree is like two teams. Both pick the same random password, by chance.
Snowflake's column-level security gives you Dynamic Data Masking and External Tokenization. Per the docs, masking policies are schema-level objects. A database and schema must exist in Snowflake first. Only then can the policy touch a column. The policy runs at query time. It runs everywhere that column shows up. External Tokenization lets you tokenize before load. It lets you detokenize at query time, through outside functions. That is the useful gap. The masking-policy plumbing stays Snowflake-specific. Even when the token provider is not. That policy object lives inside a Snowflake account. It cannot govern a Databricks table.
Databricks works the same way, in reverse. In Unity Catalog a column mask is a SQL UDF. It takes the column value. It hands back the real value, or a masked one. One mask per column. You wire it up with ALTER TABLE ... ALTER COLUMN ... SET MASK. Or with an ABAC policy on controlled tags. Those UDFs run inside Databricks. They cannot govern a Snowflake table.
Run both on their own. You get other keys, other algorithms, other UDF logic. The same customer id lands as two other tokens. Any join across the two warehouses, on that column, fails. No clever policy config fixes this, on either side. The fix is one shared keyed transform, applied outside each engine. Both warehouses then get tokens that already agree. Note this. External Tokenization and Databricks UDFs can both call one same outside engine. That is the sanctioned way to make them agree, not fight.
The reverse direction works the same way. It is worth walking once. A special Snowflake query hits a masking policy. That policy fires an External Tokenization outside function. That function calls your outside engine. The engine looks the token up in the vault. Or it decrypts, if you chose a reversible cipher. Either way it returns the plaintext. A special Databricks query hits a Unity Catalog UDF. That UDF calls the very same engine. Same vault, same lookup. Both warehouses detokenize through one authorization point. The reverse mapping lives in just one place. A place you can audit and revoke. Not two places that can drift apart.
What does deterministic tokenization actually preserve, and what does it destroy?
Deterministic tokenization keeps equality. It destroys almost every other trait. Plan your pipeline around that. You avoid most of the pain.
Equal plaintext yields equal token. So equi-joins, dedup, GROUP BY, and COUNT DISTINCT all still work on the token column. Order, ranges, and substrings do not survive. A LIKE '%smith%' on a tokenized name column returns nonsense. This tears up the byte structure of the plaintext. Google's deterministic transform emits one of two forms. One is a base64 AES-SIV ciphertext, reversible with the key. The other is an HMAC digest, not reversible. Either way, neither form bears any substring relationship to the input.
Here is the honest engineering reality. A match is the only thing you get for free. Does a query need fuzzy match? Or prefix search? Or a range filter on a sensitive field? Then tokenization is the wrong tool for that field. No key setup changes that. Decide which columns are join keys. Decide which ones need in-the-clear semantics. Do this before you tokenize. Not after you ship, with analysts filing bug reports. I walk through the ingest-time version of this call. See tokenizing PII before it reaches the LLM.
Why is a deterministic token also a linkability oracle?
Determinism cuts two ways. Equal plaintext always maps to equal token. That is what makes tokens joinable. It is also what turns them into a re-identification oracle.
Anyone holding the key can guess a token back. Feed a dictionary of likely inputs through the transform. Compare the outputs against your token column. You have known-plaintext re-linking. Low-cardinality fields are the soft target. Sex, ZIP, birth year, a country code. There are few values to guess. A key holder can list them all in seconds. This linkability idea is standard in pseudonymization. It is not something the Google doc itself flags. Credit the general de-identification literature. Not any one vendor page.
Here is the standard defense. Let the key double as a secret pepper. An attacker with no key cannot list any of it. The HMAC mixes in a secret. That secret never appears in the data. A dictionary of sex, ZIP, and birth-year values still gets them nowhere. That is just why low-cardinality fields survive at all. The entropy lives in the key, not the value. Pair the pepper with per-field key split. A single leaked key then re-links one field. Not your whole low-cardinality estate. The risk that remains is the key holder. The security of the whole scheme collapses to key custody. Treat customer-held keys as a main control here. Not an optional add-on.
On cipher choice: NIST sets format-preserving modes FF1 and FF3. In Special Publication 800-38G. Both are modes of AES. FF3 was later found to have cryptanalytic flaws. It was revised to FF3-1 in the revision SP 800-38G Rev. 1. Take that as a warning. Pick a sound keyed cipher. That part is not optional. Rolling your own is not an answer. On compliance: GDPR treats pseudonymized data as personal data. This is under Recital 26 and Article 4(5). Tokens are pseudonymous PII in regulatory scope. Not anonymized and out of scope. Treat them like the PII they stand in for.
What do agents and the OWASP Agentic Top 10 have to do with tokens?
Agents change the picture. The moment you let one loose across both. A join-preserving token lets that agent work over Snowflake and Databricks. Without ever touching raw PII. It joins, dedups, and reasons on TOK_ values. Only a separate, allowed detokenization path can reverse them. One holding the vault key.
Agents are a fresh attack surface. The industry now has a list for it. The OWASP Top 10 for Agentic Applications came out December 9, 2025. It is the first flagship list built for autonomous agents. It runs ASI01 through ASI10. The top category is ASI01 Agent Goal Hijack. Attackers hide new goals in documents, emails, or RAG results. The agent then treats them as orders. OWASP cites EchoLeak as an example.
EchoLeak is tracked as CVE-2025-32711. It was disclosed in June 2025, at CVSS 9.3. It was a zero-click prompt injection, in Microsoft 365 Copilot. A single crafted email made Copilot exfiltrate internal data. With no user interaction. Aim Labs responsibly disclosed it. It was patched server-side. No confirmed exploitation in the wild. An independent academic case study later walked through the exploit chain. The authoritative severity and patch record lives in the CVE-2025-32711 advisory.
The related MCP tool-poisoning class works the same way. Simon Willison documented it in April 2025. Bad instructions get tucked into a tool's description. The model reads it. The user never sees it happen. It is a proof of concept, not a breach.
Tokens earn their keep in this setting. Say a hijacked agent gets talked into exfiltrating your join key. It walks off with opaque TOK_ values. Not names and card numbers. That shrinks the blast radius of an ASI01 goal-hijack. Down to pseudonyms. The Model Context Protocol security best practices push the same way. Spec revision 2025-06-18. From the identity side. MCP servers acting as OAuth proxies must build consent, per client. This avoids the confused-deputy trap. Clients must include the RFC 8707 resource parameter. Each access token then binds to one specific server. Token binding and data tokenization share one instinct. Assume the agent will be manipulated. Make what it can reach worth as little as possible.
Best practices to prevent token leakage across warehouses
This is an engineering threat model. Not a full compliance program. Read the list as a floor. Build up from there. Six things keep tokens consistent:
- Hold your own keys, with tight custody. The whole scheme's security collapses to key custody. This control matters most. Say the token provider holds the key. A vendor now owns your re-linking risk.
- Rotate with versioned keys. Never swap the key underneath live tokens. Here is the trap that turns a control into an outage. Rotate an HMAC key the naive way and each token changes. That quietly breaks each old join the tokens were built to keep. The fix is a key-generation id. Stamp it alongside each token. Or namespace it in. Plan a re-tokenization pass at cutover, too. Take an old token and a new token, for one person. They must never look like a match. Rotation belongs on this list. A careless rotation is the exact failure this design stops.
- Separate keys per context or per field. Use a different key for marketing than for clinical data. A leaked key then re-links one domain, not your whole estate. That is what keeps low-cardinality fields safe.
- Treat tokens as PII still in scope. Pseudonymized data is not anonymized data. Keep them under the same access rules, retention limits, and audit. Same as the plaintext they represent.
- Put detokenization behind strong authz and audit. Reversing a token should be a special, logged, revocable action. No service gets it, by default. Bind it to identity. The way our auth layer binds each tool call.
- Design for equality only. Equi-join and dedup are the operations you get. Say a pipeline quietly leans on substring or range behavior. Over a tokenized column. It fails in ways that look like data-quality bugs. That is the hardest kind of bug to find. Catch it at design time.
None of this is rare. Mostly it means one thing. Do not let convenience quietly rebuild the plaintext you removed.
How do you govern the data plane, not just the agent?
Most agent-security advice stops at the prompt: guardrails, refusals, input filters. That governs what the agent will do. It does nothing about what the agent can reach. Both layers count. The data plane keeps consistent tokens across warehouses instead of raw PII. It is the layer most teams skip.
Governing the reachable surface looks like this. Tokenize sensitive fields at ingest. A hijacked agent then finds tokens. Not the raw PII it hoped for. Authorize per tool call, with mid-session revocation. A session that turns hostile gets cut off, mid-flight. Not at the next login. Seal each call into a tamper-evident audit chain. One you can verify after the fact. An incident review then becomes a matter of proof. Not taking anyone's word. That is the shape of the DataShield architecture. The tokenization piece is deliberately kept outside the warehouse. One customer-held vault drives one HMAC. It lands the same way, in Snowflake and Databricks. That is exactly why the cross-warehouse join survives.
Two honest caveats. Hear them from me now, not later. DataShield's join-consistency and HMAC-vault behavior described here are the product's stated features. They are not claims you can check against public sources. Keep that in mind. DataShield does not have a SOC 2 attestation yet. It is planned, not attested. Is a token vault vendor cagey about either one? That tells you something. Want to pressure-test the join-consistency claim against your own two-warehouse setup? Start a scoped evaluation. Bring your ugliest cross-platform join.
Watch: related explainers
Background on tokenization and the agent-security context around it.
Frequently asked questions
Can I run a LIKE or substring query on a tokenized column?
No. Deterministic tokenization keeps equality and equi-joins only. It tears up the byte structure of the plaintext. So LIKE, prefix search, and range filters fail on the token. If a field needs those, do not tokenize it. Or keep a separate, controlled path for it.
Do Snowflake and Databricks produce the same token for the same value? By default, no.
No. Snowflake masking policies are schema-level objects, scoped to a Snowflake account. Databricks column masks are Unity Catalog UDFs, scoped to Databricks. Set up on their own, they use other keys and other logic. So they tokenize the same customer id into two other tokens. That breaks any cross-warehouse join. You need one shared keyed transform, run before both engines. Or both engines calling one shared outside service.
Are deterministic tokens the same as GDPR anonymized data?
No. Deterministic tokens are pseudonymous data. Under GDPR (Recital 26 and Article 4(5)), pseudonymized data is still personal data. It stays in regulatory scope. Keep tokens under the same access rules, retention, and audit as the plaintext they stand in for. Do not treat tokenization as an off-ramp from compliance.
What happens if the tokenization key leaks?
The token becomes a re-identification risk. Anyone with the key can push guessed values through the same transform. They then match the outputs to your token column. That re-links records. Low-cardinality fields like sex, ZIP, and birth year are the most exposed. That is why customer-held keys are a top control. So are per-context key splits and versioned key rotation.
Which keyed algorithm should I use for cross-warehouse tokens?
For join keys you never need back, a keyed HMAC over the value works. That is what DataShield uses. Need format-preserving, reversible tokens instead? NIST sets FF1 and FF3-1 in Special Publication 800-38G. Use FF3-1: FF3 had flaws. Whatever you pick, keep the key somewhere you control. The whole scheme comes down to who holds that key.