If you've spent any time tracking privacy-preserving AI, you know the running joke about Fully Homomorphic Encryption (FHE): it’s the holy grail of cryptography, and it always will be. Running inference on encrypted data without decrypting it sounds amazing on paper—until you hit the brutal compute overhead and realize you need a PhD in lattice cryptography just to run a linear regression.
Google’s HEIR is an open-source Fully Homomorphic Encryption (FHE) compiler that dramatically simplifies the deployment of privacy-preserving AI. It automates complex cryptographic setups for models trained in PyTorch or TensorFlow, converting what was previously a multi-month research project into a streamlined, week-long engineering workflow for encrypted inference.
Today, Google engineers open-sourced HEIR, an FHE compiler that aims to fix the developer-experience bottleneck once and for all.
Instead of forcing teams to hand-roll custom cryptographic circuits every time a model architecture changes, HEIR treats encrypted inference as a compiler problem. You feed it a standard model trained in PyTorch or TensorFlow, and it outputs a deployable, encrypted binary.
Here is a breakdown of why this matters for the AI hardware and systems ecosystem.
—
The Engineering Shift: From Manual Math to Automated Toolchains
Until now, deploying FHE meant writing low-level cryptographic code by hand. If a machine learning engineer tweaked a single layer in a PyTorch network, the crypto team often had to rewrite the underlying noise management and encryption parameters from scratch.
HEIR bridges this divide. It serves as an intermediate representation layer that automates parameter selection, layout optimization, and target generation. It doesn't eliminate the fundamental overhead of FHE, but it compresses what used to be a multi-month research project into a workflow any competent software engineer can integrate in under a week.
Google first signaled this direction back in 2023. Since then, four specialized hardware acceleration startups—Belfort Labs, Niobium, Cornami, and Optalysys—have joined the ecosystem to build hardware targets against HEIR's intermediate representations.
—
4 Production Demos on Plain CPUs
The team didn't just drop a whitepaper; they released four open-source end-to-end demos running on standard single-threaded CPUs, with full latency benchmarks publicly logged:
1. Deep Learning Recommendation Systems (DLRM): Built with Belfort Labs, LG, and NYU. User feature vectors remain fully encrypted end-to-end; the cloud model scores recommendations without ever viewing raw user behavior.
2. Credit Card Fraud Detection: Developed alongside Niobium and hardshell.ai. The model evaluates transaction anomalies while cardholder metadata remains completely encrypted.
3. Network Intrusion Detection (Kitsune): Analyzes network traffic streams to flag malicious patterns without the cloud provider ever seeing inside raw packet payloads.
4. Voice Keyword Spotting: Implemented with Belfort Labs to detect trigger phrases locally on edge audio without streaming unencrypted mic audio to an external server.
Across these benchmarks, single-thread CPU latencies range from 127 milliseconds to 1.8 seconds. That is still too high for hyper-scale real-time web traffic, but for asynchronous, high-stakes workloads, it is genuinely practical today—even before ASIC acceleration kicks in.
—
Why Pure Cryptography Beats Hardware Enclaves
Google intentionally took a pure cryptographic approach here rather than leaning on Trusted Execution Environments (TEEs) or confidential hardware enclaves.
TEEs rely on silicon vendor trust, complex firmware attestation, and are historically vulnerable to side-channel leakage. FHE’s security boundary, by contrast, is anchored entirely in mathematics. Even if the host machine is fully compromised, the underlying data remains unreadable without the client-side key.
Over the past three years, the raw algorithmic overhead of FHE has dropped faster than most hardware builders anticipated. HEIR translates those algorithmic breakthroughs into developer accessibility. It is already being used as a research platform across Georgia Tech, CMU, Tsinghua University, and more than a dozen other institutions, underpinning four peer-reviewed papers.
—
What This Means for Startups and Hardware Builders
For enterprise teams in healthcare, fintech, and defense, the architectural pattern changes immediately. You don't have to ship fragile on-premise model deployments to paranoid enterprise clients anymore, nor do clients have to hand over raw data. They encrypt locally, run inference on your cloud infrastructure, and decrypt the result locally.
Several financial institutions are already rolling this pattern into gray-box testing for Anti-Money Laundering (AML) pipelines.
As a hardware founder, this is the milestone I’ve been waiting for: the software abstraction layer has stabilized. Once compilers like HEIR give developers a unified target, custom silicon and optical accelerators finally have a fixed target to accelerate.
All demo source code is live in Google's GitHub repository. Hardware-accelerated benchmarks are slated to drop next.
