Encryption at Rest

A field tagged mfx:"encrypted" is automatically encrypted before it reaches the database and decrypted on read. The plaintext never appears in the table; the column stores a self-describing envelope. This page covers the full subsystem: the tag, the key provider interface, the storage format, unique-constraint handling, and key rotation.

Declaring an encrypted field

type Patient struct {
    maniflex.BaseModel
    Name string `json:"name" mfx:"required,filterable,sortable"`
    SSN  string `json:"ssn"  mfx:"encrypted,key:patient-pii"`
}

The column’s Go and DB types remain string. Storage is the prefix enc: followed by a base64-encoded binary envelope:

enc:Aa1z...   (envelope bytes embed the keyID)

The enc: prefix lets the framework distinguish ciphertext from any legacy plaintext that may exist in the column — useful for incremental migration of an existing table.

What encryption costs you

The trade-offs are deliberate and worth being explicit about:

• No filtering. A WHERE ssn = ? would have to match an envelope that includes a random nonce. Encrypted fields cannot be filterable.
• No sorting. Same reason. Encrypted fields cannot be sortable.
• Uniqueness via HMAC. A mfx:"encrypted,unique" field gets a companion {field}_hmac TEXT UNIQUE column. See the next section.
• The KeyProvider is required. Reads degrade to returning the raw stored ciphertext; writes are rejected with 500 ENCRYPTION_NOT_CONFIGURED until a provider is configured.

For columns that need to be queryable but contain sensitive data, store a non-sensitive lookup key (a hashed identifier) in a separate field and encrypt only the payload.

Access paths

Encryption is applied on every access path, not just HTTP:

• HTTP pipeline — automatic.

• Typed helpers (maniflex.Create / Read / Update / List) and ctx.GetModel(name) — encrypt on write and decrypt on read using the ServerContext’s KeyProvider (set automatically inside a request).

• Background workers / CLIs using maniflex.NewBackground(...) — call bg.SetKeyProvider(srv.KeyProvider()) so typed access to encrypted models encrypts and decrypts:

  bg := maniflex.NewBackground(ctx, srv.DB(), srv.Registry())
  bg.SetKeyProvider(srv.KeyProvider())
  p, _ := maniflex.Read[Patient](bg, id) // decrypted
  

The raw adapter (srv.DB().Create/FindMany) is deliberately encryption-agnostic — it stores and returns ciphertext verbatim, which is what RotateEncryptionKey relies on. Use the typed helpers or ctx.GetModel for transparent crypto.

Configuring a KeyProvider

maniflex.Config.KeyProvider must be set before any model with encrypted fields is exercised. Two implementations ship in pkg/encryption, both constructed as plain struct literals.

EnvKeyProvider — keys from environment variables

import "github.com/xaleel/maniflex/pkg/encryption"

server := maniflex.New(maniflex.Config{
    KeyProvider: &encryption.EnvKeyProvider{
        Prefix:     "MYAPP_KEY",
        IndexKeyID: "blind-index",
    },
    // ...
})

The env var name for a given keyID is derived as {Prefix}_{KEYID_UPPER}, with hyphens replaced by underscores and the result uppercased:

Each variable holds a base64-encoded 32-byte (256-bit) AES key. Generate one with:

openssl rand -base64 32

The provider accepts either standard or URL-safe base64.

VaultKeyProvider — HashiCorp Vault Transit

server := maniflex.New(maniflex.Config{
    KeyProvider: &encryption.VaultKeyProvider{
        Address: "https://vault.example.com",
        Token:   os.Getenv("VAULT_TOKEN"),
        Mount:   "transit",            // optional, default "transit"
        IndexKeyID: "blind-index",      // dedicated Transit HMAC key
        // TokenSource: tokenSource,    // optional, refreshed per request
        // Client: customHTTPClient,    // optional, cloned before use
    },
})

keyID maps to a Vault Transit key name; the plaintext is sent to /v1/{mount}/encrypt/{keyID} and Vault returns its own vault:v1:... ciphertext, which the provider embeds in the envelope. A Vault key rotation is transparent — Vault decrypts ciphertexts encrypted with any prior version of the key automatically.

Vault addresses must use HTTPS. AllowInsecureHTTP is an explicit opt-out for isolated development or test environments and must not be enabled in production. The provider uses a private HTTP client with a 10-second whole-operation timeout by default. An earlier deadline on the request context wins. Set Timeout explicitly to tune the bound; a negative value disables the client timeout and leaves the operation bounded only by its context.

For renewable AppRole, Kubernetes, JWT, or other short-lived authentication, implement VaultTokenSource (or use VaultTokenSourceFunc). It is called before every Vault request and takes precedence over the static Token.

A custom backend implements the maniflex.KeyProvider interface:

type KeyProvider interface {
    Encrypt(ctx context.Context, keyID string, plaintext []byte) ([]byte, error)
    Decrypt(ctx context.Context, envelope []byte) ([]byte, error)
    KeyIDOf(envelope []byte) (string, error)
    HMAC(ctx context.Context, keyID string, data []byte) ([]byte, error)
}

type BlindIndexKeyProvider interface {
    BlindIndexKeyID() string
}

Encrypt returns a self-describing binary envelope that embeds the keyID. Decrypt reads the keyID from the envelope, so callers don’t supply it. HMAC produces a deterministic keyed digest used for unique indexes. BlindIndexKeyProvider is an optional capability. The shipped providers implement it through IndexKeyID; custom providers should implement it when encrypted+unique models need online key rotation.

Unique encrypted fields

A normal UNIQUE constraint on an envelope is useless — each envelope contains a random nonce, so two encryptions of the same plaintext are different ciphertexts. The framework solves this with an HMAC companion column.

Email string `json:"email" mfx:"encrypted,unique"`

AutoMigrate emits two columns:

On every write, the DB step calls KeyProvider.HMAC with IndexKeyID and stores the result in the companion. The index key must be dedicated to this purpose and must not rotate with field-encryption keys. The HMAC is deterministic for a given (index key, plaintext) pair, so the database can enforce uniqueness without ever seeing the plaintext.

If IndexKeyID is omitted, writes retain the legacy behavior of using the field-encryption key for compatibility, but online rotation of a model with encrypted+unique fields is refused. Configure the index key before the first such row is written. For an existing table, quiesce writes and backfill the companion digests under the dedicated key before attempting online rotation; the rotation preflight rejects any legacy digest.

Setting it also separates key material between the two algorithms. On the fallback the same 32 bytes are the AES-GCM key and the HMAC-SHA256 key; with IndexKeyID set they resolve to different env vars — or different Transit keys — holding different secrets. There is no known attack that combines the two, so this is hygiene rather than an exposure, but it is hygiene that costs one environment variable and cannot be retrofitted: changing the index key changes every digest it has already produced.

Because the bill arrives late — writes succeed, and the refusal comes at the first rotation, once the table is full of digests nobody can re-derive — the framework says so at boot:

WARN encrypted unique fields are indexed under the field encryption key because
     the KeyProvider names no blind-index key; RotateEncryptionKey will refuse
     this model, and existing digests cannot be re-derived
     model=Patient fields=ssn hint="set IndexKeyID on the KeyProvider before writing data"

Config.Strict promotes that warning to a startup error, so a production deployment cannot reach its first write on the fallback by accident.

Reads strip the HMAC column from responses automatically; clients see only the decrypted plaintext on email and never the digest.

When the unique check fires, the adapter returns *maniflex.ErrConstraint and the DB step converts it to 409 CONFLICT — same path as any other unique violation.

If the index cannot be created

The UNIQUE index over the HMAC column is what actually enforces the constraint. If AutoMigrate cannot create it, the migration fails rather than starting a server whose mfx:"unique" is claiming a guarantee the database is not enforcing:

AutoMigrate: could not create unique index uidx_users_email_hmac on
users(email_hmac): UNIQUE constraint failed — the model declares these
column(s) unique, so the server will not start without the constraint. The
usual cause is rows that already violate it; de-duplicate them and start again

The usual cause is exactly that: rows already holding duplicate values in the field. De-duplicate them and start again. A plain (non-unique) index that fails to build is only warned about — a missing one costs a table scan, not a constraint.

Per-domain keys

The key:NAME sub-option routes a field to a specific key identifier:

type Record struct {
    maniflex.BaseModel
    PaymentToken string `json:"payment_token" mfx:"encrypted,key:billing"`
    MedicalNote  string `json:"medical_note"  mfx:"encrypted,key:medical"`
}

A KeyProvider that backs different keys with different secrets (or a Vault transit mount) lets you scope access by domain — the billing team holds the billing key; medical staff hold the medical key; the application process holds both. Rotating one does not affect the other.

When key: is omitted, the framework uses the keyID "default". Either configure a key under that name or always tag with an explicit key.

Decryption on the read path

The DB step runs the decryption pass after every read:

• For list and read operations, decryptFields replaces every enc:<base64> value with the decrypted plaintext.
• HMAC companion columns are always stripped from the response.
• Values that do not have the enc: prefix are left as-is — important for gradual adoption: enable encryption on a column whose existing rows are plaintext, and only new writes get encrypted.
• Rows pulled in by ?include= are decrypted too. A relation’s row reaches the serializer straight from the adapter rather than through the DB step’s own pass, so before v0.2.5 an encrypted field on an included model came back as base64 ciphertext while the same model read directly returned plaintext. If a relation cannot be decrypted the field is left as stored and a warning is logged, rather than failing the whole request over one included row.

If KeyProvider is nil but a model has encrypted fields, reads return the raw stored ciphertext (so the application still functions in some read-only sense), but writes are refused. Configuring a provider is the only way to write encrypted columns.

Key rotation

First update the model’s key:NAME tag (and deploy it) so concurrent writes use the new encryption key. Then maniflex.RotateEncryptionKeyWithOptions(ctx, server, modelName, oldKeyID, newKeyID, options) re-encrypts every old-key envelope:

report, err := maniflex.RotateEncryptionKeyWithOptions(
    ctx, server, "Patient", "v1", "v2",
    maniflex.EncryptionRotationOptions{PageSize: 100},
)
if err != nil {
    log.Printf("stopped after %q; row failures: %+v", report.LastID, report.Failures)
    log.Fatal(err)
}
log.Printf("re-encrypted %d rows", report.Rotated)

The compatibility wrapper RotateEncryptionKey still returns (int, error). The detailed API adds Failures, LastID, and Complete.

Rotation resolves the model’s per-model adapter, then preflights the requested range before writing. Malformed base64, invalid envelopes, decrypt failures, and mismatched legacy blind indexes are returned with their row ID, field, and stage. It never reports success after silently skipping corruption.

The operation is not atomic across all rows. On an interruption or adapter error, resume with AfterID: report.LastID; already-rotated envelopes are idempotently skipped. If report.Failures is non-empty, repair those rows and restart from the beginning so the full remaining old-key set is preflighted. Both encryption keys and the stable index key must remain available until report.Complete is true.

For large tables, run the rotation as a background job rather than at startup. Each row is a separate UPDATE, so the operation is bound by the database’s write throughput.

What an envelope looks like

The exact envelope format is the KeyProvider’s concern. The two shipped providers use slightly different layouts, but both put a self-describing header in front of the ciphertext so KeyIDOf can extract the keyID without decrypting.

EnvKeyProvider — AES-256-GCM with an inline nonce:

[ version:1 ][ keyIDLen:2 (BE) ][ keyID:N ][ nonce:12 ][ gcmCiphertext+tag:M ]

New envelopes use version 0x02, which binds the version byte and keyID into the GCM tag as additional authenticated data (AAD), so neither can be altered without failing decryption. Legacy 0x01 envelopes — sealed without that binding — still decrypt, and a key rotation re-writes them as 0x02.

VaultKeyProvider — Vault returns a vault:v1:... ciphertext that embeds its own versioning, so the envelope carries no nonce:

[ version:1 ][ keyIDLen:2 (BE) ][ keyID:N ][ vaultCiphertext:M ]

Vault envelopes use version 0x01 (Vault Transit authenticates internally). The keyID length is a 16-bit big-endian integer, and the framework stores the binary envelope as the string enc:<base64> in the column.

Encrypt produces the blob; Decrypt parses the header to recover the keyID, then routes to the right key. KeyIDOf reads the keyID without decrypting — useful for audit logging and for the rotation loop above.

A custom provider need not follow either format; the framework only cares that Encrypt and Decrypt are inverses and that KeyIDOf works on the output of Encrypt.

Compatibility with other features

Operational checklist

• Set Config.KeyProvider before any encrypted-field model is registered.
• Back keys with a secret store (env vars from a vault, HashiCorp Vault Transit, AWS KMS). Never commit a key to source control.
• For staged rollout, deploy the schema (email_hmac column) before the application change that starts encrypting — and the application change before the migration that backfills existing rows.
• Keep both keys active throughout a rotation; remove the old key only after RotateEncryptionKey has reported every row migrated.
• Treat KeyIDOf(envelope) as the source of truth for “which key encrypted this row” — useful for auditing the rotation.