maniflex
Annotated Go structs in. A full REST API out.
maniflex is a Go framework that turns a plain struct into a production REST API —
filtering, pagination, relations, soft-delete, file uploads, and an OpenAPI 3.1
spec — with no generated code and no per-endpoint boilerplate. Behaviour is
declared with mfx: struct tags and customised through a composable
six-step middleware pipeline.
type Post struct {
maniflex.BaseModel
maniflex.WithDeletedAt
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,enum:draft|published|archived"`
UserID string `json:"user_id" mfx:"required,filterable"`
}
Register that struct and you get GET/POST /posts, GET/PATCH/DELETE /posts/{id},
?filter=status:eq:published, ?sort=title:asc, ?page=2&limit=20,
?include=user, soft-delete semantics, and an entry in /openapi.json.
Why maniflex
- Reflection, not codegen. Register a struct at startup; routes, schema, and validation are derived from it at runtime. Nothing to regenerate when a field changes.
- One module, two dependencies. The core
maniflexmodule depends only on chi and uuid. Postgres, Redis, Kafka, NATS, bcrypt and friends live in satellite modules you pull in only if you import them. - A pipeline you can reach into. Every request flows through six ordered
steps — Auth → Deserialize → Validate → Service → DB → Response. Hook
middleware
Before,After, orReplaceat any step, scoped by model or operation. - Batteries included, swappable. Ready-made middleware for JWT auth, unique validation, password hashing, multi-tenancy, audit logging, CORS, and more — each one just a function you can replace.
- SQLite or Postgres. Both backends share one SQL adapter. Develop against pure-Go SQLite (no CGo, no external service), deploy on Postgres.
Core concepts
Five ideas carry the whole framework. Understand these and the rest of the docs slot into place.
Model
A Go struct that embeds maniflex.BaseModel and declares its fields with mfx:
struct tags. The struct is the single source of truth: it defines the database
table, the JSON request and response shapes, the validation rules, and which
fields are filterable or sortable. Optional embeds add behaviour — maniflex.WithDeletedAt
turns on soft-delete — and naming conventions like a UserID field declare
relations.
→ Models & BaseModel, Field Tags Reference,
Relations
Registry
The collection of every registered model, built by MustRegister at startup.
It is consumed in two places: the HTTP router reads it to mount routes, and the
DB adapter reads it to run migrations and resolve relations. This is why
MustRegister must run before sqlite.Open / postgres.Open — the adapter
is handed the populated registry.
→ Getting Started
Pipeline
Six ordered steps every request flows through: Auth → Deserialize → Validate → Service → DB → Response. The pipeline is the unit of customisation — instead of writing handlers, you attach middleware to the step where your logic belongs. → Pipeline Overview, ServerContext
Middleware
A func(ctx *maniflex.ServerContext, next func() error) error registered on a pipeline
step. Registration is scoped with maniflex.ForModel(...) and maniflex.ForOperation(...),
and positioned with maniflex.AtPosition(maniflex.After) (or maniflex.Before, the default, or maniflex.Replace). Set
ctx.Response and return without calling next() to short-circuit the request.
→ Writing Middleware, Middleware Catalogue
Adapter
The database backend implementing the storage interface. Two ship in-tree —
db/sqlite (pure-Go, no CGo) and db/postgres — and both share one SQL core.
Inject one with server.SetDB(db), which patches the pipeline’s DB step in
place.
→ Database Backends, Transactions
Where to go next
- Getting Started — install, define your first model, run the server.
- Models & Tags — the full
mfx:tag reference and relation conventions. - The Pipeline — how requests flow and where to hook in.
- Middleware Catalogue — ready-made middleware for every step.
- Querying the API — filtering, sorting, pagination, includes.
maniflex requires Go 1.25.12 or newer.
Getting Started
This guide takes you from an empty directory to a running REST API with a
filterable, paginated posts resource — in about five minutes and roughly
thirty lines of Go.
Prerequisites
- Go 1.25.12 or newer — check with
go version. - That’s it. The first example uses the pure-Go SQLite backend, so there is no database server to install and no CGo toolchain to configure.
1. Create the project
mkdir blog && cd blog
go mod init blog
go get github.com/xaleel/maniflex
maniflex itself pulls in only two dependencies — chi
and uuid. The SQLite adapter lives in its own
satellite module, so add it explicitly:
go get github.com/xaleel/maniflex/db/sqlite
2. Define a model
A model is a plain struct that embeds maniflex.BaseModel. The mfx: struct tags
declare how each field behaves — what’s required, what can be filtered, what
can be sorted.
Create main.go:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,enum:draft|published|archived"`
}
maniflex.BaseModel contributes the id, created_at, and updated_at fields, so
you never declare them yourself. See Field Tags Reference for every
tag and Models & BaseModel for the embeds.
3. Wire up the server
Registration order matters: models must be registered before the database is opened, because the SQLite adapter needs the registry to run migrations and resolve relations.
func main() {
// 1. Create the server — no DB yet.
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
// 2. Register models — this populates the registry.
// BaseModel's columns are readonly and nothing more, so opt created_at
// into the query surface to sort by it.
server.MustRegister(Post{}, maniflex.ModelConfig{
BaseModelTags: map[string]string{"created_at": "filterable,sortable"},
})
// 3. Open SQLite with the populated registry.
db, err := sqlite.Open("./blog.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
// 4. Inject the adapter into the pipeline.
server.SetDB(db)
// 5. Serve.
if err := server.Start(); err != nil {
log.Fatal(err)
}
}
Migration (on by default) creates and updates tables to match your structs on
startup — convenient for development. For an in-memory database that resets on
every run, pass ":memory:" to sqlite.Open.
4. Run it
go run .
The server is now listening on :8080, and Post{} has a full set of routes
mounted under the /api prefix:
| Method | Path | Action |
|---|---|---|
POST | /api/posts | create a post |
GET | /api/posts | list posts |
GET | /api/posts/{id} | read one post |
PATCH | /api/posts/{id} | update a post |
DELETE | /api/posts/{id} | delete a post |
5. Make some requests
Create a post:
curl -X POST localhost:8080/api/posts \
-H 'Content-Type: application/json' \
-d '{"title":"Hello","body":"First post","status":"published"}'
List, filter, sort, and paginate — all from the query string:
# Only published posts
curl 'localhost:8080/api/posts?filter=status:eq:published'
# Newest first, ten per page
curl 'localhost:8080/api/posts?sort=created_at:desc&page=1&limit=10'
Filtering and sorting only work on fields tagged filterable / sortable —
that’s why Title and Status carry those tags above. BaseModel’s id,
created_at and updated_at are readonly and nothing more, so they opt in
through ModelConfig.BaseModelTags at registration instead of a struct tag —
that’s the config passed to MustRegister above. The full filter grammar
is in Querying.
What you get for free
From that one struct, maniflex derived:
- Five REST endpoints with JSON request/response handling.
- Field validation (
required,enum) on every write. - Query-string filtering, sorting, and pagination.
- A generated table kept in sync by
AutoMigrate. - An OpenAPI 3.1 document ready to mount explicitly through
Config.Documentation.
No generated code, no per-endpoint handlers.
Where to go next
- Quickstart Tutorial — build a small app end to end.
- Models & BaseModel — relations, soft-delete, file fields.
- The Request Pipeline — the six steps every request flows through, and where to hook in your own middleware.
- Querying — the full filter, sort, and
includegrammar. - Database Backends — switching from SQLite to PostgreSQL.
App Anatomy
A maniflex app has very little machinery of its own — the framework derives routes, schema, and validation from your structs, so the code you write is mostly models and middleware. This page shows how to lay that code out, starting from a single file and growing into packages as the app gets bigger.
The smallest app
A maniflex app is just a package main that does four things in order:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
type Message struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
}
func main() {
server := maniflex.New(maniflex.Config{Port: 8080, PathPrefix: "/api"})
server.MustRegister(Message{})
if db, err := sqlite.Open("./app.db", server.Registry()); err == nil {
defer db.Close()
server.SetDB(db)
} else {
log.Fatal(err)
}
if err := server.Start(); err != nil {
log.Fatal(err)
}
}
Four steps — create server → register models → open and set DB → serve.
Everything below is about where the code for each step lives as the app grows.
The ordering is load-bearing: MustRegister must run before sqlite.Open,
because the adapter is handed the populated registry. See
Getting Started.
A typical project layout
Once an app has more than a handful of models, split it into packages by responsibility, not by model. A layout that scales well:
myapp/
├── go.mod
├── main.go # wiring only: create, register, set DB, serve
├── config.go # maniflex.Config assembly, env-var reading
├── models/ # one file per model — the structs and their tags
│ ├── user.go
│ ├── post.go
│ └── comment.go
├── middleware/ # custom pipeline middleware
│ ├── auth.go
│ ├── audit.go
│ └── register.go # attaches all middleware to the pipeline
└── internal/ # non-framework code: services, clients, helpers
└── mailer/
└── ...
Nothing here is enforced by the framework — maniflex never scans directories. It is a convention that keeps each file answering one question.
What goes in each file
main.go — wiring, nothing else
main.go should read top-to-bottom as the four-step sequence and contain no
business logic. Its whole job is to assemble the pieces and call Start():
func main() {
server := maniflex.New(config.Load())
server.MustRegister(
models.User{},
models.Post{},
models.Comment{},
)
db, err := sqlite.Open("./app.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
middleware.Register(server) // all pipeline hooks, in one place
if err := server.Start(); err != nil {
log.Fatal(err)
}
}
If you can’t see the four steps at a glance, something belongs in another file.
config.go — building maniflex.Config
Keep maniflex.Config construction — and any environment-variable reading — out of
main.go. A single Load() function makes the app’s knobs easy to find:
package config
func Load() maniflex.Config {
return maniflex.Config{
Port: envInt("PORT", 8080),
PathPrefix: "/api",
DisableAutoMigrate: env("APP_ENV", "dev") == "production",
}
}
See Configuration for every maniflex.Config field.
models/ — one file per model
Each file holds one struct, its mfx: tags, and the relation fields that point
at other models. This is the heart of the app — the struct is the table, the
JSON shape, and the validation rules all at once.
// models/post.go
package models
type Post struct {
maniflex.BaseModel
maniflex.WithDeletedAt // opt-in soft-delete
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,enum:draft|published|archived"`
UserID string `json:"user_id" mfx:"required,filterable"` // BelongsTo User
Comments []Comment `json:"comments,omitempty"` // HasMany
}
Put model-spanning relations in whichever file is the “owning” side and let Go’s
package scope resolve the rest — all models share the models package, so
Post can reference Comment freely. See Models & BaseModel,
Field Tags Reference, and Relations.
middleware/ — your pipeline hooks
A middleware is a func(ctx *maniflex.ServerContext, next func() error) error. Group
related hooks per file (auth.go, audit.go), and keep one register.go that
attaches them all — so there is exactly one place to see how the request
pipeline has been customised:
// middleware/register.go
package middleware
func Register(s *maniflex.Server) {
s.Pipeline.Auth.Register(bearerToken,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete))
s.Pipeline.Service.Register(hashPassword,
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate))
s.Pipeline.DB.Register(auditLog,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After))
}
The middleware functions themselves live in their topic files. See Writing Middleware and the Middleware Catalogue for hooks that already ship with the framework.
internal/ — everything that isn’t maniflex
Code that has nothing to do with the framework — a mail client, a payment SDK
wrapper, domain calculations — goes under internal/. Middleware in the
Service step calls into these packages; the packages themselves never import
maniflex. This keeps the framework-facing layer thin and your business logic
unit-testable on its own.
Structuring a large monolith
The layer-based layout above (models/, middleware/) stays readable up to a
few dozen models. Past that, a flat models/ directory with sixty files and a
main.go that names every one of them becomes the bottleneck. Large maniflex
codebases switch from splitting by layer to splitting by domain.
Domain packages
Give each business domain its own package that owns all of its code — models, middleware, and business logic together:
myapp/
├── main.go
├── domains/
│ ├── auth/
│ │ ├── models.go # User, Role, Session, ApiKey
│ │ ├── middleware.go # token auth, password hashing
│ │ └── register.go # exports Models and Register(s)
│ ├── catalog/
│ │ ├── models.go # Product, Category, Variant
│ │ ├── middleware.go
│ │ └── register.go
│ └── orders/
│ ├── models.go # Order, LineItem, Invoice, Refund
│ ├── middleware.go
│ └── register.go
└── internal/
A new feature now touches one directory instead of being smeared across
models/, middleware/, and internal/.
Registering models in groups
server.Register (and MustRegister) is variadic and flattens any slice
argument — so each domain can export its models as a single slice, and
main.go registers the slices side by side:
// domains/auth/register.go
package auth
// Models is every model this domain owns. One list, one place to update.
var Models = []any{
User{},
Role{},
Session{},
ApiKey{},
}
// main.go
server.MustRegister(
auth.Models, // each argument is a []any —
catalog.Models, // Register flattens them into individual models
orders.Models,
)
main.go no longer grows when a domain gains a model; only that domain’s
Models slice changes. To register everything as one list instead, concatenate
the slices: append(append(auth.Models, catalog.Models...), orders.Models...).
Registering middleware in groups
Apply the same idea to the pipeline. Each domain exposes its own
Register(s *maniflex.Server), and the top-level middleware registration just calls
each one — exactly the shape in the request from a growing app:
// domains/orders/register.go
package orders
// Register attaches every pipeline hook this domain needs.
func Register(s *maniflex.Server) {
s.Pipeline.Validate.Register(checkStock, maniflex.ForModel("Order"))
s.Pipeline.Service.Register(chargePayment,
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
s.Pipeline.DB.Register(emitOrderEvent,
maniflex.ForModel("Order"), maniflex.AtPosition(maniflex.After))
}
// main.go (or a thin top-level middleware/register.go)
func registerMiddleware(s *maniflex.Server) {
auth.Register(s)
catalog.Register(s)
orders.Register(s)
}
Each domain controls its own pipeline hooks; the top-level function is just a table of contents.
Co-locating per-model middleware
For middleware that belongs to exactly one model, skip the separate Register
call entirely — ModelConfig.Middleware lets you attach hooks at registration
time, scoped to that model automatically:
server.MustRegister(
Order{}, maniflex.ModelConfig{
Middleware: &maniflex.ModelMiddleware{
Validate: []maniflex.MiddlewareFunc{checkStock},
Service: []maniflex.MiddlewareFunc{chargePayment},
},
},
)
This keeps a model and its rules in one declaration — useful when the hook is meaningless without the model.
Conventions that keep a big monolith honest
- One registration point per concern. Exactly one place lists the model groups, and one lists the middleware groups. If you can’t find where a model is registered, the structure has drifted.
- Domains depend inward, never sideways. A domain may import
internal/andmaniflex; it should not import a sibling domain. Cross-domain relations are expressed by FK fields (a stringUserID), which need no import. internal/holds framework-free logic. Payment, mail, and pricing code lives here and never importsmaniflex, so it stays unit-testable in isolation.main.gostays a fixed size. Adding a domain adds one line to each registration list and nothing else. Ifmain.gogrows with the app, logic has leaked into it.
How a request moves through the files
Tracing one POST /api/posts shows why the layout is split this way:
- The router (built from the registry in
main.go) matches the route. - The request enters the pipeline —
Auth → Deserialize → Validate → Service → DB → Response. - At each step, the hooks from
middleware/register.gorun, scoped by model and operation. Validatechecks themfx:tags declared inmodels/post.go.Servicemiddleware may call intointernal/for business logic.- The adapter injected via
SetDBruns the SQL at theDBstep.
Each file owns one stage of that journey — which is exactly why a growing app stays readable.
Where to go next
- Example 1: Simple Blog — this layout filled in for a real three-model app.
- The Request Pipeline — the six steps in depth.
- Configuration — every
maniflex.Configfield.
Example 1: Simple Blog
This is the first worked example: a small blog API built end to end. It uses
only what the previous pages covered — models and mfx: tags,
the four-step setup, and SQLite. No relations, no middleware, no pipeline
customisation yet; those arrive in later chapters. The goal is to see a complete,
runnable app with nothing unexplained in it.
What we’re building
A blog with two independent resources:
- Post — an article with a title, body, and a publication status.
- Subscriber — an email address signed up for the newsletter.
The two are unrelated — each is its own table with its own endpoints — which keeps the example to concepts already introduced.
The whole app
The blog is small enough to live in a single main.go, the
smallest-app shape from App Anatomy:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
// Post is a blog article.
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,sortable,enum:draft|published|archived"`
}
// Subscriber is a newsletter sign-up.
type Subscriber struct {
maniflex.BaseModel
Email string `json:"email" mfx:"required,filterable"`
Name string `json:"name" mfx:"filterable,sortable"`
}
func main() {
// 1. Create the server.
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
Documentation: maniflex.DocumentationConfig{Public: true},
})
// 2. Register both models — populates the registry.
// Opt created_at into the query surface to sort by it.
server.MustRegister(
Post{}, maniflex.ModelConfig{
BaseModelTags: map[string]string{"created_at": "filterable,sortable"},
},
Subscriber{},
)
// 3. Open SQLite with the populated registry, then inject it.
db, err := sqlite.Open("./blog.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
// 4. Serve.
if err := server.Start(); err != nil {
log.Fatal(err)
}
}
That is the entire blog. Run it:
go run .
Reading the models
Every field choice maps to a tag covered in Getting Started:
| Field | Tags | Effect |
|---|---|---|
Title | required,filterable,sortable | must be present; usable in ?filter= and ?sort= |
Body | required | must be present; not queryable |
Status | required,...,enum:draft|published|archived | rejected unless one of the three values |
Email | required,filterable | must be present; filterable but not sortable |
Name | filterable,sortable | optional; queryable and sortable |
maniflex.BaseModel adds id, created_at, and updated_at to both structs, so
those are never declared by hand. With migration on by default, the posts and
subscribers tables are created to match on startup.
The endpoints you get
Registering the two structs mounts a full REST surface under /api:
| Method | Path | |
|---|---|---|
POST | /api/posts | create a post |
GET | /api/posts | list posts |
GET | /api/posts/{id} | read one post |
PATCH | /api/posts/{id} | update a post |
DELETE | /api/posts/{id} | delete a post |
Subscriber gets the identical five routes under /api/subscribers.
Using the API
Create a post
curl -X POST localhost:8080/api/posts \
-H 'Content-Type: application/json' \
-d '{"title":"Hello World","body":"My first post","status":"draft"}'
The response echoes the stored row, including the id and timestamps that
BaseModel filled in.
Validation in action
Leave out a required field, or send a status outside the enum, and the
write is rejected before it reaches the database:
# Missing body, and status is not in the enum
curl -X POST localhost:8080/api/posts \
-H 'Content-Type: application/json' \
-d '{"title":"Broken","status":"weekly"}'
# → 400, the response names the offending fields
Update and delete
# Publish the post — PATCH only sends the fields that change
curl -X PATCH localhost:8080/api/posts/<id> \
-H 'Content-Type: application/json' \
-d '{"status":"published"}'
curl -X DELETE localhost:8080/api/posts/<id>
List, filter, sort, paginate
All from the query string, on the fields tagged filterable / sortable:
# Only published posts
curl 'localhost:8080/api/posts?filter=status:eq:published'
# Newest first
curl 'localhost:8080/api/posts?sort=created_at:desc'
# Page two, five per page
curl 'localhost:8080/api/posts?page=2&limit=5'
# Combine them
curl 'localhost:8080/api/posts?filter=status:eq:published&sort=title:asc&limit=5'
Add a couple of subscribers and the same querying works there too:
curl -X POST localhost:8080/api/subscribers \
-H 'Content-Type: application/json' \
-d '{"email":"[email protected]","name":"Ada"}'
curl 'localhost:8080/api/subscribers?sort=name:asc'
The API documents itself
You never wrote a schema, yet the server already publishes one. Alongside the
model routes, maniflex auto-generates an OpenAPI 3.1 specification describing
every endpoint, field, and validation rule — derived from the same structs and
mfx: tags:
curl localhost:8080/api/openapi.json
The spec updates itself whenever a model changes; there is nothing to
regenerate. To browse it as interactive documentation, the framework ships an
HTML viewer in static/openapi.html that loads /api/openapi.json — open
http://localhost:8080/static/openapi.html while the server is running.
The OpenAPI step is fully customisable later; see OpenAPI Spec.
What this example showed
- A complete, runnable app from two plain structs and a four-step
main. mfx:tags driving validation (required,enum) and queryability (filterable,sortable).- Multiple models registered in one call, each with an independent REST surface.
- Filtering, sorting, and pagination with no query code written.
- A self-updating OpenAPI 3.1 spec at
/api/openapi.json.
Where to go next
Everything here treated the two models as separate islands. Real apps connect
them — a post belongs to an author, a comment belongs to a post. That, and
the mfx: tags beyond the basics, are the next chapters:
- Models & BaseModel — the embeds and what they contribute.
- Field Tags Reference — every
mfx:tag. - Relations — connecting models with foreign keys.
Architecture
This page explains how maniflex is put together. The reference pages describe each piece in isolation; here we look at how they fit. A reader who finishes this page should be able to point at any other doc and predict roughly what it covers.
The five pieces
┌──────────────────────┐
register → │ Registry │ ← models discovered here
└──────────────────────┘
│ │
│ ▼
│ ┌──────────────────────┐
│ │ Database adapter │
│ │ (sqlite | postgres) │
│ └──────────────────────┘
▼
┌──────────────────────┐
│ Router │ ← chi v5
└──────────────────────┘
│
▼
HTTP request → ┌──────────────────────────────────────────────────┐
│ Pipeline: │
│ Auth → Deserialize → Validate → │
│ Service → DB → Response │
└──────────────────────────────────────────────────┘
│
▼
HTTP response ← APIResponse envelope
The framework has five primary moving parts:
| Piece | What it is | Where it lives |
|---|---|---|
| Registry | An in-memory map of every registered model’s *ModelMeta — fields, tags, relations, indices, scheduled specs. | built by MustRegister, consumed by the router and the adapter |
| Router | A chi v5 Router mounted with one sub-router per registered model. | router.go |
| Pipeline | Six ordered steps that every model-route request flows through, plus a parallel three-step pipeline for /openapi.json. | pipeline.go |
| ServerContext | The single per-request struct threaded through every step. | context.go |
| DBAdapter | The backend interface implemented by db/sqlite, db/postgres, and any custom backend. | db.go |
Every other concept in the docs sits on top of these five.
Reflection, not codegen
The framework derives everything from the registered structs at startup:
ScanModelwalks the struct withreflectonce per model, builds a*ModelMeta, and inserts it into the registry.- The adapter reads the registry to emit
CREATE TABLE/ALTER TABLEstatements duringAutoMigrate. - The router reads the registry to mount the five REST routes per model
plus
/openapi.json. - The Validate step reads each request’s model meta to enforce
mfx:tag rules. - The DB step reads each request’s model meta to assemble the SQL.
Reflection runs once at registration, never per request. The per-request
path is allocation-light: a map[string]any for the body, a few string keys,
and the slice of registered middleware filtered by ForModel /
ForOperation. A model with N fields produces O(N) work at boot and O(N)
work per request, both linear in the size of the model.
This is the architectural difference from codegen frameworks: there is no
generated file to keep in sync. Changing a mfx: tag changes the runtime
behaviour the next time the process starts.
The registry is the contract
Every other piece reads from the registry; nothing writes to it after
Start(). That single rule explains several constraints:
MustRegistermust run beforesqlite.Open/postgres.Open. The adapter reads the registry during its constructor to learn about tables and relations.- Models cannot be added or removed at runtime. New models require a process restart.
- Models can be inspected from middleware via
ctx.Model, which is the*ModelMetathe router selected for this request. - Cross-model operations work because middleware reaches into the
registry through
ctx.GetModel(name)or by name in scoped registration.
The framework’s startup sequence is deliberate:
1. maniflex.New(cfg) → empty registry
2. server.MustRegister(...) → populated registry
3. sqlite.Open(..., reg) → adapter built from registry
4. server.SetDB(db) → adapter wired into the DB step
5. middleware.Register(...) → pipeline customised
6. server.Start() → router built from registry, listener opens
Start() runs AutoMigrate (if enabled) before opening the listener, so a
schema mismatch fails fast instead of corrupting writes.
The pipeline is the unit of customisation
Every HTTP request to a model route is wrapped in a ServerContext and run
through six steps in this order:
| Step | Default behaviour |
|---|---|
| Auth | passthrough |
| Deserialize | parse query string + body |
| Validate | enforce mfx: tag rules |
| Service | passthrough — business logic goes here |
| DB | dispatch to the adapter |
| Response | write the JSON envelope |
Each step has its own StepRegistry on server.Pipeline. A registration
attaches a MiddlewareFunc at Before (the default), After, or Replace
position, scoped by ForModel / ForOperation. At request time the
registry returns the matching chain for the (model, operation) pair, the
chain runs, and any step can short-circuit by setting ctx.Response and
returning without calling next().
The pipeline is the answer to “where do I put X?”:
- Identity checks — Auth.
- Coerce types, strip unknown fields — Validate (Before).
- Hash passwords, set derived fields — Service.
- Bracket the DB call — DB (Before / After).
- Webhooks, events, audit log — DB (After).
- Headers, redactions, metrics — Response.
The same answer applies whether the code lives in a catalogue middleware,
a custom function, or a per-model ModelConfig.Middleware.
The adapter is one interface
maniflex.DBAdapter has fewer than a dozen methods — FindByID, FindMany,
Create, Update, Delete, BeginTx, Raw, Ping, plus the schema
operations called by AutoMigrate. Two implementations ship:
db/sqlite— pure-Go SQLite for development.db/postgres—lib/pqfor production.
Both implementations share db/sqlcore, a SQL adapter that knows about
filters, sorts, includes, soft delete, and relations. A custom backend — an
HTTP data service, a different SQL database — implements the same interface
and is injected with server.SetDB(myAdapter). No other code changes.
The same holds for FileStorage and KeyProvider: small interfaces with
shipped implementations and obvious extension points.
Satellite modules
maniflex is a multi-module repository. The core module imports only chi and
uuid. Everything heavier — a database driver, a JWT library, a Kafka client,
bcrypt — lives in its own satellite under the same root, so a consumer
pulls in only the dependencies it actually imports.
The split keeps the core small and stable. It also keeps the trust boundary
clear: the framework’s surface area is the maniflex package; everything in
middleware/*, events/*, jobs/*, db/* is application code that
happens to ship alongside the framework.
See Satellite Modules for the full layout and import rules.
Two pipelines, one router
The router actually mounts two pipelines.
The first one — the six-step pipeline described above — handles
/<table> and /<table>/{id} for every registered model.
The second is a three-step pipeline for GET /openapi.json:
OpenAPI.Auth → OpenAPI.Generate → OpenAPI.Response
Generate derives the spec from the registry every time the endpoint is
hit, then Response serialises it. After-position middleware on Generate
can mutate the spec — change titles, add servers, install security schemes,
or rewrite arbitrary fields. See OpenAPI Spec and the
OpenAPI Middleware catalogue.
A third, trimmed pipeline runs for custom actions:
Auth → [per-action middleware] → handler → Response
The Deserialize, Validate, Service, and DB steps are skipped — actions own their body parsing and database work.
Where each feature lives in the lifecycle
| Feature | Step(s) | Notes |
|---|---|---|
mfx: tag rules (required, enum, min, …) | Validate | per-field |
| Required-on-create, immutable-on-update, readonly-strip | Validate | tied to Operation |
mfx:"file" multipart parsing | Deserialize | populates ctx.Files |
| File storage write | Service (built-in) | writes the storage key |
| Soft-delete filter on reads | DB | adapter rewrites the SQL |
mfx:"encrypted" envelope + HMAC | DB (Before for writes, after for reads) | needs KeyProvider |
Versioned history row | DB (Before for pre-image, After for write) | sibling _history table |
mfx:"scheduled" sweep | outside the request — separate runner | see Scheduled Fields |
?filter=…&sort=…&include=… parsing | Deserialize | into ctx.Query |
?include= population | DB | secondary queries after the main SELECT |
Auto-tenant filter (db.Tenancy) | DB (Before) | appends to ctx.Query.Filters |
| Audit log | DB (Before) | needs the pre-image; writes outside the request |
maniflex.WithTransaction | Service (Before) or DB (Replace) | wraps the DB step |
LockForUpdate | inside the DB step’s transaction | SELECT ... FOR UPDATE on Postgres |
For a single concrete trace of every step running on one request, see the Request Lifecycle walkthrough.
What maniflex is not
To set expectations on the architecture choice:
- Not codegen. No generated files, no separate build step.
- Not a router framework. The HTTP layer is chi; maniflex uses it.
- Not opinionated about JSON shape. The envelope is the default, but
response.Envelopelets you replace it. Errors always use the error envelope. - Not a service mesh. One process, one binary. Multi-process concerns (events, jobs, distributed locks) live in the satellite modules.
- Not magic. Every behaviour is a function in the
maniflexpackage or one of the catalogue middlewares. Read the source when in doubt; the pipeline is small.
Next
- Request Lifecycle — a single
POST /api/orderstraced end-to-end through every step. - Glossary — every framework term in one place.
- Pipeline Overview — the per-step reference.
Request Lifecycle
This page traces a single request through every piece of the framework. The
example is a POST /api/orders on an authenticated user, with a Service
middleware that hashes a derived field, a transaction wrapping the DB step,
and an audit-log middleware on DB-After. It exercises the full pipeline
without being contrived.
Setup
cfg.HTTPMiddlewares = append(cfg.HTTPMiddlewares,
response.CORSHeaders("https://app.example.com"))
server := maniflex.New(cfg)
server.MustRegister(models.Order{})
server.Pipeline.Auth.Register(auth.JWTAuth("secret"))
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
server.Pipeline.Service.Register(
service.SetField("customer_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.UserID
}),
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate),
)
server.Pipeline.DB.Register(
db.AuditLog(sink, db.WithChanges()),
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
The request
POST /api/orders HTTP/1.1
Authorization: Bearer eyJ...
Content-Type: application/json
Idempotency-Key: 2af9...
{"total": 42.50, "status": "pending"}
What happens, in order
1. The router selects the route
chi matches POST /api/orders to the sub-router mounted by mountModel
for the Order model. The matched handler calls handler.Create(meta),
which:
- Allocates a fresh
*ServerContext. - Sets
ctx.Request,ctx.Writer,ctx.Ctx. - Reads the
X-Request-Idchi added in the outer middleware and stores it onctx.RequestID. - Reads the
traceparentheader (if present) intoctx.TraceID. - Sets
ctx.Model = metaforOrder. - Sets
ctx.Operation = OpCreate. - Leaves
ctx.ResourceIDempty — create has no path id. - Calls
Pipeline.execute(ctx).
If Config.QueryTimeout is non-zero, ctx.Ctx is wrapped in a
context.WithTimeout here, so every downstream DB call inherits the
deadline.
2. The Auth step runs
Pipeline.Auth.build("Order", OpCreate) returns a chain consisting of
matching middleware in registration order, then the default Auth handler at
the end (which is a passthrough).
For our setup the chain is just [auth.JWTAuth, defaultAuth]. JWTAuth:
- Reads the
Authorizationheader. - Verifies the signature with the configured secret.
- Parses the claims and populates
ctx.Auth = &AuthInfo{UserID: …, Roles: …, TenantID: …}. - Calls
next()to continue.
A missing or invalid token would have produced ctx.Abort(401, "UNAUTHORIZED", …)
and returned without next(), ending the request right here.
3. The Deserialize step runs
The default handler parses two things:
- Query parameters →
ctx.Query(a*QueryParams). For a create, this is mostly empty — there are nofilterorsortto read. - Body. The
Content-Typeisapplication/json, so it reads up to 4 MB fromctx.Request.Body, setsctx.RawBodyto the raw bytes, and parses the JSON into the read-onlyctx.ParsedBody(a*RequestBody):
// ctx.ParsedBody now holds { "total": 42.5, "status": "pending" }
total, _ := ctx.Field("total") // 42.5
status, _ := ctx.Field("status") // "pending"
The same values are bound to the typed record ctx.Record; middleware mutate
either through ctx.SetField / ctx.DeleteField. The two are not the same
gesture: SetField(name, nil) writes the column as NULL, which is how you
discard a value the client sent, while DeleteField(name) leaves the column
out of the write entirely.
If the Content-Type had been multipart/form-data, the default handler
would route through parseMultipart instead, populating ctx.ParsedBody
with form fields and ctx.Files with the file parts.
After-position middleware on Deserialize (e.g. idempotency.Middleware,
which sees ctx.RawBody) runs next. With idempotency configured, the
middleware would compute a body hash and either replay a cached response or
fall through to step 4.
4. The Validate step runs
The default handler iterates ctx.Model.Fields and applies the mfx: tag
rules to ctx.ParsedBody:
idis stripped — the adapter assigns it.readonlyfields (created_at,updated_at) are stripped.immutablefields are stripped ifOpUpdate— not on create.requiredfields must be present.totalandstatusare required; the request supplies both, so no error.enummembership is checked onstatus—"pending"is in the allowed set, so no error.min/maxare checked on numeric fields when present.
If any rule had failed, the step would have called ctx.Abort(422, "VALIDATION_ERROR", …)
with details: [...] listing the bad fields.
Custom Validate middleware (none in this example) would run alongside the
default — Before middleware first, then the default, then After.
5. The Service step runs
Two middleware are scoped to this request:
maniflex.WithTransaction(nil) runs first:
- Sees
ctx.Tx == nil. - Calls
ctx.BeginTx(ctx.Ctx, nil), which delegates to the adapter. - Assigns the resulting
Txtoctx.Txand re-wrapsctx.Ctxwith the tx stored undertxContextKey{}. - Defers
tx.Rollback()— a no-op afterCommit. - Calls
next()to run the rest of the pipeline inside the transaction.
service.SetField("customer_id", ...) runs second:
- Resolves the callback against
ctx.Auth.UserID. - Calls
ctx.SetField("customer_id", "user-alice"), writing through to bothctx.ParsedBodyand the typedctx.Record. - Calls
next().
6. The DB step runs
The default DB step calls defaultSteps.db:
- Sees
ctx.Tx != niland constructs adbExec{adapter, tx: ctx.Tx}. - Builds the DB-column write set from the typed
ctx.Record(falling back totoDBMap(ctx.ParsedBody)for bodies the record can’t represent); here the column names match the JSON keys. - If the model had
mfx:"encrypted"fields, callsencryptFieldsto replace plaintexts withenc:<base64>envelopes and write{field}_hmaccompanions for unique ones. - Dispatches by
ctx.Operation:
result, err := exec.Create(ctx.Ctx, model, dbData)
exec.Create on a transactional dbExec calls tx.Create(...), which
runs INSERT INTO orders (...) RETURNING * (Postgres) or
INSERT ... ; SELECT ... (SQLite).
The adapter returns the inserted row as a map[string]any. The DB step
assigns it to ctx.DBResult.
If the adapter returned maniflex.ErrNotFound, the step would abort with
404 NOT_FOUND. *maniflex.ErrConstraint becomes 409 CONFLICT. A
context-cancelled error becomes 504 TIMEOUT when a server-side deadline
expired, or 499 when the client simply hung up. Any other adapter error
becomes 500 DB_ERROR.
6a. Audit-log Before middleware
We registered db.AuditLog at the default Before position (because
WithChanges() needs to read the pre-image). On OpCreate there is no
pre-image, so the middleware merely sets up to collect the post-image. It
calls next(), which runs the rest of the chain — the default DB handler
above.
After next() returns and ctx.Response is still nil (the create
succeeded), the middleware:
- Reads
ctx.DBResultfor the inserted row. - Builds an
AuditRecordwith model, operation, actor (ctx.Auth.UserID), tenant, request id, trace id, and a diff of every changed field. - Spawns a goroutine that calls
sink.Write(bgCtx, record). Audit writes are fire-and-forget — a sink error never fails the request.
7. WithTransaction commits
Control returns to WithTransaction (because we are inside its next()
call). It checks:
next()returned nil → no pipeline error.ctx.Response == nilor< 400→ no aborted step.- Calls
tx.Commit(). The deferredRollbackis now a no-op. - Runs any callbacks registered with
ctx.AfterCommit— the write is durable, so the side effects that were waiting on it may fire. - Clears
ctx.Txso any post-commit code uses the bare adapter.
If next() had returned an error, or if any step had set ctx.Response
to a status >= 400, Commit would have been skipped and the deferred
Rollback would have fired — and the AfterCommit callbacks would have
been dropped rather than run, since the write they announce did not happen.
8. The Response step runs
The default Response handler:
- Sees
ctx.Response == nil. - Sees
ctx.Operation == OpCreate. - Builds:
ctx.Response = &APIResponse{
StatusCode: http.StatusCreated,
Data: toJSONMap(ctx.DBResult.(map[string]any), model),
}
toJSONMap converts DB column names back to JSON field names and applies
hidden and writeonly filtering — any column tagged those is dropped
from the response shape.
After-position middleware on Response runs next. CORS is deliberately absent
from this step: response.CORSHeaders wraps the HTTP router, adding the
appropriate Access-Control-* headers before routing and Auth. A valid browser
preflight already returned 204 No Content before this pipeline began.
9. The envelope is written to the wire
APIResponse.Write(ctx.Writer):
- Sets
Content-Type: application/json. - Writes the status code header (
201 Created). - Encodes
{"data": {...}}to the response body. - Returns.
chi’s RequestID middleware (registered at the router root, outside the maniflex
pipeline) wraps the whole exchange — it has already set X-Request-Id on the
response by the time we get here. Maniflex’s trusted-proxy resolver is registered
alongside it only when Config.TrustProxyHeaders is set, so RemoteAddr reflects
the forwarded client IP just for servers that opted into trusting proxy headers.
The dispatch cleanup
After the response is written, the handler runs its cleanup phase:
- Closes any open multipart file readers in
ctx.Files. - Removes the multipart temporary directory.
- Lets the
*ServerContextgo out of scope; it is garbage-collected with the request.
The framework does not pool or reuse ServerContext values. The per-request
allocation is small; the simplicity is worth more than the saved
allocations.
What changes for other operations
Different operations exercise slightly different paths:
OpRead/OpList— Validate runs but is a no-op (it only fires for create/update). The DB step callsFindByIDorFindMany. Response for List wraps withmeta: {total, page, limit, pages}.OpUpdate— Like create, butctx.ResourceIDis set,immutablefields are stripped in Validate, and the DB step callsUpdate. The audit middleware fetches the pre-image beforenext()so theChangesdiff has both sides.OpDelete— No body to deserialize, no Validate work. The DB step callsDelete(which becomes a soft-deleteUPDATEfor models withWithDeletedAt). The default Response is204 No Content.OpAction— The trimmed pipeline runsAuth → action middleware → handler → Response. The handler is responsible for its own body parsing, validation, and database calls./openapi.json— A separate three-step pipeline (OpenAPI.Auth → Generate → Response) builds the spec from the registry every time, then writes the JSON document.
What happens on errors
Three error paths at every step:
| Trigger | Effect |
|---|---|
Middleware returns a non-nil error from next() | Bubbles up; later steps are skipped. The chain returns the error to the handler, which logs and writes a 500 INTERNAL envelope. |
Middleware calls ctx.Abort(...) and returns nil without next() | ctx.Response is set; subsequent steps are skipped (because next() was never called); the Response step’s default reads ctx.Response and writes it. |
| Panic anywhere in the chain | PanicRecoverer catches it, logs through Config.PanicLogger, writes a 500 PANIC envelope. |
A transaction in flight is rolled back by WithTransaction’s deferred
Rollback in all three cases — the same code path that handles success.
What this tour did not show
- Multi-tenant scoping with
db.Tenancy— would have appended toctx.Query.Filtersin step 5/6. mfx:"file"uploads — would have parsed multipart in step 3 and written bytes toFileStoragebetween steps 5 and 6.mfx:"scheduled"— fires outside the request, in a separate Scheduled Runner goroutine.mfx:"versioned"— would have written a sibling history row in step 6’s After phase, in the same transaction.- The OpenAPI pipeline — same shape, three steps, separate registrations.
Each of those is covered in its own page; the lifecycle in step-by-step form is the same.
Models & BaseModel
A model is a Go struct registered with the server. From it, maniflex derives a database table, the JSON request and response shapes, the set of REST routes, and the validation applied to every write. This page covers what a struct must contain to be a valid model, how it maps to a table, and the options available at registration. Field-level tags are documented in Field Tags Reference; relationships in Relations.
Definition
A model is an ordinary struct that embeds maniflex.BaseModel:
type Article struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
}
Registration validates the struct and adds it to the registry:
server.MustRegister(Article{})
Register returns an error; MustRegister panics on failure and is intended
for use in main or package initialisation. A struct is rejected at
registration if it is not a struct type or does not embed BaseModel.
Register every model before the first call to Handler or Start. That call
atomically seals the server’s routes, specifications, and middleware pipeline.
A later Register returns maniflex.ErrRegistrationClosed without changing
the registry; MustRegister panics on the same error. Actions, rollups,
computed fields, realtime documentation, global search, storage, and pipeline
middleware share the same setup window.
Two embedding rules are enforced there as well:
- Embed by value, never by pointer.
*maniflex.BaseModel— or any other pointer embed — is left nil by the record scanner, so every field access through it panics on the first request. It is refused at registration instead. - Two fields cannot map to the same column. A field shadowing one of
BaseModel’s columns produced two entries with the same DB name, and the framework then disagreed with itself about which was meant: lookups took the first, writes took the last. Rename one, or give it an explicitdb:"...".
BaseModel
Every model must embed maniflex.BaseModel. It contributes three columns common to
all tables:
type BaseModel struct {
ID string `json:"id" db:"id" mfx:"readonly"`
CreatedAt time.Time `json:"created_at" db:"created_at" mfx:"readonly"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at" mfx:"readonly"`
}
ID— the primary key, a UUIDv4 string assigned by the framework on create. What that guarantees, and what it rules out, is the Record Identity contract.CreatedAt— set once, when the row is created.UpdatedAt— refreshed on every update.
All three are managed by the framework and all three are readonly: values
supplied for them in a request body are ignored rather than stored — including
id, so a client cannot choose the primary key. Because they are part of
BaseModel, they are never declared on individual models.
A struct that does not embed BaseModel — or otherwise lacks an id column —
fails registration.
Querying the BaseModel columns
readonly is the only default. None of the three columns is filterable or
sortable unless the model says so — filterable and sortable widen a model’s
public query surface, and that is a decision each model makes rather than
inherits.
BaseModel lives in the framework, so its struct tags are the one place you
cannot edit. ModelConfig.BaseModelTags is the knob instead:
server.MustRegister(Post{}, maniflex.ModelConfig{
BaseModelTags: map[string]string{
"id": "filterable,sortable",
"created_at": "filterable,sortable,index",
},
})
Keys are DB column names; values use the same comma-separated syntax as an mfx
struct tag. Each column accepts only the options meaningful for it:
| Column | Accepts |
|---|---|
id | filterable, sortable |
created_at | filterable, sortable, index, hidden |
updated_at | filterable, sortable, index, hidden |
Anything else is a registration error. index is not offered on id because
id is the primary key and already indexed, so the option would only add a
redundant duplicate.
Options are unioned onto readonly, never replacing it. There is no way to
make created_at client-writable through this — a form that let readonly fall
off by omission would turn a typo into a silently writable timestamp.
Without the opt-in, ?sort=created_at:desc and ?filter=created_at:gte:...
return 400, and the error names BaseModelTags as the fix.
Keyset pagination.
cursor_field:created_atrequirescreated_atto be sortable, and does not grant it implicitly — so a model using it needs the matchingBaseModelTagsentry too. Registration fails with an explanatory error if you write one half without the other.
Field mapping
Each exported field of a model maps to a database column. Three struct tags control the mapping:
| Tag | Purpose |
|---|---|
json | the field’s name in request and response bodies |
db | the column name; defaults to the snake_case field name if omitted |
maniflex | field behaviour — validation, filterability, and so on |
A minimal field needs only a json tag; db is derived and mfx is optional.
The mfx tag is the largest of the three and has its own reference in
Field Tags Reference. Fields that name a related model — for example
a UserID foreign key — are interpreted as relations; see
Relations.
Nullability is the Go type
A pointer field gets a NULL column; everything else gets NOT NULL. No
tag controls this — the type is the whole declaration:
type Note struct {
maniflex.BaseModel
Title string `json:"title"` // NOT NULL — cannot be null
Body *string `json:"body"` // NULL — may be null
}
That decides what a request may send. {"body": null} stores SQL NULL;
{"title": null} is refused with 422 naming the field, because the column has
no null to store:
{
"error": {
"code": "VALIDATION_ERROR",
"details": [{
"field": "title",
"message": "field \"title\" cannot be null; its type has no null value — send a value, omit the field, or make it a pointer to allow null"
}]
}
}
Three things are distinct and stay distinct: null (refused unless the field is
a pointer), "" (a value, stored as written), and omitting the key (leaves
the stored value alone on a PATCH).
If you need to tell “empty” from “not set”, make the field a pointer. A non-pointer field genuinely cannot represent the difference — it reads back as the zero value either way.
Note the corollary for anything reading the database directly: because every
non-pointer field is NOT NULL, NOT NULL says nothing about whether a value
is required, and the zero value rather than NULL is what “absent” means. If
you are writing a migration script, a reporting query, or another service
against a maniflex schema, read Field Schema & Nullability first — that
distinction has cost real data.
Table names
By default the table name is the struct name converted to snake_case and pluralised:
| Struct | Table |
|---|---|
Article | articles |
BlogPost | blog_posts |
Category | categories |
Analysis | analyses |
Matrix | matrices |
A short list of Latin and Greek stems inflect classically — analysis, basis,
axis, diagnosis, thesis (and hypothesis), datum, medium, matrix,
vertex, criterion, phenomenon — including inside a compound name, so
BloodAnalysis becomes blood_analyses. Everything else takes the English
ending: album → albums, complex → complexes, and index → indexes,
since that is the plural the database world uses.
To use a different name, pass a ModelConfig with TableName set when
registering:
server.MustRegister(
Article{}, maniflex.ModelConfig{TableName: "articles"},
)
TableName is also how you pin an existing table whose generated name has
changed — see the v0.3.0 note in the changelog if you have a model named after
one of the stems above.
Registration options
ModelConfig carries per-model options. All fields are optional; an omitted
ModelConfig applies the defaults described above.
| Field | Purpose |
|---|---|
TableName | override the derived table name |
SoftDelete | opt the model into soft deletion — see Soft Delete |
Middleware | pipeline middleware scoped to this model, installed at registration — see Writing Middleware |
Versioned | record field-change history in a sibling {model}_history table |
VersionedDiffOnly | with Versioned, store only changed fields rather than full snapshots |
Indices | additional database indexes created during AutoMigrate |
ExportEnabled | mount GET /:model/export (CSV / XLSX) — see CSV / XLSX Export |
MaxExportRows | row cap for the export endpoint; default 100,000 |
QueryLimits | override individual global query/aggregate complexity limits for this model; see Configuration |
AggregateEnabled | mount GET /:model/aggregate?aggregate=<url-encoded JSON> (grouped count/sum/avg/min/max) — see Aggregations |
OptimisticLock | enable If-Match / ETag concurrency control on PATCH and DELETE |
Adapter | route this model to a separate database adapter |
Singleton | expose the model as a single-row resource (GET / PATCH, no id) — global, or one row per tenant when scoped; see Singleton models |
Headless | register the model fully but mount no REST routes, freeing its path for a custom action — see Serving a model’s own path from an action |
Optimistic locking (OptimisticLock)
When OptimisticLock: true, every PATCH and DELETE request that includes an
If-Match header is checked against the current record’s ETag before the write
executes. A mismatch returns 412 Precondition Failed (PRECONDITION_FAILED).
Requests without If-Match are unaffected — the flag opts in to enforcement,
not mandatory locking.
The ETag format is identical to the one emitted by response.Cache (MD5 of the
JSON response body), so clients can use the header from a preceding GET directly:
server.MustRegister(Invoice{}, maniflex.ModelConfig{OptimisticLock: true})
server.Pipeline.Response.Register(
response.Cache(response.CacheConfig{MaxAge: 300}),
maniflex.ForModel("Invoice"),
maniflex.ForOperation(maniflex.OpRead),
maniflex.AtPosition(maniflex.After),
)
GET /invoices/42 → 200 ETag: "d41d8cd9..."
PATCH /invoices/42 If-Match: "d41d8cd9..." → 200
PATCH /invoices/42 If-Match: "stale" → 412
PATCH /invoices/42 If-Match: * → 200
PATCH /invoices/99 If-Match: * → 404 (no such record)
If-Match: * is the RFC 9110 wildcard: it holds for any existing record, so it
means “overwrite whatever is there, but do not create it” rather than pinning a
particular version. It still takes the row lock, so it is safe to use on a
contended record — it just does not care which version it lands on.
The check and the write it guards run as a single transaction, with the record
held under a row lock (SELECT … FOR UPDATE on Postgres) from the ETag
comparison until the write commits. Two clients holding the same ETag therefore
cannot both succeed: the loser waits on the lock, then re-reads a record whose
ETag has moved on and gets its 412. When the request already runs inside a
transaction (maniflex.WithTransaction) the guard joins it and the lock is held
until that transaction commits; otherwise the DB step opens and commits one of
its own.
Singleton models (Singleton)
Some resources are inherently single-row: an application config record, a set of
feature flags, the banner an admin edits and every client reads at launch. With
Singleton: true the model drops its collection and item routes and exposes just
two endpoints on the bare table path — no id in the URL:
GET /:model → read the one row
PATCH /:model → update the one row
There is no POST, DELETE, or list endpoint; requesting them returns
405 Method Not Allowed, and there is no /:model/:id subtree.
The single backing row is provisioned lazily under the well-known
maniflex.SingletonID on first access, from each column’s default. So the first
GET returns defaults before anything has been written, and PATCH always
targets an existing row — it behaves like an upsert:
type AppConfig struct {
maniflex.BaseModel
MaintenanceMode bool `json:"maintenance_mode" mfx:"default:false"`
MinAppVersion string `json:"min_app_version" mfx:"default:1.0.0"`
Banner string `json:"banner"`
}
server.MustRegister(
AppConfig{}, maniflex.ModelConfig{Singleton: true, TableName: "config"},
)
GET /config → 200 {"data":{"id":"singleton","maintenance_mode":false,"min_app_version":"1.0.0","banner":""}}
PATCH /config {"maintenance_mode": true} → 200 {"data":{"id":"singleton","maintenance_mode":true, ...}}
GET /config → 200 (reflects the update)
POST /config → 405
Because the row is auto-provisioned from column defaults, a singleton model may
not declare mfx:"required" fields — there would be no value to satisfy them on
first access. Such a model is rejected at registration. Give fields sensible
mfx:"default:…" values (or make them pointers) instead.
One row per tenant
The example above is one row for the whole application. The other common shape is one row per tenant — a storefront, a profile, a per-org settings record — which is near-universal in B2B SaaS.
You get it by scoping the model the same way you scope any other: register a
db.Tenancy or db.ForceFilter for it on the DB step. The singleton then
resolves and provisions the caller’s row rather than a global one.
type StoreSite struct {
maniflex.BaseModel
OwnerID string `json:"owner_id" db:"owner_id" mfx:"filterable,unique,default:"`
Banner string `json:"banner" mfx:"default:untitled"`
}
server.MustRegister(StoreSite{}, maniflex.ModelConfig{Singleton: true})
server.Pipeline.DB.Register(
db.Tenancy("owner_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.Claims["owner_id"].(string)
}),
maniflex.ForModel("StoreSite"),
)
GET /store_sites (as owner A) → 200 A's storefront, created on first access
GET /store_sites (as owner B) → 200 B's storefront — a different row
PATCH /store_sites (as owner A) → 200 updates A's, never B's
There is no separate SingletonScope setting, because a bare column name cannot
say where the value comes from — whether it is ctx.Auth.UserID, a tenant
claim, or something else. The scoping middleware already answers that, so the
singleton reads its scope from the request’s forced filters and there is exactly
one place to configure it.
The route shape is unchanged: still the bare table path, still no id, still no
POST/DELETE. Only which row it addresses changes.
A few consequences worth knowing:
- A scoped row keeps an ordinary generated primary key.
maniflex.SingletonIDnames the global row only; one fixed id could not name one row per scope. - Give the scope column a unique index (
mfx:"unique", above). Two concurrent first accesses would otherwise both provision a row; with the index, the loser collides and re-reads the winner’s. - A request with no scope gets the global row. If your resolver returns
nilfor an unauthenticated caller, that caller reads theSingletonIDrow. Scope-or-refuse is whatdb.Tenancydoes (it answers403when it cannot determine the tenant);db.ForceFilterapplies no filter instead. - The scope has to be one the framework can write, not merely read. It is
stamped onto the provisioned row, so it must be a plain equality — which is
what
db.Tenancyanddb.ForceFilterbuild. A scope that names no single value (aninfilter, or adb.ForceFilterViawhose value lives on another table) cannot provision a row and is refused rather than creating one its own author could not then read.
This replaces the previous workaround — Headless plus a hand-written action —
which cost 40–60 lines and, because actions skip the Validate
step, silently gave up every
mfx tag rule (required, enum, min/max, immutable) and the generated
OpenAPI schema along with them.
Upgrading: a singleton that already has a global row and then gains a scope does not migrate that row — it has no scope column value, so it matches nobody and each tenant is provisioned a fresh one. Backfill or drop it deliberately.
ModelConfig registration order
A ModelConfig is positioned immediately after the model it configures:
server.MustRegister(
User{},
Article{}, maniflex.ModelConfig{Versioned: true},
Comment{},
)
Here User and Comment use defaults; only Article is versioned.
Two argument shapes are registration errors, because there is no valid reading of either and both used to silently discard the config you wrote:
- A
ModelConfigat position 0 (no preceding model to attach to). - Two
ModelConfigs in a row (the second has no fresh model to bind to).
See Strict mode for the full set of configuration problems caught at startup.
Optional embeds
Beyond BaseModel, the framework provides embeds that add columns and switch on
behaviour when present:
| Embed | Adds | Effect |
|---|---|---|
maniflex.WithDeletedAt | deleted_at (nullable timestamp) | timestamp-based soft delete |
maniflex.WithIsDeleted | is_deleted (boolean) | flag-based soft delete |
Embedding one of these is equivalent to setting SoftDelete in ModelConfig.
The two approaches and their query semantics are covered in
Soft Delete.
type Article struct {
maniflex.BaseModel
maniflex.WithDeletedAt // DELETE marks deleted_at instead of removing the row
Title string `json:"title" mfx:"required"`
}
Registration order
Models must be registered before the database adapter is opened. The adapter is constructed from the registry — it reads the registered models to run migrations and resolve relations — so the registry must be complete first:
server.MustRegister(User{}, Article{}, Comment{}) // 1. populate the registry
db, err := sqlite.Open("./app.db", server.Registry()) // 2. build the adapter from it
server.SetDB(db) // 3. inject the adapter
Registering a model after SetDB has no effect on an already-open adapter.
Next
- Field Tags Reference — every
mfx:tag and its meaning. - Relations — foreign keys and slice fields.
- Soft Delete —
WithDeletedAt,WithIsDeleted, and query behaviour.
Record Identity
Every record in a Maniflex application is identified by one column, id, holding
a string. This page states what that means as a v1 contract: what the framework
guarantees, what it assumes, and what it does not support.
The behaviour described here is pinned by tests/e2e/identity_test.go.
The contract
Identity is a single string column named id. It is contributed by the
embedded maniflex.BaseModel; a model that lacks it fails registration. There
is no composite primary key and no alternative identity column.
The framework generates the value. On insert, when id is empty, the adapter
assigns a UUIDv4 in canonical lowercase form — 9f8e7d6c-…, 36 characters,
random. Nothing about the value is meaningful: it is not sequential, not
time-ordered, and carries no tenant, type, or shard information.
Clients never choose it. id is mfx:"readonly", and the Validate step
strips the column from every write body unconditionally — including a value a
middleware stamped with ctx.SetField, which is a deliberate exception to the
rule that server-set values survive. An "id" in a POST or PATCH body is
ignored, not rejected: the request succeeds and the framework’s value is used.
The value never changes. No generated route reassigns an id. Updates, restores, and version history all keep the row’s original identity.
On the wire it is an opaque string. Generated OpenAPI declares both the id
property and the {id} path parameter as {"type":"string","format":"uuid"},
and the property as readOnly. Nothing parses or validates the id in the
request path: an id of any shape that does not match a row produces 404, never
400. Treat ids as opaque on the client side — compare them for equality, do
not order or parse them.
In the database it is TEXT PRIMARY KEY on every supported driver,
PostgreSQL included. The framework does not use a native uuid column type, a
sequence, or an identity column.
Relations carry the same string. A foreign key column stores the target
row’s id verbatim, and ?include= resolves relations by string equality.
Foreign key columns are therefore string (or *string when the relation is
optional).
Pagination assumes ids are unordered but unique. Keyset pagination orders by
(cursor field, id), using the id only to break ties so a page boundary is
total. Because v4 ids are random, that tiebreak order is arbitrary — stable
across a walk, but not meaningful. Do not sort by id expecting insertion
order; sort by created_at (opting in through
BaseModelTags).
Adapter interfaces are string-typed. FindByID, Update, Delete, and
their transactional counterparts all take id string. A custom adapter
implements that signature; a multi-column key cannot be expressed through it.
Assigning your own id
Below the request pipeline, the adapter generates an id only when none was supplied. Code writing through the model accessor may therefore choose its own:
row, err := ctx.GetModel("Invoice").Create(map[string]any{
"id": "INV-2026-0001",
"amount": 5000,
})
The framework itself relies on this: maniflex.SingletonID is the fixed id of
a singleton model’s single row.
This is supported, with the responsibilities that come with it:
- Uniqueness is yours. A collision surfaces as a primary-key constraint error from the database, not as a framework-level validation message.
- URL safety is yours. The id appears in
/{model}/{id}paths. Keep it to characters that survive a path segment unescaped. - The OpenAPI document still says
format: uuid. Consumers generating clients from the spec may validate against it. - It does not reach the HTTP surface. There is no way for a client, or for a middleware on the request path, to supply an id — only server-side code writing through the accessor or an adapter directly.
Use it for rows your application names rather than discovers: a singleton, a fixed configuration row, a record keyed by an external system’s identifier.
Not supported in v1
These are absent by decision, not by oversight. Support for them would be additive, so applications that need them are not blocked from adopting v1 — but nothing in v1 should be read as promising them.
| Client-supplied ids | No route or configuration accepts an id from a request. |
| Composite / multi-column keys | Unrepresentable through the adapter interface, which is single-string throughout. |
| Natural keys as the identity | A natural key belongs in its own mfx:"unique" column; routes still address the row by id. |
| Integer or auto-increment ids | BaseModel.ID is a string. Declaring an integer id column instead is untested and behaves inconsistently across drivers. |
| Alternative id formats | UUIDv7, ULID, and prefixed ids are not configurable. There is no id-generator hook. |
Native uuid columns | Identity is stored as TEXT on every driver. |
| Lookup by natural key on the id route | GET /model/{value} matches the id column only. Use ?filter=slug:eq:value on the list route. |
Why this shape
A single opaque string is the narrowest assumption that every other generated feature can be written against: one route shape, one foreign-key type, one cursor tiebreaker, one adapter signature. Random v4 ids are also safe to expose — they leak neither row counts nor creation order, which sequential ids do.
The cost is real and worth stating: random ids cluster poorly in a B-tree index, so very large tables pay more on insert than they would with a time-ordered id. If that matters for a specific table, keep the generated id as the primary key and add your own time-ordered column with an index for the access pattern that needs it.
Field Schema & Nullability
This page is the frozen contract for the DDL maniflex emits: what a Go field becomes as a column, and what that column’s nullability does and does not tell you. It exists for the people reading the database from outside the application — a migration script, a reporting replica, BI, another service — because the encoding is not the one most of them assume.
The short version, and the one thing to take away:
NOT NULLdoes not mean “required”, andNULLis not how maniflex says “no value”. A non-pointer field is alwaysNOT NULL, and its zero value —'',0,false— is what “absent” means.
What a field becomes
For a model whose fields cover every case:
type SchemaSpec struct {
maniflex.BaseModel
ReqText string `db:"req_text" mfx:"required"`
ReqNum int `db:"req_num" mfx:"required"`
OptText string `db:"opt_text"`
OptNum int `db:"opt_num"`
OptRate float64 `db:"opt_rate"`
OptFlag bool `db:"opt_flag"`
NullText *string `db:"null_text"`
NullNum *int `db:"null_num"`
Tagged string `db:"tagged" mfx:"default:pending"`
}
AutoMigrate emits exactly this:
CREATE TABLE "schema_specs" (
"id" TEXT PRIMARY KEY,
"created_at" TEXT NOT NULL DEFAULT '0001-01-01T00:00:00Z',
"updated_at" TEXT NOT NULL DEFAULT '0001-01-01T00:00:00Z',
"req_text" TEXT NOT NULL,
"req_num" INTEGER NOT NULL,
"opt_text" TEXT NOT NULL DEFAULT '',
"opt_num" INTEGER NOT NULL DEFAULT 0,
"opt_rate" REAL NOT NULL DEFAULT 0,
"opt_flag" INTEGER NOT NULL DEFAULT 0,
"null_text" TEXT NULL,
"null_num" INTEGER NULL,
"tagged" TEXT NOT NULL DEFAULT 'pending'
)
| Field shape | Column | Meaning of “absent” |
|---|---|---|
non-pointer, mfx:"required" | NOT NULL, no DEFAULT | cannot be absent |
| non-pointer, optional | NOT NULL DEFAULT <zero> | the zero value |
| pointer | NULL | NULL |
any, mfx:"default:x" | DEFAULT 'x' | the declared default |
tests/e2e/schema_contract_test.go pins every row of that table against the
DDL actually emitted, so this page cannot drift from the code.
Reading required-ness from the schema
Both a required and an optional non-pointer column are NOT NULL. The only
schema-level difference is that a required column carries no DEFAULT:
"req_text" TEXT NOT NULL -- required
"opt_text" TEXT NOT NULL DEFAULT '' -- optional
So an external consumer that needs the distinction must test NOT NULL and
no default. Testing NOT NULL alone reads every optional column as required.
This is worth stating plainly because getting it wrong is expensive and silent.
A migration tool that read NOT NULL as “this reference must exist”, and
dropped rows whose parent was missing, discarded every storefront order in a
production database — the order_id column it was checking is legitimately
empty until an order is confirmed.
If you can read the API instead of the schema, prefer it: openapi.json lists
required fields explicitly in each model’s create schema, and it says what it
means rather than implying it.
Getting a genuinely nullable column
Declare the field a pointer. That is the supported way to say “this may have no value” to the database:
OrderID *string `db:"order_id" mfx:"relation"`
A pointer field is emitted NULL, gets no synthesised zero default, reads back
as null in JSON, and — since the include fix in this release — populates
correctly through ?include= when it is a BelongsTo foreign key. Rows whose
pointer key is genuinely NULL come back without the relation key at all.
Use a pointer when the difference between “not set” and “set to the zero value”
matters — an unset price versus a price of zero, an unconfirmed order versus one
attached to order "". Use a plain value when it does not.
Why the columns are not simply nullable
The obvious alternative — emit NULL for every non-required scalar — is not
something this version can do safely. AutoMigrate never rewrites an existing
column; it creates tables and adds columns, and warns about drift rather than
altering. Changing the emitted nullability would therefore apply only to newly
created tables, leaving the same model with two different schemas depending on
when its table was first created, and no mechanism to reconcile them. That needs
a migration story before it needs a DDL change.
The zero-value default is not decorative either: it is what lets
ALTER TABLE ADD COLUMN succeed against a table that already has rows, on both
SQLite and Postgres. A model can grow a field without manual DDL because of it.
Referential integrity is an application invariant
One more thing the schema will not tell you: a REFERENCES constraint is
emitted only for a BelongsTo relation carrying an on_delete action the
database can enforce. Specifically, a column gets no constraint when it is:
- a plain indexed id column with no
mfx:"relation"— the usual shape for anowner_id; - a relation left at the default
on_delete(a baremfx:"relation"), since there is no action to enforce. The exception is aJunctionModel, whose keys cascade unless the model says otherwise; - an edge whose deletion maniflex handles itself, which is where soft delete lands — those are enforced in the delete path instead, so the set of constraints emitted and the set of edges the cascade skips are drawn by the same line.
The database will therefore accept an orphan. Referential integrity in a
maniflex application is maintained by the application, not the storage layer, so
a PRAGMA foreign_key_check or a Postgres constraint scan proves less about a
maniflex database than it looks like it does.
Out of contract for v1
- Altering an existing column’s type or nullability during migration.
- Expressing required-ness in the schema in any way other than the absent DEFAULT described above.
CHECKconstraints derived frommfxvalidation tags (min:,max:,enum:); those are enforced in the Validate step, not the database.
Field Tags Reference
A model’s behaviour is declared with three struct tags on each field. This page
documents all three, and every directive accepted by the mfx tag.
The three tags
| Tag | Controls | Default if omitted |
|---|---|---|
json | field name in request and response bodies | snake_case of the Go field name |
db | database column name | the resolved json name |
mfx | field behaviour — validation, querying, and more | no directives |
Title string `json:"title" db:"title" mfx:"required,filterable,sortable"`
The mfx tag holds a comma-separated list of directives. Whitespace around each
directive is trimmed. Directives are either flags (a bare word) or
key-value directives (key:value).
An unrecognised directive is a registration error, and the message names the directive it thinks you meant:
maniflex: model "User" field "Role" has unknown mfx option
"read_only" (did you mean "readonly"?) — an unrecognised option is
not applied, so a protective directive that is misspelt leaves the
field unprotected
Directives used to be matched exactly and anything else discarded in silence.
For a descriptive directive that was merely puzzling — a misspelt sortable just
meant no sorting — but for a protective one it was a hole: mfx:"read_only"
left Readonly false, so the field stayed writable by any client, with nothing
in the logs or the OpenAPI spec to show it. Directives are case-sensitive and
lowercase; mfx:"Readonly" is rejected rather than quietly accepted.
Excluding a field
A field is dropped from the model entirely — no column, not in any payload — if
any of its three tags is set to -:
Internal string `mfx:"-"` // excluded
Cache string `json:"-"` // excluded
Scratch string `db:"-"` // excluded
Validation directives
These constrain the values a field accepts on write.
| Directive | Effect |
|---|---|
required | the field must be present in a create request |
enum:a|b|c | the value must be one of the pipe-separated options |
min:N | numeric minimum (N is a number) |
max:N | numeric maximum |
minlen:N | minimum length — characters for a string, items for a list |
maxlen:N | maximum length |
default:V | value applied when the field is absent; cast to the field’s type |
Status string `json:"status" mfx:"required,enum:draft|published|archived"`
Priority int `json:"priority" mfx:"min:1,max:5,default:3"`
Password string `json:"password" mfx:"required,writeonly,minlen:8"`
Message string `json:"message" mfx:"maxlen:5000"`
Magnitude vs. length
min:/max: bound a number’s value. minlen:/maxlen: bound a length
— of a string, or of a list. Putting one where the other belongs is a
registration error naming the tag you wanted:
maniflex: model "User" field "Password" has mfx:"min:"/"max:" but its Go type
is string — those bound a number's magnitude — for a length bound use
mfx:"minlen:"/"maxlen:"
The error exists because the mistake used to be invisible. mfx:"max:5000" on a
string reads exactly like “at most 5000 characters”, and instead rejected every
non-empty value at runtime with field "message" must be a number — nothing at
startup, and an error message that named neither the tag nor the type. This
framework’s own tutorial shipped Password string mfx:"min:8" and rejected
every password.
The range a bound may span
A min:/max: value is parsed as a float64, which represents every integer
up to 2^53 (9007199254740992) exactly and only some beyond it. On an
integer-typed field a bound at or past that point is a registration error rather
than a bound enforced as some nearby number:
maniflex: model "Ledger" field "Balance" has an mfx:"max:" bound that float64
rounded to 9223372036854776000, at or beyond 2^53 (9007199254740992) — a bound
is parsed as a float64, which cannot represent every integer past that point, so
the bound enforced would not be the one written.
The rounding is not conservative in either direction: max:9007199254740995
becomes ...996, a ceiling one looser than written, and
max:9223372036854775807 — “cap at MaxInt64” — becomes a number above
MaxInt64, so the guard can never fire and the column is silently unbounded.
Float-typed fields are unaffected: a bound there is inexact by the nature of the
column, and max:1e20 asks for a magnitude rather than an exact value. For an
integer range wider than 2^53, check it in a Validate middleware instead.
String length is counted in characters, not bytes. A cap sized against English would otherwise reject the same message written in Arabic or carrying emoji — a limit that behaves differently depending on the writer’s language, and never for the author.
For a maniflex.FileKeys field use max_count: rather than maxlen:; it is
the file-specific bound and has a default. Declaring maxlen: there is a
registration error pointing at it.
Malformed values are refused at registration
A value these directives cannot use is a startup error, not a silent no-op:
| Written | Result |
|---|---|
mfx:"min:abc", mfx:"max:" | registration error — not a number |
mfx:"maxlen:abc", mfx:"minlen:-1" | registration error — not a whole number |
mfx:"min:8" on a string | registration error — use minlen: |
mfx:"maxlen:10" on an int | registration error — use max: |
mfx:"minlen:10,maxlen:5" | registration error — unsatisfiable |
mfx:"enum:", mfx:"enum:a||b" | registration error — empty option |
mfx:"readonly,required" | registration error — unsatisfiable |
The first two used to be dropped on the floor, so a tag that reads as a
constraint enforced nothing. readonly,required is the same unsatisfiable shape
as hidden,required: readonly strips the field from the request before the
required check runs, so the check never sees it and the field was quietly
optional. (hidden implies readonly, so that combination reports its own more
specific message.)
Separately, a value that a min/max bound cannot be measured against — a
string sent to a numeric field — now fails validation with “must be a number”
rather than skipping the check.
How default: is applied
default:V becomes a SQL DEFAULT clause on the column. Nothing applies it in
Go — it fires because the INSERT omits the column, so it takes effect whenever
a write does not name the field. That holds for both doors into a model:
| Create path | Column omitted when |
|---|---|
POST /:model | the request body has no such key |
maniflex.Create[T] | the struct field is at its Go zero value |
The Go zero is the only signal a struct can give, so a defaulted column cannot
be given an explicit zero through Create[T] — Priority: 0 against
default:3 stores 3. Make the field a pointer when that distinction matters:
Priority *int `json:"priority" mfx:"default:3"` // nil → 3, new(int) → 0
Only defaulted columns behave this way. A zero on a field with no default:
tag is written as that zero, on both paths.
Before v0.2.5,
Create[T]wrote every column at its Go value and adefault:never fired — the same model created over HTTP and in Go disagreed. In the same release,default:on a pointer field started reaching the schema at all; it was previously emitted only forNOT NULLcolumns and silently dropped.
Write-access directives
These govern whether a field can be set by a client, and when.
| Directive | Effect |
|---|---|
readonly | stripped from all write operations; values sent by a client are ignored |
immutable | accepted on create, rejected on update |
BaseModel’s id, created_at and updated_at are all readonly, and that is
their only default — none of them is filterable or sortable until the model opts
in with ModelConfig.BaseModelTags.
Use immutable for
values that are set once and must not change afterwards, such as an owner ID.
Both mean “not from a client”. A value the server stamps via ctx.SetField —
db.Tenancy writing a tenant column, an auth middleware writing an owner — is
kept, even on a readonly or immutable field. Only values parsed from the
request body are stripped.
They cover a multipart upload too. A mfx:"file" field carrying readonly,
hidden, or immutable on update refuses an uploaded part with
422 VALIDATION_ERROR, and is left out of the multipart/form-data schema. For
a file the client should upload but never see the storage key of, use
writeonly — see Who may write a file field.
Email string `json:"email" mfx:"required,immutable"`
ApiKey string `json:"api_key" mfx:"readonly"`
Response-visibility directives
Both directives drop the field from API responses. They differ in whether the client may write the field.
| Directive | Read in responses | Write on create / update |
|---|---|---|
writeonly | no | yes |
hidden | no | no |
writeonlyis for values the client must supply but should never see again — typically passwords, and the storage key of a private file field. The field is included in the create and update request schemas; only the response is scrubbed.hiddenis for values clients have no business touching at all — server-managed internals, audit fields, derived data. The field is dropped from create and update schemas as well, so it cannot be set from the API. It is still stored, and code running inside the pipeline (a middleware on the Service step, for example) can populate it.
// Client sets it on create, never sees it back.
Password string `json:"password" mfx:"required,writeonly"`
// Server-managed; client can neither read nor write it.
InternalScore float64 `json:"internal_score" mfx:"hidden"`
hidden therefore implies readonly — writing it out as mfx:"hidden,readonly"
is allowed but redundant. If you want a field the client writes but never reads,
that is writeonly, and spelling both (mfx:"hidden,writeonly") leaves it
writable: the explicit directive wins over the implication.
mfx:"hidden,required" is rejected at registration. The two contradict —
hidden stops the client sending the field, required insists it does — so no
request could satisfy it. A value the client must supply but never reads back is
mfx:"writeonly,required".
A field that is both readonly and not hidden is the opposite case: visible
in responses, never accepted from the client.
json:"-" is treated as hidden and read-only: the field stays a real,
persisted column that the server owns, but it never appears in API responses and
is never accepted from the client. This matches the Go convention that json:"-"
affects serialization, not schema. To exclude a field from the database entirely
(no column), use db:"-" or mfx:"-" instead.
Record-locking directives
These freeze a whole record — not just a field — once its state matches a condition. Useful for terminal states in business workflows (posted invoices, closed pay periods, confirmed POs).
| Directive | Effect |
|---|---|
lock_when:field=value | when the existing record’s field equals value, updates and deletes return 422 RECORD_LOCKED |
Multiple lock_when directives accumulate; any matching condition locks
the record. The directive can be written on any field — the referenced
field is what matters.
Both halves of the directive are checked at registration, so you never ship a
rule that silently never matches: the JSON name has to resolve to a real field,
and the value has to be one that field could hold. lock_when:count=five on an
integer column is a startup error rather than a lock that never fires, as is
lock_when against a time.Time or a struct — the directive tests equality and
covers string, bool and numeric columns.
The comparison is made on the field’s own type rather than on the two sides’
printed forms, so it does not depend on which shape a driver returns: SQLite
reports a bool column as an integer, Postgres as a bool, and a NUMERIC arrives
through lib/pq as text. 1 is accepted where the field is a bool.
type Invoice struct {
maniflex.BaseModel
Number string
Status string `mfx:"enum:draft|posted|void,lock_when:status=posted,lock_when:status=void"`
Amount int
}
The transition into a locked state is itself allowed — when the request arrives, the loaded record is still in its previous state. After that update commits, the record becomes frozen.
lock_when is checked before the write reaches the adapter. On update it runs in
the Validate step, ahead of that step’s other rules — but only when the request’s
scope is already in place. A forced filter registered on the DB step
(db.Tenancy, db.ForceFilter) has not been applied at that point, and a guard
reading the row by id alone would answer 422 RECORD_LOCKED for a record the
caller’s own reads 404 on, disclosing both that it exists and that it has
reached the locked state. So the check moves to the DB step in that case, after
the scope is enforced: the same refusal, one step later. Declaring
maniflex.ProvidesScope()
hoists the scope ahead of Validate and keeps the early abort.
On delete it always runs in the DB step, likewise after the scope. Creates are exempt: there is no prior state to check.
The guard fails closed. It reads the record through the request’s transaction
when one is active — so it sees state the same request has written but not yet
committed — and if that read fails for any reason other than “no such record”,
the request is rejected (500 DB_ERROR) rather than allowed through unchecked.
Pessimistic lock directive
| Directive | Effect |
|---|---|
lock_scope:ModelName | before a create, acquire a SELECT … FOR UPDATE lock on the row referenced by this field’s value |
Eliminates manual ctx.LockForUpdate calls in the most common case: a
create that must read-then-write a shared resource without a concurrent
write sneaking in between.
type Dispense struct {
maniflex.BaseModel
StockID string `json:"stock_id" db:"stock_id" mfx:"required,lock_scope:StockBalance"`
Quantity int `json:"quantity" db:"quantity" mfx:"required,min:1"`
}
Requirements:
- The model must run inside a transaction. Register
maniflex.WithTransaction(nil)on the Service step; otherwise the DB step aborts with500 LOCK_SCOPE_NO_TX. - The referenced model name must be registered. A typo is caught at startup
(in
Handler()), so it never reaches production silently. - If the referenced row does not exist, the create returns
404 NOT_FOUND. - The referenced row must be in the request’s scope. When a forced filter
names a column the referenced model also carries, the row is looked up through
that filter first, so a create naming another tenant’s row gets the same
404as one naming a row that is not there — without it, the id was a probe and the lock landed on a row the caller cannot see. A referenced model that carries no such column cannot be scoped this way and is still located by id alone.
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForModel("Dispense"),
maniflex.ForOperation(maniflex.OpCreate),
)
Comparison with ctx.LockForUpdate:
lock_scope tag | ctx.LockForUpdate | |
|---|---|---|
| Declaration | struct tag | custom Service middleware |
| Fields locked | one per tag directive | any ID at runtime |
| Requires transaction | yes (enforced at runtime) | yes (enforced at call time) |
| Use when | one fixed FK to lock | dynamic or multiple targets |
See Transactions for the underlying ctx.LockForUpdate
and BeginTx APIs.
Query directives
These opt a field into the query string. A field is not filterable or sortable unless explicitly tagged.
| Directive | Effect |
|---|---|
filterable | the field may be used in ?filter= |
sortable | the field may be used in ?sort= |
searchable | the field is indexed for native full-text search (?q=); text columns only |
cursor_field:<name> | opt the model into keyset (cursor) pagination; <name> is the column to walk by — it must be sortable, non-nullable, and a scalar or time.Time field |
See Querying for the filter, sort, and cursor-pagination grammar.
Schema directives
| Directive | Effect |
|---|---|
unique | a hint to the adapter to add a UNIQUE constraint on the column |
index | create a (non-unique) index on the column during AutoMigrate |
Slug string `json:"slug" mfx:"required,unique"`
Email string `json:"email" mfx:"index"`
index creates an index named idx_<table>_<column>. It is skipped when the
column is already covered by another index — a unique constraint on the same
field (databases index unique columns implicitly), a ModelConfig.Indices entry,
or a scheduled-column auto-index — so adding it is always safe. Indexing a
foreign-key column (e.g. mfx:"index" on UserID) is a common, valid use.
unique is enforced on both the create-table and the add-column (ALTER TABLE)
paths — AutoMigrate creates a UNIQUE INDEX named uidx_<table>_<column> in both
cases. Adding a unique column to a table that already holds rows with duplicate
values in that column fails migration (the index build errors, naming the
table and column) rather than silently dropping the constraint; resolve the
duplicates before deploying.
JSON and other custom columns
A bare map[string]any, map[string]string, or []string field has no SQL
column mapping and fails AutoMigrate with a clear error. Wrap it in a named
type that implements maniflex.SQLTyper (plus driver.Valuer + sql.Scanner)
so it controls its own column type — e.g. a JSONMap that maps to JSONB on
Postgres and TEXT on SQLite. maniflex.LocaleString is a built-in example. To
keep such a field out of the database entirely, tag it mfx:"-".
A NOT NULL column of such a type is given a zero-value DEFAULT so that adding
it to a table which already has rows succeeds. For a type whose SQLType is one
the migrator has no literal rule for — BLOB, BYTEA, UUID, NUMERIC(12,2) —
that default is a CAST to the column’s own type, and the value cast is the
type’s zero driver.Valuer output where it has one, else '', {} or [] by
kind. Give the type a Value() a zero receiver can answer if the generic choice
is wrong for it.
To filter inside such a column, see json_array / json_object above.
Relation directives
A field may declare a relationship to another model. Relations are opt-in —
an <Name>ID field is a plain column unless you tag it.
| Directive | Effect |
|---|---|
relation | marks an FK field as a BelongsTo; the target is inferred from the field name (AuthorID → Author) |
relation:Name | explicit relation; Name is the companion struct field carrying the target type |
relation:Name;onDelete:action | sets the referential action — cascade, setNull, or restrict |
through:Model | on a slice field, declares a many-to-many relation through the named junction model |
norelation | deprecated no-op — relations are no longer inferred from the ID suffix, so nothing to opt out of |
onDelete sub-options are joined to the relation: directive with a semicolon,
not a comma. Relationships are covered in full in Relations.
A field whose name ends in ID (e.g. UserID) is a plain scalar column unless
tagged mfx:"relation". So a value column that merely ends in ID — an external
reference, an opaque token — needs no special tag:
ExternalID string `json:"external_id"` // just a string, not a relation
UserID string `json:"user_id" mfx:"relation"` // → User (opt in)
File upload directives
file marks a field as a file-upload field. The column stores the storage key;
multipart form-data is then accepted for create and update on the model.
The field’s Go type must be string (one key) or maniflex.FileKeys (many).
Any other type is a registration error — every rule below is keyed on the column
being a storage key, so on another type they would all be silently skipped.
| Directive | Effect |
|---|---|
file | mark the field as a file upload |
max_size:N | maximum file size; accepts KB, MB, GB suffixes, or plain bytes. On FileKeys, per file |
max_count:N | FileKeys only — maximum number of keys (default 100) |
accept:p1|p2 | allowed MIME-type patterns, e.g. image/*|application/pdf |
auto_delete:false | keep the stored file when the record is hard-deleted or the field is replaced (default: delete it) |
upload:presigned | mount POST /{model}/{field}/upload-url so the client uploads straight to storage |
file_acl:private | (default) response carries the raw storage key |
file_acl:signed | response replaces the key with a pre-signed URL (TTL: Config.FilesConfig.SignedURLTTL, default 1h) |
file_acl:public | response replaces the key with a permanent / long-lived URL |
Avatar string `json:"avatar" mfx:"file,max_size:2MB,accept:image/*"`
Logo string `json:"logo" mfx:"file,file_acl:public,accept:image/*"`
Images maniflex.FileKeys `json:"images" mfx:"file,accept:image/*,max_count:10"`
See File Fields & Uploads for the upload workflow, and
Many files per field for FileKeys.
Encryption directives
| Directive | Effect |
|---|---|
encrypted | the field is encrypted at rest (AES-256-GCM) and decrypted on read |
key:name | the key name passed to the key provider; defaults to default |
Encrypted fields cannot be filtered or sorted, because the stored value is
ciphertext. If unique is also set, a companion {field}_hmac column enforces
uniqueness without exposing the plaintext.
SSN string `json:"ssn" mfx:"encrypted,key:patient-pii"`
Scheduled directives
The scheduled directive declares a time-driven transition on a timestamp
field — for example, soft-deleting a row once a timestamp passes. The directive
only marks the field; the transitions are applied by a background runner
documented in Events & Background Jobs.
It is an advanced feature with several sub-options joined by semicolons:
ExpiresAt time.Time `json:"expires_at" mfx:"scheduled;soft-delete"`
PublishAt time.Time `json:"publish_at" mfx:"scheduled;field=status;from=draft;to=published"`
| Sub-option | Effect |
|---|---|
soft-delete | soft-delete the row when the timestamp is reached |
hard-delete | permanently delete the row when the timestamp is reached |
field=F | the field to change |
from=V | apply only when field currently equals V |
to=V | the value to set field to |
Locale directives
These apply to fields declared as maniflex.LocaleString — a multilingual
string stored as a JSON object keyed by locale code (e.g. {"en":"Finance","ar":"مالية"}).
Mark a field as locale-aware with mfx:"locale". All other locale directives
require locale to be present.
| Directive | Effect |
|---|---|
locale | marks the field as a LocaleString; enables locale-aware response serialisation |
json_array | column holds a JSON array; enables the has / not_has filter operators on it |
json_object | column holds a JSON object; enables has:key=value / not_has:key=value |
split | (default) response emits "name" = resolved string and "name_i18n" = full map |
resolve | response always emits "name" as a plain string; no companion field |
dynamic | response emits a string when ?locale= is set, the full map otherwise |
default_locale:code | field-level fallback locale (e.g. default_locale:ar) when the client did not request a specific locale |
type Department struct {
maniflex.BaseModel
Name maniflex.LocaleString `json:"name" mfx:"locale,filterable,sortable"`
Bio maniflex.LocaleString `json:"bio" mfx:"locale,resolve,default_locale:ar"`
}
The resolved locale for a request follows a precedence chain:
?locale= param → Accept-Language header → default_locale tag → model
DefaultLocale → app LocaleOptions.Default → "en".
See Localization for the full LocaleResolver setup and filtering/sorting behaviour.
Quick reference
| Directive | Category |
|---|---|
required | validation |
enum:… min: max: default: | validation |
readonly immutable | write access |
hidden writeonly | response visibility |
filterable sortable searchable cursor_field:… | querying |
unique index | schema |
relation relation:… through:… | relations |
file max_size: max_count: accept: auto_delete:false file_acl: upload:presigned | file upload |
encrypted key:… | encryption |
scheduled;… | scheduled transitions |
locale split resolve dynamic default_locale:… | localization |
json_array json_object | JSON columns |
- | exclude the field |
Relations
A relation connects two models through a foreign-key column or a junction
table. Relations are declared on the struct — by a field name, a tag, or a
slice — and are populated on demand via the ?include= query parameter.
maniflex recognises three kinds:
| Kind | Direction | Declared by |
|---|---|---|
| BelongsTo | this row holds the FK | an FK field tagged mfx:"relation" (or mfx:"relation:Name") |
| HasMany | the other table holds the FK | a slice field of the related type |
| ManyToMany | a junction table connects both sides | a slice field with mfx:"through:Junction" |
BelongsTo
Relations are opt-in: tag the foreign-key field mfx:"relation". The target
model is inferred from the field name with the trailing ID stripped
(UserID → User).
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required"`
UserID string `json:"user_id" mfx:"required,filterable,relation"` // → User
}
UserID is a foreign key to User, keyed user (snake-case of the trimmed
field name) — the value used in ?include=, in nested filters
(?filter=user.role:eq:admin), and in nested sorts (?sort=user.name:asc).
The FK field is also a regular column. Tag it filterable if clients need to
query by it, exactly like any scalar.
Note: an
<Name>IDfield is a plain scalar column unless you tag itmfx:"relation". A value column that merely ends inID—ExternalID,CloudEventID, a third-party token — stays a plain column with no relation and no?include=key. (The legacymfx:"norelation"opt-out is now a deprecated no-op, kept only so existing models compile.)
When the field name doesn’t match the target model, name it explicitly with
mfx:"relation:Target". A bare mfx:"relation" on a field that doesn’t end in
ID fails at startup: there’s no suffix to strip, so the target would be
inferred from the whole field name — a guess that is almost never a real model.
A relation whose target model is simply never registered is allowed by default (it may be a plain foreign id that wants no relation tag) and rejected under strict mode.
Including the related row
curl 'localhost:8080/api/posts?include=user'
Each post in the response gains a user object populated from the related
table. Multiple includes are comma-separated:
curl 'localhost:8080/api/posts/<id>?include=user,comments'
BelongsTo (explicit)
When the FK field name does not match the target model — for example, a
ManagerID pointing to User — declare the relation with mfx:"relation:Name"
and add a companion field of the target type:
type Team struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required"`
ManagerID string `json:"manager_id" mfx:"required,filterable,relation:Manager"`
Manager User `json:"manager,omitempty"`
}
ManagerIDis the column that stores the FK.Manageris the companion field: it carries the target type (User) so the framework can resolve the relation. It is not a column itself — its only role is type information for relation scanning.
The relation key here is manager (snake-case of Manager), so the include
becomes ?include=manager.
A relation:Name directive without a matching companion field fails
registration.
HasMany
The inverse side: a slice of the related struct, declared on the model that does not hold the FK.
type User struct {
maniflex.BaseModel
Name string `json:"name"`
Posts []Post `json:"posts,omitempty"` // populated when ?include=posts
}
There is no column on users for this — Posts is purely a relation
declaration. The FK is expected on the related table, named after the parent
model: Post is expected to carry user_id. (That column comes from UserID
on Post, as in the BelongsTo example above.)
The relation key for a slice is the snake-case of the field’s JSON name, here
posts.
curl 'localhost:8080/api/users/<id>?include=posts'
ManyToMany
A many-to-many relation uses a third junction model that carries the two FKs.
Both sides declare a slice with mfx:"through:JunctionModel":
type Product struct {
maniflex.BaseModel
Name string `json:"name"`
Tags []Tag `json:"tags,omitempty" mfx:"through:ProductTag"`
}
type Tag struct {
maniflex.BaseModel
Label string `json:"label"`
Products []Product `json:"products,omitempty" mfx:"through:ProductTag"`
}
// The junction model — register it just like any other.
type ProductTag struct {
maniflex.BaseModel
ProductID string `json:"product_id" mfx:"required,filterable,relation"`
TagID string `json:"tag_id" mfx:"required,filterable,relation"`
}
All three models must be registered. The junction’s FK columns are declared as
BelongsTo relations (mfx:"relation" on ProductID and TagID), so the
framework can resolve which side of the join goes where.
?include=tags on a product follows the junction and returns the related tags
directly.
Junction payload (_through)
A junction often carries data of its own — a role, a position, a date the link
was made. When it does, each included row gains a _through object holding
those columns:
{
"id": "tag-1",
"label": "blue",
"_through": { "position": 2, "linked_at": "2026-07-19T09:00:00Z" }
}
The two foreign keys and the junction’s id are excluded — they say nothing the
response does not already carry.
The junction is a model like any other, and _through honours its tags. A
column marked mfx:"hidden" or mfx:"writeonly" is absent, exactly as it would
be in a response from the junction’s own endpoint:
type ProductTag struct {
maniflex.BaseModel
ProductID string `json:"product_id" mfx:"required,filterable,relation"`
TagID string `json:"tag_id" mfx:"required,filterable,relation"`
Position int `json:"position"` // in _through
InvitedBy string `json:"invited_by" mfx:"hidden"` // not in _through
}
Two further exclusions are worth knowing:
mfx:"encrypted"columns are omitted, along with their_hmaccompanions. The junction payload has no decryption pass, so the alternative would be emitting ciphertext — no use to a client, and it discloses the encryption envelope. Put a value you need to read on the related model, not on the junction.- Columns the junction model does not declare are omitted. A column present in the table but absent from the model is schema drift, and dumping it is what used to make this a leak.
Before v0.2.5
_throughwas copied verbatim from the junction row, so hidden and write-only columns surfaced on every include.
Which models are join tables
A junction can be declared three ways. In order of precedence:
| How | When to use | |
|---|---|---|
| Explicit relation | mfx:"through:Junction" on a slice field | you want the relation named on the model |
| Explicit marker | embed maniflex.JunctionModel | the junction carries columns of its own |
| Auto-detected | two BelongsTo relations, no other columns | a plain link table |
Auto-detection accepts only the unambiguous shape — two foreign keys, an id,
timestamps, and nothing else:
type ProductTag struct { // auto-detected
maniflex.BaseModel
ProductID string `json:"product_id" mfx:"relation"`
TagID string `json:"tag_id" mfx:"relation"`
}
Add a column of its own and detection stops, because that shape is equally what an ordinary entity looks like:
type Order struct { // NOT a junction — nothing is inferred
maniflex.BaseModel
CustomerID string `json:"customer_id" mfx:"relation"`
AddressID string `json:"address_id" mfx:"relation"`
Total int `json:"total"`
}
Before v0.2.6
Orderwas treated as a join table, so Customer and Address silently gained a many-to-many through it. If you relied on detection for a junction that carries payload, embedJunctionModel— see below.
A junction that carries payload says so:
type Enrollment struct {
maniflex.BaseModel
maniflex.JunctionModel
StudentID string `json:"student_id" mfx:"relation"`
CourseID string `json:"course_id" mfx:"relation"`
Term string `json:"term"` // in _through
}
JunctionModel is embedded alongside BaseModel, not instead of it — a
junction is an ordinary model with an id. The model must have exactly two
BelongsTo relations to distinct models; anything else is a registration error.
ModelConfig.DisableAutoJunction opts a model out of detection entirely, for a
link-shaped model that should not gain the relation.
Unique links
Junction pairs are not unique by default. Declare it when they should be:
type ProductTag struct {
maniflex.BaseModel
maniflex.JunctionModel `mfx:"unique"`
ProductID string `json:"product_id" mfx:"relation"`
TagID string `json:"tag_id" mfx:"relation"`
}
This emits a UNIQUE index over the two key columns, so a repeated link is
refused with 409. It also lets an include collapse duplicate pairs: with the
declaration a repeat is corruption, without it a repeat is data.
Off by default because a junction carrying its own attributes may legitimately
repeat a pair — Enrollment{student_id, course_id, term} holds one row per
term, and each carries its own _through payload. A pure link table almost
always wants the tag.
Adding it to a table that already holds duplicates fails the migration until
they are cleaned up. That is why it is separate from the marker: embedding
JunctionModel changes nothing about the schema, so declaring what a model is
never risks a migration.
Junction deletes
A junction’s foreign keys default to ON DELETE CASCADE, so deleting either
endpoint takes its link rows with it — a link to a row that no longer exists
says nothing. An explicit mfx:"on_delete:..." on the column wins, and edges
where either side soft-deletes are handled in maniflex’s own delete path rather
than by a database constraint (see below).
Cascading deletes
A BelongsTo relation may declare what happens to this row when the parent it
points at is deleted, using the onDelete sub-option:
AuthorID string `json:"author_id" mfx:"relation:Author;onDelete:cascade"`
| Action | Effect on this row when the referenced row is deleted |
|---|---|
cascade | this row is deleted too |
setNull | the FK column is set to NULL (the field must be a pointer, so it is nullable — otherwise a registration error) |
restrict | the parent delete is refused with 409 DELETE_RESTRICTED while this row exists |
| omitted | nothing — the delete leaves this row untouched (its FK dangles) |
The action is validated at startup: it must target a registered model, and
setNull needs a nullable FK. cascade recurses — deleting a row deletes its
children, then their children — and reference cycles are handled.
How it is enforced (and soft delete)
The action is enforced one of two ways, chosen automatically per relation. The rule is that maniflex enforces the edge itself whenever it has something to do beyond deleting the row:
- Nothing to do beyond the row → a real database
FOREIGN KEY … ON DELETE …constraint carries it, and the database enforces it natively (and enforces referential integrity on insert: a row naming a non-existent parent is refused with409). - Either side soft-deletes → the database cannot help (a soft delete is an
UPDATE, soON DELETEnever fires, and a DB cascade can only hard-delete), so maniflex enforces it in the delete request’s own transaction instead. A soft-delete child of a cascaded parent is soft-deleted identically — itsdeleted_atis set, the row is not removed. - The child is
Versionedor is a rollup child → maniflex enforces it, and noON DELETEclause is emitted for that edge. A databaseON DELETEremoves the rows and tells nobody: the history would lose itsdeleteentries and the rollup would keep counting children that are gone. On the maniflex path the child keeps both — a cascaded row records the same history a direct delete would, and every rollup it fed is recomputed once at the end of the sweep. The FK constraint is what is dropped, not the integrity: the children are deleted before the parent, in the same transaction.
Either way the whole deletion is atomic: a restrict that fires rolls back any
cascade that ran alongside it.
Large fan-outs
On the maniflex-enforced path the children of a deleted parent are walked in
pages of 500, ordered by id and bounded by the last id seen, and each page is
applied before the next is read. Only the id column is selected, so the memory
a delete needs is set by the page size rather than by how many children the row
has. restrict does not read the children at all — it asks how many there are.
The work is still proportional to the fan-out, and it all happens inside the delete request’s transaction: deleting a row with a million descendants is a long transaction holding locks the whole time. Prefer soft-deleting the parent and reaping in a job when the fan-out is that large.
A child that is Versioned or is a rollup child costs one extra read per row —
the walk selects only id, and the history row’s diff and the rollup’s foreign
key both need the rest of it. That is the price of the bookkeeping; a child with
neither pays none of it.
Existing SQLite tables. New FK constraints are declared when a table is created. SQLite cannot add a foreign key to a table that already exists, so adding
onDeleteto a model with a pre-existing SQLite table does not retroactively add the constraint — recreate the table, or rely on the soft-delete path, which is enforced in the application layer regardless. Postgres adds the constraint on the next migration.
Soft delete and relations
When the related model uses soft delete, rows whose deleted marker is set are
omitted from ?include= results — the same filter the framework applies to
list endpoints. See Soft Delete.
Scoping and ?include=
A request’s forced filters — the scope imposed by db.Tenancy or
db.ForceFilter — are applied to included rows as well as to the primary read,
for every relation kind, wherever the related model carries the filtered column.
For a many-to-many they apply to the junction rows too, on the same terms:
a link row is read only if it is in scope, whenever the junction carries the
column.
This matters because the foreign key is the client’s to set. Without it, a
caller who can write an FK (or a many-to-many junction row) puts their own row
inside another tenant’s ?include=, and an attach pulls another tenant’s record
into their own response.
A scoped write now refuses such a key up front — every BelongsTo key a create
or update sets is read through the request’s scope, and one naming a parent the
caller cannot see is refused with that parent’s 404, junction rows included.
The include scoping here is the second line: it holds for a row that reached the
table by some other road — an unscoped back-office write, an import, a row
predating that check.
// Tenancy on both models; the include is scoped by the same filter.
server.Pipeline.DB.Register(
db.Tenancy("org_id", orgOf),
maniflex.ForModel("Order", "OrderLine"),
)
What is not covered. Be precise about this, because the gap is narrow but real:
- A related model with no such column is not scoped. That is the right answer for the case that produces it — a shared lookup table (currencies, categories, statuses) is not tenant-partitioned and has nothing to scope by — but if a model is partitioned by something the filter cannot name, its includes are unscoped — and so is the key check on the write, which uses the same “does the parent carry this column” rule. Validating that FK is yours.
- Relation-path filters (
db.ForceFilterVia) are skipped: they are written against the primary model’s relations and mean nothing on the related table. - A link between two of your own rows written by another tenant is caught only if the junction carries the scope column. Scoping the related rows alone cannot catch it: both endpoints are yours, so both pass. Put the tenant column on a partitioned junction and register the scope on it, and the write is refused outright; without that column the junction is shared, and the link is neither refused nor hidden.
Before this release the junction was read unscoped and without its soft-delete condition, so a link another tenant wrote between two of your records surfaced in your include with that tenant’s
_throughpayload, and a soft-deleted link kept materialising its related row.
Quick reference
| Goal | Declaration |
|---|---|
Belongs to User (FK column matches) | UserID string |
Belongs to User under a different name | ManagerID string with mfx:"relation:Manager" + Manager User companion |
Has many Post | Posts []Post (other side carries UserID) |
Many-to-many via ProductTag | Tags []Tag with mfx:"through:ProductTag", on both sides |
| Cascade on parent delete | mfx:"...,onDelete:cascade" |
| Populate in a response | ?include=user,comments |
Soft Delete
A soft-deleted row is left in the database but marked as deleted. DELETE
requests flip the marker; list, read, and include queries hide rows whose
marker is set. This page covers how a model opts in, the two storage styles,
and the query semantics that follow.
Opting in
There are two ways to enable soft delete on a model. Both produce the same behaviour; pick whichever fits the declaration style of the rest of the model.
By embed
Embed one of the framework’s marker types:
type Article struct {
maniflex.BaseModel
maniflex.WithDeletedAt // timestamp-based soft delete
Title string `json:"title"`
}
| Embed | Column added | Storage style |
|---|---|---|
maniflex.WithDeletedAt | deleted_at — nullable timestamp; NULL means not deleted | timestamp |
maniflex.WithIsDeleted | is_deleted — boolean; false means not deleted | flag |
Both columns are tagged readonly and filterable. They are not part of any
write request — the framework manages them.
By configuration
The same setup, expressed at registration:
server.MustRegister(
Article{}, maniflex.ModelConfig{
SoftDelete: maniflex.SoftDeleteConfig{
Enabled: true,
Field: "deleted_at",
FieldType: maniflex.SoftDeleteTimestamp, // or maniflex.SoftDeleteBool
},
},
)
If both an embed and a ModelConfig.SoftDelete are present, the explicit
config wins.
Choosing between timestamp and boolean
Both styles work; they differ in what you can tell from the column afterwards.
WithDeletedAtrecords when the row was deleted, which makes audit trails, “deleted in the last 30 days” queries, and undelete-with-context possible. It is the default choice.WithIsDeletedstores only the fact of deletion. Use it when the surrounding system already records deletion timestamps elsewhere, or when a boolean fits an existing schema better.
Delete semantics
For a soft-deletable model, DELETE /api/<table>/{id} updates the marker
instead of removing the row:
| Style | What DELETE does |
|---|---|
| Timestamp | sets deleted_at to the current UTC time |
| Boolean | sets is_deleted to true |
The endpoint, the response, and the status code are the same as for a hard-delete model; only the underlying SQL differs.
Query semantics
Once enabled, soft-deleted rows are filtered out everywhere the framework reads the table:
- List (
GET /<table>) — only un-deleted rows are returned. - Read (
GET /<table>/{id}) — a soft-deleted row returns404. - Includes — relations populated via
?include=skip soft-deleted children, and a many-to-many skips links whose junction row is soft-deleted. - Update —
PATCHon a soft-deleted row returns404; the row is treated as absent. - Delete — a second
DELETEon the same row returns404and leaves the marker as it was, so the original deletion time survives. This holds inside a transaction too.
To surface the marker for clients that need it (e.g. an admin tool), filter on it explicitly:
# Only soft-deleted rows
curl 'localhost:8080/api/articles?filter=deleted_at:ne:null'
deleted_at and is_deleted are filterable, so the standard filter grammar
applies — see Querying.
Restoring a row
Opt a model in with ModelConfig.RestoreEnabled to mount a restore endpoint:
server.Register(Article{}, maniflex.ModelConfig{RestoreEnabled: true})
curl -X POST 'localhost:8080/api/articles/<id>/restore'
The request carries no body — it names the row in the URL and says only
“undo the delete”. A successful restore returns 200 with the restored record,
so the client need not re-read it.
| Status | When |
|---|---|
200 | restored; the record is returned |
404 | no such row, or the row exists and is not deleted (mirroring the re-delete guard) |
501 RESTORE_UNSUPPORTED | the configured adapter does not implement Restorer |
It is off by default, and deliberately so: un-deleting is a privileged operation, and an endpoint that appeared merely because a version was upgraded would not be covered by the authorisation an app had already written. The route is only mounted for models that actually soft-delete — on a hard-delete model there is nothing to restore.
It dispatches as an update
A restore runs as OpUpdate, not as an operation of its own. That is the point:
every middleware an app already registered for “who may modify this row” — auth,
tenancy, force filters, audit — governs un-deleting it, with nothing to rewrite
and no new operation constant to discover.
// This already covers restore. Nothing further to do.
server.Pipeline.Auth.Register(auth.RequireRole("editor"),
maniflex.ForOperation(maniflex.OpUpdate))
Where the two must be told apart — an audit sink recording “restored” rather
than “updated”, or a validation rule that only makes sense against a body — read
ctx.IsRestore(). Note a restore’s ctx.ParsedBody is empty, so body-driven
validation middleware sees nothing to check.
Scoping is enforced too. Because a soft-deleted row is invisible to every read path, the restore cannot read its target back to check scope the way an update does; instead the request’s forced filters are applied to the restore statement itself. A caller cannot un-delete a row outside their tenancy by knowing its id.
What it writes
Only the delete marker is cleared. updated_at is left untouched, so a restore
does not read as an edit to caches, sync clients, or anything else watching that
column — the audit or event record is where a restore is recorded.
Cascade is not undone. A restore brings back the row it names and nothing
else: nothing records which children an onDelete:cascade removed, so restore
each explicitly, or model the relationship so the children survive the parent’s
deletion. See Relations.
Custom adapters
The endpoint needs a database adapter implementing maniflex.Restorer. The
bundled SQLite and Postgres adapters do; a third-party adapter that does not is
unaffected and answers 501. It is a separate interface, not a DBAdapter
method, so adding it broke nothing:
type Restorer interface {
Restore(ctx context.Context, model *ModelMeta, id string, q *QueryParams) (any, error)
}
Apply q’s filters (the request’s forced scope, possibly nil) to the statement,
and return ErrNotFound when no row matches — including a row that exists but is
not deleted.
Interaction with hard delete
A model is either soft- or hard-delete; the choice is a property of the model, not the request. If you need a true hard delete on a soft-deletable model — for example, to honour an erasure request — perform it through a raw query or a custom action that bypasses the standard handler.
Quick reference
| Goal | Declaration |
|---|---|
| Timestamp soft delete | embed maniflex.WithDeletedAt |
| Boolean soft delete | embed maniflex.WithIsDeleted |
| Soft delete with a custom column | ModelConfig.SoftDelete |
| List only deleted rows | ?filter=deleted_at:ne:null (timestamp) or ?filter=is_deleted:eq:true (boolean) |
| Restore a deleted row | ModelConfig.RestoreEnabled → POST /:model/{id}/restore |
File Fields & Uploads
A field tagged mfx:"file" accepts an uploaded file alongside the model’s
JSON. The column stores an opaque storage key; the bytes live in a
configured FileStorage backend. Standalone upload, download, and delete
endpoints can also be mounted (see Standalone file endpoints).
Config changed. File settings now live under a single
Config.FilesConfigstruct (maniflex.FilesConfig). The old flatConfig.FileStorage,Config.FileSignedURLTTL, andConfig.FileMiddlewarefields have been removed. The mapping is:
Old ( Config.…)New ( Config.FilesConfig.…)FileStorageStorageFileSignedURLTTLSignedURLTTLFileMiddlewareBeforeMiddlewares(implied by FileStorage != nil)MountEndpoints— now explicit, see the footgunNew:
KeyGen(custom storage-key layout) andAfterMiddlewares(post-handler observation).
Declaring a file field
Add the file directive to a string field:
type Article struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required"`
Cover string `json:"cover" mfx:"file,max_size:2MB,accept:image/*"`
}
The column’s Go and DB types are string — what is stored is the storage key
returned after the upload. The on-disk bytes are managed by the storage
backend; the database row holds only the reference.
A file field’s Go type must be string (one key) or
maniflex.FileKeys (many). Anything else is a
registration error: every file rule is keyed on the column being a storage key,
so on another type they would all be skipped.
Tag sub-options:
| Sub-option | Effect |
|---|---|
file | mark the field as a file upload |
max_size:N | per-field size limit; suffixes KB, MB, GB or plain bytes. On a FileKeys field it bounds each file, not their total |
max_count:N | FileKeys only — maximum number of keys (default 100). See Many files per field |
accept:p1|p2 | allowed MIME-type patterns, e.g. image/*|application/pdf |
auto_delete:false | keep the stored file when the row is hard-deleted or the field is replaced (default: delete) |
upload:presigned | mount POST /{model}/{field}/upload-url so the client uploads straight to storage instead of through the app — see Direct-to-storage uploads. Requires a backend that can presign (storage/s3; not LocalStorage) |
upload:stream | multipart upload, but piped straight to storage as it arrives instead of buffered to the app server’s disk first — see Streaming uploads. Mutually exclusive with upload:presigned |
file_acl:private | (default) response carries the raw storage key; downloads go via /files/<key> or the per-model attachment route |
file_acl:signed | response replaces the key with a pre-signed URL valid for Config.FilesConfig.SignedURLTTL (default 1h). Requires FileStorage.URL() |
file_acl:public | response replaces the key with a permanent / long-lived URL (e.g. S3 7-day max). Pair with public-read ACL on the bucket for true permanence |
Those three are the whole set — anything else is a registration error naming the
field. A typo used to be read as private, which looks like the safe direction
and is not the safe outcome: the field was asked to serve signed or public URLs
and served raw storage keys instead, with nothing to say why.
Who may write a file field
file_acl decides how the value is presented. The access tags decide who may
set it, and they mean the same thing for a file field as anywhere else — a
multipart upload is a client write like any other:
| Tag | Client may upload | Key in responses | Attachment route |
|---|---|---|---|
| (none) | yes | yes | yes |
writeonly | yes | no | yes |
readonly | no | yes | yes |
hidden | no | no | yes |
immutable | on create only | yes | yes |
mfx:"file,writeonly" is the combination to reach for when the storage key
should not appear in responses but the file must stay uploadable and
downloadable. The client posts the file, every response omits the key, and
GET /{model}/{id}/{field} still streams the bytes. The OpenAPI read schema
marks the field writeOnly: true, so a generated client models it correctly
rather than expecting a key that never arrives.
type Document struct {
maniflex.BaseModel
// Uploadable, never echoed, still downloadable through the attachment route.
Scan string `json:"scan" db:"scan" mfx:"file,writeonly,max_size:10MB"`
// Server-managed: only the application sets this.
Report string `json:"report" db:"report" mfx:"file,readonly"`
}
A client upload to a readonly, hidden, or already-set immutable field is
refused with 422 VALIDATION_ERROR, and those fields are left out of the
multipart/form-data schema so a generated client never offers the part.
Before this was enforced,
readonlyandhiddenwere applied by stripping the field from the parsed JSON body — which a multipart part never passes through. Amfx:"file,readonly"field therefore refused a key reference but accepted an upload, so a client could set a server-managed document on create and replace its bytes on update. If you were relying on that to get “uploadable but not echoed”,writeonlyis the tag that means it.
accept matches the content type the client declared on the multipart part; when
the part declares nothing (or the generic application/octet-stream), the type is
sniffed from the first 512 bytes. A declared type is a client-supplied claim, so
treat accept as an input filter, not a security boundary — downloads are what
enforce safety, via X-Content-Type-Options: nosniff and forced attachment for
anything outside the inline allowlist (see
Standalone file endpoints).
Tag detail is in Field Tags Reference.
Many files per field (FileKeys)
A gallery, or any attachment set, is a maniflex.FileKeys column:
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required"`
Images maniflex.FileKeys `json:"images" mfx:"file,accept:image/*,max_size:5MB,max_count:10,file_acl:signed"`
}
It stores as a JSON array (JSONB on Postgres, TEXT on SQLite), so ordering is
preserved exactly as written — a gallery keeps its sequence. Every rule a
single-key field enforces applies per key: existence, max_size (per file,
not per array), accept, file_acl signing on read, auto_delete, and cleanup
on hard delete.
Write by key reference, not multipart. Upload each file first — via
POST /files or a
presigned upload — then send the
keys:
PATCH /posts/1
{"images": ["uploads/a1/one.jpg", "uploads/b2/two.jpg"]}
A multipart upload to a FileKeys field is refused (422). Multipart carries
one file per field, so it could only ever store one key and would silently drop
the rest; and routing many large files through the app process is what presigned
uploads exist to avoid.
A PATCH replaces the whole array, as it does any column. With auto_delete
(the default), keys present before the write and absent after it are deleted from
storage once the write commits. The diff is a set difference, so reordering
deletes nothing, and a key you keep is never touched:
PATCH /posts/1 {"images": ["uploads/a1/one.jpg", "uploads/c3/three.jpg"]}
// one.jpg kept — still referenced
// two.jpg deleted — dropped by this write
// three.jpg kept — newly referenced
max_count bounds the array (default DefaultMaxFileCount, 100). Every key
is Stat’d against storage to enforce the field’s rules, so an uncapped array
would be one request buying N storage round-trips. Over the cap is 422 TOO_MANY_FILES.
No attachment route is mounted for a FileKeys
field — that route streams one object’s bytes and a list names no single one.
Use file_acl:signed to have each key rewritten to a URL in the response.
file_acl modes
type Attachment struct {
maniflex.BaseModel
Logo string `mfx:"file,file_acl:public,max_size:1MB,accept:image/*"`
Resume string `mfx:"file,file_acl:signed,max_size:5MB,accept:application/pdf"`
Notes string `mfx:"file"` // implicit private — raw key in the response
}
The rewrite happens in the Response step on create, read, list, and update. A
null/empty value passes through unchanged — no fabricated URLs to nothing.
Configure the signed-URL lifetime once on Config:
maniflex.New(maniflex.Config{
FilesConfig: maniflex.FilesConfig{
SignedURLTTL: 15 * time.Minute, // default: 1h
},
})
S3Storage.URL uses awss3.NewPresignClient; ttl=0 (public mode) maps to
the AWS 7-day maximum.
LocalStoragecannot sign.URLreturns the server-relative/files/<key>for both signed and public modes, ignoring the TTL entirely — sofile_acl:signedagainst it yields a permanent path where a time-limited one was asked for, and how exposed that path is depends on whatever guardsGET /files/*. That is a weaker guarantee than the tag requests, so the framework no longer lets it pass unremarked: it warns at boot, naming the model and fields, andConfig.Strictmakes it a startup error.Use a signing backend, or mark the field
file_acl:private— its downloads go through the per-model attachment route, which enforces the same auth as reading the parent record. Bring an HMAC layer only if you need signed URLs and local disk.A backend declares this by implementing
maniflex.SignedURLCapable. Not implementing it means “I can sign”, soS3Storageand any third-party signer need do nothing.
Configuring storage
Uploads require a FileStorage implementation, set on
Config.FilesConfig.Storage. The framework ships one backend; bring your own
for cloud storage.
import "github.com/xaleel/maniflex/storage"
fs, err := storage.NewLocalStorage("./uploads")
if err != nil {
log.Fatal(err)
}
defer fs.Close()
server := maniflex.New(maniflex.Config{
Port: 8080,
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MountEndpoints: true, // mount POST/GET/DELETE /files — see the footgun below
},
})
LocalStorage pins the directory with Go’s directory-scoped os.Root; file
and metadata operations may follow symlinks only while they remain inside that
root. Links which escape it are rejected. Close the backend when the process no
longer uses it to release the root directory handle.
FileStorage is a small interface — Store, Retrieve, Delete, Exists,
URL — making S3, R2, GCS, or any other key-value store straightforward to
adapt.
When Storage is nil, model endpoints still accept JSON, but multipart
uploads and the standalone /files routes respond with 501 Not Implemented.
MountEndpoints is explicit
MountEndpoints gates only the standalone /files routes and defaults to
false. Setting Storage alone is not enough to mount them:
| You set | Multipart on mfx:"file" fields | Per-model attachment routes | Standalone /files |
|---|---|---|---|
Storage only | ✅ enabled | ✅ mounted | ❌ not mounted (404) |
Storage + MountEndpoints: true | ✅ enabled | ✅ mounted | ✅ mounted |
MountEndpoints: true, Storage nil | ❌ 501 | ❌ 501 | ✅ mounted, returns 501 |
Migration footgun. Before this release a non-nil
FileStorageauto-mounted/files. That is no longer implied — if you relied on the standalone endpoints, addMountEndpoints: true. Model file fields and per-model attachment routes remain gated onStoragealone and are unaffected.
Upload size limits
A multipart request is capped at 32 MB in total by default, across every
part. Anything larger is rejected with 413 BODY_TOO_LARGE while it streams —
before a byte is written to a temp file, let alone to storage. Raise or lower
the ceiling per app:
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MaxUploadBytes: 100 << 20, // total request size (default 32 MB)
MaxUploadMemory: 8 << 20, // buffered in RAM before spooling (default 32 MB)
}
MaxUploadBytes bounds the request; the per-field max_size tag bounds an
individual attachment within it. Both apply, and the request ceiling is
checked first — so max_size:2MB on a field does not stop a client from sending
a 50 GB body, but MaxUploadBytes does.
For a tighter limit on one model, register
body.MaxBodySize on the Deserialize step; it
overrides MaxUploadBytes for the models it is scoped to.
Custom storage keys with KeyGen
By default a POST /files upload is stored under
uploads/<uuid>/<sanitised-filename>. Override KeyGen to control the layout —
for example, to shard by tenant read from the request context:
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MountEndpoints: true,
KeyGen: func(ctx *maniflex.ServerContext, h *multipart.FileHeader) string {
tenant := ctx.Auth.Claims["tenant"] // populated by a BeforeMiddleware
return fmt.Sprintf("%s/%s", tenant, h.Filename)
},
},
The returned string is used verbatim as the storage key — sanitise any
user-supplied component yourself (the default uses sanitizeFilename). KeyGen
applies only to the standalone POST /files route; per-model attachment keys
are framework-generated.
KeyGen controls the layout of a key; it does not bind the key to a caller.
The framework adds a scope prefix on top of whatever KeyGen returns — see
Key ownership and FilesConfig.KeyScope.
S3, R2, MinIO, DigitalOcean Spaces
The satellite module maniflex/storage/s3 ships a FileStorage implementation
backed by the AWS SDK v2. It works against any S3-compatible service.
import "github.com/xaleel/maniflex/storage/s3"
store, err := s3.New(ctx, s3.Config{
Bucket: "my-app-uploads",
Region: "us-east-1",
// Endpoint, UsePathStyle, KeyPrefix, ACL are all optional.
})
if err != nil { log.Fatal(err) }
server.SetStorage(store)
Credentials follow the standard AWS resolution chain (env vars, shared
config, IAM instance role, IRSA, ECS task role). Override with
Config.AWSConfig when you need a custom credential provider, HTTP client,
or retry policy.
Per-service tips:
| Service | Endpoint | UsePathStyle |
|---|---|---|
| AWS S3 | leave empty | false |
| MinIO | http://localhost:9000 | true |
| Cloudflare R2 | https://<account>.r2.cloudflarestorage.com | false |
| DigitalOcean Spaces | https://<region>.digitaloceanspaces.com | false |
Use KeyPrefix to share one bucket across environments
(KeyPrefix: "staging/") — callers pass logical keys and never see the
prefix. File metadata (filename, size, content type) is stored as native
S3 object metadata so objects remain browsable via the AWS console and
aws s3 cp without the maniflex layer.
How uploads work
A model containing one or more file fields accepts multipart/form-data on
create and update, in addition to JSON:
- Form fields named the same as JSON fields populate the row’s scalar values.
- Form file parts named after a
filefield are streamed to storage; the resulting key is written to the column.
A form carries only strings, so a value bound for a column that is not one is
converted before it is written, and a string that will not convert is 422
rather than stored:
| Column | Accepted | Note |
|---|---|---|
| number | 7, 1.5 | out of range for the column’s width is refused |
| boolean | true / false, 1 / 0, on / off | on is what a checked checkbox posts; an unchecked one posts nothing, which is absence |
| timestamp | RFC 3339 (2020-01-02T03:04:05Z) | a bare date is refused |
| string | anything | passed through, so 01234 stays "01234" |
An empty value for a non-string column is read as no value at all, because
a browser posts every input in the form including the ones nobody filled in. A
nullable column (*int) stores NULL; one whose type has no null is refused
by name, the same answer the same body gets as JSON.
Conceptually:
POST /api/articles
Content-Type: multipart/form-data; boundary=...
--...
Content-Disposition: form-data; name="title"
The First Post
--...
Content-Disposition: form-data; name="cover"; filename="hero.png"
Content-Type: image/png
<bytes>
--...
The response is the usual JSON envelope; the cover field carries the storage
key the client uses to fetch the file later.
The framework rejects an upload before it reaches storage if it violates the
field’s max_size or accept constraints.
Sending a pre-uploaded key
A file field also accepts a plain string in JSON — the storage key of a file
already uploaded via the standalone endpoint. This is useful when the upload
is decoupled from the record creation (large files uploaded ahead of time,
re-using an existing file, and so on).
The key is checked against storage before the write: setting a file field to
a string key that does not exist in the configured FileStorage is rejected with
422 FILE_NOT_FOUND, so a record can never reference a dangling key. (Pass JSON
null to clear the field.) In production the key exists because the client
uploaded it first — via the multipart part, a prior POST /files, or a
presigned upload. In tests, seed the
key into the shared storage before referencing it.
The field’s max_size and accept rules apply here too, checked against the
object actually in storage — the same bytes get the same answer whichever way they
arrived. This is new in v0.2.3: before it, this path checked only that the key
existed, so uploading out of band and referencing the key was a way past both
rules. See the changelog if you relied on that.
Key ownership
A storage key is bound to the principal that mints it, so a key one caller
learns cannot be pinned onto another caller’s record. Keys are not secrets — they
travel in signed URLs and file_acl:private responses — and without this a caller
who obtained another record’s (or another tenant’s) key could reference it on their
own record, and an auto_delete field could then delete a blob it never owned.
Every minting path — POST /files, a presigned upload, and a multipart upload
through a model — prefixes the key with a hash of the caller’s scope, and a
reference by a different scope is refused with 403 FILE_FORBIDDEN. The scope
defaults to ctx.Auth.TenantID (so a tenant’s members share one), else
ctx.Auth.UserID. Override it when your principal lives elsewhere:
FilesConfig: maniflex.FilesConfig{
Storage: store,
// Bind each key to the user rather than the tenant. Guard the nil case:
// an anonymous request has no principal, and returns an unscoped key.
KeyScope: func(ctx *maniflex.ServerContext) string {
if ctx.Auth == nil {
return ""
}
return ctx.Auth.UserID
},
}
The scope must resolve consistently across the mint request and the later
reference — if your POST /files auth and your model auth populate ctx.Auth
differently, read whatever both share.
A key minted while no principal was present is unscoped and referenceable by
anyone; returning "" from KeyScope opts a request out. Keys minted before
v0.2.3 carry no scope marker and are likewise left to the existence check, so
upgrading breaks no reference to an already-stored key — the guarantee holds for
every key minted under a principal from v0.2.3 on.
Direct-to-storage uploads (upload:presigned)
By default an upload travels client → app → storage, and the app holds the whole body while it does: the multipart form is drained before the handler runs, and the in-memory buffer defaults to the same 32 MB as the body cap, so nothing spools to disk either. A 60 MB video therefore costs 60 MB of server memory and two hops of bandwidth to store one object.
Add upload:presigned and the bytes go straight to the bucket:
type Post struct {
maniflex.BaseModel
Title string `json:"title"`
Video string `json:"video" mfx:"file,upload:presigned,accept:video/mp4,max_size:60MB"`
}
That mounts one extra route:
POST /posts/video/upload-url
There is no record id in that path, deliberately: a create-time file field has no record yet, so a record-scoped route could not serve one. The same route works for create and update.
The flow is two phases, and the second one is just an ordinary write:
① POST /posts/video/upload-url
{"filename": "clip.mp4", "content_type": "video/mp4", "size": 41231234}
→ 200 {
"url": "https://bucket.s3.amazonaws.com/",
"method": "POST",
"fields": { "key": "...", "policy": "...", "x-amz-signature": "..." },
"key": "uploads/<uuid>/clip.mp4",
"max_size": 62914560,
"expires_at": "2026-07-17T13:05:00Z"
}
② the client POSTs the file straight to `url` as multipart/form-data,
sending every entry of `fields` first and the file last
③ POST /posts {"title": "...", "video": "uploads/<uuid>/clip.mp4"}
→ the ordinary create, which verifies the object and stores the key
Phase ③ is the completion step, and there is nothing else to call: the record
either names the key or it does not. That is why no pending-upload state exists to
reconcile — if a client uploads and never completes, no record references the
object and nothing is corrupt (the object itself is an orphan; see auto_delete).
The field’s rules bind at both ends. At ① the declared content_type and
size are checked against accept and max_size, so a URL is never minted for a
file the field would refuse. The limits are then pinned into the signature —
S3’s POST policy carries a content-length-range, so S3 itself rejects an
oversize body — and at ③ the stored object’s real size and type are checked again.
That last check is the one that matters: a signature can only bound what the
backend enforces, and the record is what makes an object real.
The client never chooses the key. It is minted server-side through
FilesConfig.KeyGen (so a per-tenant prefix scheme covers presigned uploads too)
and returned in the response. A client that could name the key could aim its
upload at another record’s object.
The authorisation covers one key, not one write. It is short-lived, not one-shot. Neither an S3 POST policy nor a presigned PUT can be spent — the storage backend verifies a signature, and a signature that verifies once verifies every time until it expires. So whoever holds the response from ① may repeat ② as often as they like inside the window — including after ③ has stored the key, overwriting the bytes a record already points at. No backend can close this; it is a property of signed URLs rather than of this framework.
Three things bound it. The key is minted server-side, so a replay can only
rewrite that object and cannot reach another. The mint route is authenticated,
so the window opens only for callers you already trust with the write. And
expires_at in the response is the hard edge — set by FilesConfig.SignedURLTTL,
default 1 hour.
An hour is generous for a capability that stays live after the upload finishes. Shorten it if your clients upload promptly — but note the same knob bounds signed downloads, so the two cannot be tuned apart; pick the shorter requirement. If the bytes must be immutable once written, verify them at ③ and copy the object to a key the client was never authorised for.
Auth applies. The mint route runs Auth → handler → Response, so whatever
gates the model gates the minting — granting the right to write an object is not
something to leave unauthenticated. Note the operation is
maniflex.OpPresignUpload, not OpCreate: the mint is not the create and can
precede one by minutes, so scope middleware with ForOperation(OpPresignUpload)
if you need it there.
Backend support. Presigning requires a backend that can mint one.
storage/s3can (AWS S3, R2, MinIO, Spaces, …).LocalStoragecannot, and says so: the route answers501 PRESIGN_UNSUPPORTEDrather than handing back an unsigned URL, which would be an open write endpoint rather than a degraded presigned one. Use the ordinary multipart upload withLocalStorage.
Streaming uploads (upload:stream)
upload:presigned takes the bytes off the app entirely, which is ideal when the
client can do the two-phase dance and the backend can presign. When it cannot —
LocalStorage, or a client that only knows how to POST one multipart form — the
bytes still have to pass through the app, and by default the whole request is
buffered first (in memory up to MaxUploadMemory, the rest to temp files) before
a single byte reaches storage. upload:stream removes that landing: the field’s
part is piped straight into FileStorage.Store as it arrives off the socket.
type Post struct {
maniflex.BaseModel
Title string `json:"title"`
Video string `json:"video" mfx:"file,upload:stream,accept:video/mp4,max_size:60MB"`
}
Nothing about the request shape changes — it is the same multipart POST /posts
as an unflagged file field. Only where the bytes go changes: to the backend
directly, not to the app’s disk first.
Two rules bind differently when the length is not known until the stream ends:
acceptis checked first, from the part’s declared type (or a sniff of the first 512 bytes when it declares none), so a disallowed type is refused with415before anything is stored.max_sizeis enforced mid-stream: the upload is cut off and answered413the moment it runs past the limit. By then a partial object exists in storage — the request ends non-2xx, so the same orphan cleanup that protects every failed upload deletes it (see Automatic cleanup). A backend that can reject early (S3 rejects an oversize multipart part) still saves the bandwidth; the framework catches the rest.
Because the length is unknown while streaming, FileStorage.Store receives a
FileMeta.Size of 0 and must store an unsized reader — read it to EOF rather
than trusting Size. storage/s3 does this transparently (the AWS SDK uploader
switches to a multipart upload); LocalStorage copies the reader as-is. A custom
backend that assumed a known length needs to handle the unsized case before
enabling upload:stream against it.
Which one?
upload:presignedwhen the bytes should never touch the app and the backend can presign — the best choice for the very largest files.upload:streamwhen they must pass through (to scan, transform, or because the backend cannot presign) but should not land on disk on the way. Plainfile(buffered) for everything else — it is the default for a reason, and streaming trades a pre-storage validation gate for a stored-then-cleaned one. The twoupload:strategies are mutually exclusive on one field; setting both is a registration error.
Standalone file endpoints
When MountEndpoints is true, three routes are mounted under PathPrefix:
| Method | Path | Action |
|---|---|---|
POST | /files | upload a single file (multipart, field name file) |
GET | /files/{key...} | stream the file with its original content type |
DELETE | /files/{key...} | remove the file from storage |
POST /files returns 201 with
{"data": {"key": "...", "content_type": "...", "size": ..., "filename": "..."}}.
The returned key is the value to store in a file-tagged column. It streams
the file part straight to storage as it arrives (see
Streaming uploads), so a large standalone
upload never lands on the app server’s disk; size in the response is the length
measured while streaming. A custom FilesConfig.KeyGen here sees a
*multipart.FileHeader whose Size is 0, since the length is not yet known —
key off Filename/Header instead.
GET /files/{key...} streams the body with Content-Type and Content-Length
from the stored metadata. For safety it always sends X-Content-Type-Options: nosniff and serves only an allowlist of content types (common images, PDF,
plain text) inline; everything else — including text/html and
image/svg+xml — is sent as a Content-Disposition: attachment download so a
stored file cannot execute script on the API origin. The original filename is
serialized as a standards-compliant media-type parameter rather than copied
verbatim from storage metadata; quotes and controls cannot create extra header
parameters, and names requiring UTF-8 use filename*. Missing keys return
404.
These endpoints are storage-key-addressed and have no built-in auth
when BeforeMiddlewares is empty. Set it to wrap the routes with the
same pipeline middleware (e.g. JWT, role checks) that protects your model
endpoints:
maniflex.New(maniflex.Config{
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MountEndpoints: true,
BeforeMiddlewares: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret, auth.JWTOptions{}),
auth.RequireRole("admin"),
},
},
})
Each middleware sees a synthesised ServerContext (Request, Writer, Ctx,
RequestID, logger — no Model/Operation, since these routes are outside
the model pipeline). Aborting the context short-circuits the request
before the file handler runs. Leaving BeforeMiddlewares empty keeps the
pre-fix behaviour for backward compatibility, but production deployments
should populate it — anyone who guesses a key could otherwise delete
arbitrary files. The server logs a warning at startup when /files is
mounted without BeforeMiddlewares. For a deliberately public file service,
set FilesConfig.AllowPublic; this suppresses strict/production validation
errors but installs no middleware and changes no runtime behavior.
Before vs. after middleware
BeforeMiddlewares run before the handler and own request control: they
can authenticate, populate ctx.Auth, or short-circuit (abort / set
ctx.Response) to replace the response entirely.
AfterMiddlewares run after the handler has served the request. Because
the handler streams its response (status, headers, body) straight to the
client, the response is already committed by the time an after-middleware runs —
so they are for observation and side effects only (audit logging, metrics,
cleanup), never for altering the response. Read the outcome with:
AfterMiddlewares: []maniflex.MiddlewareFunc{
func(ctx *maniflex.ServerContext, next func() error) error {
status := ctx.Writer.(interface{ Status() int }).Status()
log.Printf("file request completed: %d", status)
return next()
},
},
Setting ctx.Response from an after-middleware is ignored (and logged) rather
than corrupting the already-sent body. To alter, replace, or block the
response, use BeforeMiddlewares.
Per-model attachment routes
For each mfx:"file" field on each model, the framework mounts a record-
scoped download path:
GET /:model/:id/:file_field
E.g. GET /api/patients/123/discharge_summary streams the file referenced
by Patient.DischargeSummary for record 123.
Unlike GET /files/{key...}, this route runs through the read pipeline
for the parent record — the same Auth, soft-delete, and tenancy
middleware that protect GET /api/patients/123 also protect the download.
Use this for any attachment whose access depends on the parent row.
Response codes:
| Status | Meaning |
|---|---|
200 | file streamed with Content-Type, Content-Disposition, Content-Length |
206 | a Range request was satisfied — see Resumable downloads |
404 NOT_FOUND | the record does not exist (or is soft-deleted) |
404 FILE_NOT_SET | the record exists but the field is null/empty |
404 FILE_NOT_FOUND | the field references a key that is missing from storage |
401 / 403 | whatever the Auth middleware decided |
The route is only mounted when Config.FilesConfig.Storage is configured; with
no storage backend, the route is absent and requests return 404 from the
router. (Unlike the standalone /files endpoints, per-model attachment routes
do not require MountEndpoints.)
Internally this is dispatched as its own operation, maniflex.OpReadAttachment.
Middleware filtered by ForOperation(OpRead) does apply to attachment
requests: an attachment is a read of one record, so whatever decides who may
read that record decides who may download its file. The implication runs one way
only — ForOperation(OpReadAttachment) means attachment requests alone.
Before v0.2.5 an
OpRead-scoped middleware did not run for attachments, so tenancy written that way left the attachment route unscoped.
Resumable downloads (HTTP Range)
Both download routes — GET /files/{key...} and the per-model attachment
route — honour the Range request header, so a client can resume an
interrupted download or seek into a large media file instead of refetching it
from the start.
# the last 500 bytes
curl -H 'Range: bytes=-500' localhost:8080/api/files/uploads/<uuid>/clip.mp4
# resume from where a failed download stopped
curl -H 'Range: bytes=1048576-' localhost:8080/api/patients/123/scan
A satisfiable range answers 206 Partial Content:
| Header | Value |
|---|---|
Content-Range | bytes 1048576-2097151/8388608 — the window, and the object’s full length |
Content-Length | the window’s length, not the object’s |
Accept-Ranges | bytes (also sent on full 200 responses, so clients know they may resume) |
The security headers a full download sends — X-Content-Type-Options: nosniff
and the Content-Disposition inline/attachment decision — are identical on a
206.
Response codes:
| Status | When |
|---|---|
206 | the range resolved to a window of the object |
200 | no Range header, or one the server declined (see below) |
416 RANGE_NOT_SATISFIABLE | the range is well-formed but starts past the end of the object; Content-Range: bytes */<size> tells the client the real length |
Ranges the server declines. A malformed header, an unrecognised unit, and a
multi-range request (bytes=0-99,200-299) are all answered with the whole
object and 200 rather than an error — the spec permits ignoring any Range,
and serving several windows would require a multipart/byteranges body.
A client that asks for several windows gets the object and slices it itself.
Backend support
How much a range actually saves depends on the storage backend:
| Backend | Behaviour |
|---|---|
LocalStorage | seeks the file; only the window is read |
storage/s3 (S3, R2, MinIO, Spaces) | pushes the range into GetObject, so only the window leaves the bucket — you are not billed egress for bytes the client did not ask for |
A custom backend implementing RangeRetriever | same as above |
| A custom backend that does not | still works: the Range header is ignored and the whole object is served with 200, and Accept-Ranges is not advertised |
To support ranges in your own backend, implement the optional
maniflex.RangeRetriever alongside FileStorage:
func (s *MyStorage) RetrieveRange(
ctx context.Context, key string, offset, length int64,
) (io.ReadCloser, maniflex.FileMeta, error) {
// Return exactly the bytes [offset, offset+length).
}
The framework resolves the Range header against the size your Stat reports
before calling, so offset and length are always absolute and in bounds —
you never have to parse a header or clamp a window. Adding the method is all
that is needed; there is nothing to register.
Automatic cleanup
By default, a file field’s stored bytes are removed when:
- the record is hard-deleted, or
- the field is overwritten by an update.
Setting auto_delete:false opts out, leaving the file in storage for
out-of-band lifecycle management. Soft-deleted rows never trigger cleanup —
the file is preserved until the row is hard-deleted.
Bring-your-own storage
Implement maniflex.FileStorage:
type FileStorage interface {
Store(ctx context.Context, key string, r io.Reader, meta FileMeta) error
Retrieve(ctx context.Context, key string) (io.ReadCloser, FileMeta, error)
Delete(ctx context.Context, key string) error
Exists(ctx context.Context, key string) (bool, error)
Stat(ctx context.Context, key string) (FileMeta, error)
PresignUpload(ctx context.Context, key string, opts PresignUploadOptions) (*PresignedUpload, error)
URL(ctx context.Context, key string, opts PresignURLOptions) (string, error)
}
StatandPresignUploadare new in v0.2.3 and break every third-party backend until it implements them.
Statreturns an object’s metadata without fetching its body. Returnmaniflex.ErrFileNotFoundfor a missing key. ItsSizeandContentTypeare what afilefield’smax_sizeandacceptare checked against when a record references a key, so they must describe what is really stored — a client that uploaded 5 GB to a 60 MB field is caught here or nowhere. If your backend has no cheap metadata call,Retrieveand measure; it is correct, merely slower.
PresignUploadmints a direct-to-storage authorisation. If your backend cannot, returnmaniflex.ErrPresignUnsupported— the framework turns that into a clean501. Do not return an unsigned URL instead: that is not a degraded presigned upload, it is an unauthenticated write endpoint. Pinopts.MaxSizeinto the signature if you can (S3’s POST policy does, viacontent-length-range; a presigned PUT cannot), since the framework’s own check can only run once the bytes are already stored and paid for.
URL’s signature changed in v0.4.2 and breaks every third-party backend until updated. Thettl time.Durationparameter became aPresignURLOptionsstruct so callers can pin the response headers a URL serves with, not only how long it lives.The mechanical migration is
ttl→maniflex.PresignURLOptions{TTL: ttl};opts.TTLkeeps every meaningttlhad, zero still being the permanent / public mode. A backend that cannot honour the response overrides needs nothing further — ignoring them is supported, andLocalStoragedoes exactly that.
Retrieve returns maniflex.ErrFileNotFound when the key does not exist.
Delete should also return maniflex.ErrFileNotFound for missing keys so
the standalone DELETE /files/* handler can surface a 404 without an extra
Exists round-trip; backends that cannot detect the case atomically (e.g. S3
DeleteObject succeeds for missing keys) may return nil instead — both are
treated as “delete succeeded”. Store is given a framework-generated key of
the form uploads/<uuid>/<sanitised-filename>; create any intermediate
directories or object prefixes as needed.
Signed URL options
URL takes a PresignURLOptions. Beyond TTL, every field asks the backend to
pin a response header into the URL, so one signed link can serve an object
differently from how it is stored:
url, err := store.URL(ctx, key, maniflex.PresignURLOptions{
TTL: 15 * time.Minute,
Download: true,
Filename: "invoice-2026.pdf",
})
Download and Filename are the common case, and are encoded for you —
non-ASCII names come out in the RFC 5987 filename* form rather than as a raw
UTF-8 filename= that standards-compliant browsers discard. Backends must call
opts.ContentDisposition() rather than reading the fields directly, so every
backend encodes a name the same way. Set ResponseContentDisposition yourself
only for a form those two cannot express; it always wins over them.
The remaining fields — ResponseCacheControl, ResponseContentEncoding,
ResponseContentLanguage, ResponseContentType, ResponseExpires — map to
S3’s response-* query parameters, and VersionID selects one version of an
object in a versioned bucket.
They are best-effort: a backend honours what it can and ignores the rest.
LocalStorage ignores all of them and returns /files/<key> as before. That is
deliberate rather than unfinished — it serves from your application’s own
origin, where the parameters would be unsigned, so honouring a caller-chosen
ResponseContentType over stored bytes would be a stored-XSS vector. S3’s are
safe because they sit inside the signature.
Options set on a signed URL are not reachable from mfx:"file_acl:signed",
which still mints a URL with the configured TTL and nothing else. Call
FileStorage.URL yourself where you need the overrides.
Optional: RangeRetriever
Beyond FileStorage, a backend may implement maniflex.RangeRetriever to serve
byte windows for resumable downloads:
type RangeRetriever interface {
RetrieveRange(ctx context.Context, key string, offset, length int64) (io.ReadCloser, FileMeta, error)
}
It is optional and additive — a backend that omits it keeps working, serving the
whole object with 200 for a Range request. Implement it if your backend can
fetch a window natively (S3’s GetObject Range, a file Seek), since that is
where the saving is; the bounds are pre-resolved against your Stat size, so
they are always absolute and in range.
Storage backends are also expected to:
-
honour
ctxcancellation inStore— long uploads must abort when the request deadline elapses or the server is shutting down, -
refuse keys that name the backend’s own bookkeeping, if it keeps that bookkeeping in the same key namespace as the objects — sibling
.meta.jsonfiles, asLocalStoragedoes. Two rules make the difference between a guard and the appearance of one:Put the check in whatever function turns a key into a path, not in each method.
LocalStoragehad it inStoreandRetrieveand was missing it inStat,ExistsandDelete, so a client could delete the sidecar of any file it could name; the file kept serving, permanently stripped of its content type and download filename.Compare the way the storage layer compares. Windows and macOS match filenames case-insensitively, and Windows ignores trailing dots and spaces, so
x.META.JSON,x.meta.json.andx.meta.jsonall openx.meta.jsonthere. An exact suffix match reads as airtight and holds only on Linux.
Filenames flowing through the framework-generated key are sanitised to the
charset [A-Za-z0-9._-] (other runes become _), leading dots are stripped,
and the result is truncated to 120 characters. CR / LF / NUL bytes never
survive into the storage key.
File fields vs. static files
File fields handle user-supplied content. They are unrelated to Static Files, which serves a fixed directory of assets you ship with the app.
| File fields | Static files | |
|---|---|---|
| Source | uploaded at runtime | committed to the repo |
| Storage | FileStorage backend | local disk |
| URL | /files/<key> | /static/<path> |
| Configured by | Config.FilesConfig.Storage | a static/ directory |
Static Files
Alongside the generated model API, maniflex can serve a directory of plain static files — HTML, CSS, JavaScript, images, downloads — straight off disk. This is useful for a small admin page, a landing page, or assets referenced by an OpenAPI viewer, without standing up a separate web server.
How it works
Static serving is opt-in: set StaticDir to the directory you want served,
and every file inside it is served under the /static URL path.
server := maniflex.New(maniflex.Config{
StaticDir: "static", // serve ./static — nothing is served unless you name a dir
})
myapp/
├── main.go
└── static/
├── index.html → GET /static/index.html
├── css/app.css → GET /static/css/app.css
└── logo.png → GET /static/logo.png
Files are served verbatim, so a single-page app is served in full — its
index.html at the directory root and every nested asset at its own path:
static/
├── report.json → GET /static/report.json
└── admin/ → GET /static/admin/ (serves index.html)
├── index.html
└── scripts/app.js → GET /static/admin/scripts/app.js
If StaticDir is left empty, no static route is mounted and maniflex serves only
the API. If it is set but the directory does not exist, the server logs a warning
and skips the mount; the rest of the API is unaffected.
Changed in v0.2.1. Static serving used to default to
<cwd>/static— anystatic/directory in the working directory was published at/static/automatically. It is now opt-in: setStaticDirexplicitly. If you relied on the old default, addStaticDir: "static".
Customising the directory and prefix
Three maniflex.Config fields control static serving:
server := maniflex.New(maniflex.Config{
StaticDir: "public", // serve ./public
StaticPrefix: "/assets", // under /assets instead of /static
})
| Field | Default | Effect |
|---|---|---|
StaticDir | "" | filesystem directory served; empty serves nothing. A relative path resolves against cwd |
StaticPrefix | /static | URL prefix the directory is mounted under (at the router root) |
StaticDisabled | false | set true to turn serving off even when StaticDir is set |
StaticDirectoryListing | false | serve a listing for a directory with no index.html; 404 otherwise |
StaticDisabled exists so an app that sets StaticDir unconditionally can still
flip serving off from an env var or flag without clearing the field.
The /static route
A few details follow from how the route is mounted (the buildRouter block in
router.go):
- Resolved from the working directory. A relative
StaticDirresolves against<cwd>, wherever the process was started — not the location of the binary. Run the server from the project root, orcdthere first, so a relative path like"static"is found. - Mounted outside
PathPrefix. Static files live at/static/...(or yourStaticPrefix), not/api/static/.... ThePathPrefixfrommaniflex.Configscopes only the model API and/openapi.json; the static mount sits at the router root. - Trailing-slash redirect. A request to
/static(no trailing slash) is301-redirected to/static/. Requests below it are served directly. - Directory listing. A directory request serves its
index.html. Without one it answers404, unlessStaticDirectoryListingis set — a listing names every file in the directory, including ones nothing links to, so it is opt-in.
Behind maniflex.Mount
maniflex.Mount forwards PathPrefix and nothing else, so the second bullet
above has a consequence: a mounted server answers 404 for every asset it
serves standalone. Mount warns about it at startup, naming the prefix that
went dark, whenever StaticDir is set and StaticDisabled is not.
Forward the prefix yourself to fix it, reusing the server’s own handler rather
than an http.FileServer — which has none of the limits below:
r := chi.NewRouter()
maniflex.Mount(r, server)
// Static lives outside PathPrefix, so Mount does not carry it across.
inner := server.Handler()
r.Handle("/static/*", http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, chi.NewRouteContext()))
inner.ServeHTTP(w, req)
}))
The fresh chi.RouteContext is the working part: it lets the inner router
re-route from the URL. r.Mount("/static", server.Handler()) reads as the
obvious equivalent and answers 404, because chi hands a mounted handler an
already-stripped route path.
Set StaticDisabled when something else serves the assets — a CDN, or the outer
router’s own file handler — and the warning goes with it.
What is reachable
The mount is deliberately narrower than a plain file server, so pointing
StaticDir at a directory that is also a working tree does not publish it:
| Behaviour | |
|---|---|
| Methods | GET and HEAD only; anything else is 405 |
| Dotfiles | any path component starting with . is 404 — .env, .git/config, .htpasswd, .ssh |
.well-known | the one exception, so ACME renewal and security.txt keep working |
| Directories | index.html, else 404 (see StaticDirectoryListing) |
| Symlinks | resolved inside the directory through os.Root; a link out of it is 404 |
Relative symlinks within the tree are followed normally. A symlink with an
absolute target is refused even when it happens to point back inside, because
os.Root cannot confirm that without resolving the path outside its own walk —
which is the race it exists to remove. Use a relative link.
None of this makes an unsafe directory safe. It is still your call which directory is published; these limits only stop the most common ways one leaks more than intended.
Static files vs. file uploads
Static serving is for assets you ship with the app. It is unrelated to the
file-upload feature, which stores user-submitted files and is wired up
separately through Config.FilesConfig.Storage and the /files endpoints. For
user uploads see File Fields & Uploads.
| Static files | File uploads | |
|---|---|---|
| URL | /static/* | /files/* |
| Source | a directory you commit and name in StaticDir | user POSTs at runtime |
| Configured by | Config.Static* | Config.FilesConfig.Storage |
| Use for | app assets, admin pages | avatars, attachments |
Localization
maniflex has first-class support for multilingual string fields. A single
maniflex.LocaleString field stores all translations in one JSON column and
the framework resolves the right one for each request automatically.
The LocaleString type
maniflex.LocaleString is a map[string]string where each key is a locale
code and each value is the translation for that locale:
{ "en": "Finance", "ar": "مالية", "fr": "Finance" }
On SQLite it is stored as TEXT (JSON-encoded). On Postgres it is stored as
JSONB, which allows GIN-indexed key lookups.
Declare a field as locale-aware with the locale directive:
type Department struct {
maniflex.BaseModel
Name maniflex.LocaleString `json:"name" mfx:"locale,filterable,sortable"`
Code string `json:"code" mfx:"required,unique"`
}
On create and update the client sends the full map:
{ "name": { "en": "Finance", "ar": "مالية" }, "code": "FIN" }
On read the framework emits a locale-resolved view (see Response modes).
Response modes
Every LocaleString field has a response mode that controls the shape of
the field in API responses. The mode is resolved in this order:
- The field’s own
mfxtag (split,resolve, ordynamic) - The model’s
ModelConfig.DefaultLocaleMode - The app’s
LocaleOptions.DefaultLocaleMode - The framework default:
split
split (default)
Two keys are emitted in the response:
name— the resolved string for the effective localename_i18n— the fullmap[string]string(always present)
{
"name": "Finance",
"name_i18n": { "en": "Finance", "ar": "مالية" }
}
The resolved string gives display code a stable string type; the companion
_i18n map gives the editor everything it needs to build a translation form.
The companion suffix defaults to "_i18n" and is configurable via
LocaleOptions.SplitSuffix.
Writing a split-mode field back
Both keys are understood on write, so a client can PATCH the object it just GETed without special-casing localized fields:
| Body | Stored |
|---|---|
"name_i18n": {"en":"Finance","ar":"مالية"} | that map, verbatim — the companion wins |
"name": "Finance" (a bare string) | {"<effective locale>": "Finance"} |
"name": {"en":"Finance"} (a map) | that map |
The companion takes precedence because it is the complete value, while name
is one locale’s rendering of it. That is what makes an echoed response lossless:
the translations the response did not show still come back in _i18n.
A bare string replaces the column rather than merging into the stored map —
the same as a map write, which has never been a per-key patch. So sending only
"name": "Finance" with ?locale=en leaves the field as {"en": "Finance"}
and drops any other translations. Send the _i18n map when you mean to keep
them.
Before v0.2.5 the
_i18nkey was ignored on write and a bare string was stored as a JSON scalar, which the next read could not parse — a GET→PATCH round-trip through a generic edit form left the record unreadable, returning 500 for that record and for the whole collection, since one bad row fails the list scan. A column still holding a scalar from that era now resolves to its value and logs a warning instead of erroring, so the row can be repaired with an ordinary PATCH.That fix reached only one of the two write paths at first: the companion was consumed just when the body also carried a bare string in
name. Sent on its own it answered200and wrote nothing — an empty column on create — and sent beside anamemap it was dropped. The table above holds on every path as of this release.
resolve
The field is always a plain string — the resolved value for the effective locale. No companion field is emitted.
{ "name": "Finance" }
Use resolve when clients only ever need one language and the extra _i18n
key adds no value.
dynamic
Replicates legacy behaviour:
- When
?locale=is present: emits a string (resolved for that locale) - When
?locale=is absent: emits the full map
The field type is non-deterministic. Not recommended for new models.
Setting up the LocaleResolver
Install the LocaleResolver middleware on the Deserialize step before
registration, so it runs before the framework’s built-in Deserialize:
server.Pipeline.Deserialize.Register(maniflex.LocaleResolver(maniflex.LocaleOptions{
Supported: []string{"en", "ar", "fr"},
Default: "en",
FromHeader: true,
RTL: []string{"ar", "he", "fa", "ur"},
}))
LocaleOptions fields:
| Field | Type | Default | Purpose |
|---|---|---|---|
Supported | []string | all locales | Whitelist of accepted locale codes; locales not in this list fall back to Default |
Default | string | "en" | App-wide fallback locale used when the request carries no recognisable preference |
FromHeader | bool | false | Also parse Accept-Language; first match in Supported wins (quality values are ignored) |
RTL | []string | — | Locale codes with right-to-left script; matching requests get "_dir":"rtl" in response meta |
DefaultLocaleMode | LocaleMode | split | App-wide default mode for all LocaleString fields |
SplitSuffix | string | "_i18n" | Companion-field suffix used in split mode |
Locale resolution chain
When resolving which string to return, the framework walks this chain (most to least specific) and returns the first non-empty match:
- Explicit
?locale=query parameter Accept-Languageheader (first match inSupported), whenFromHeader: true- Field’s
default_locale:codetag - Model’s
ModelConfig.DefaultLocale - App’s
LocaleOptions.Default(default"en") - Any non-empty value in the map (last resort)
// Field-level default: Arabic is preferred for this field even when the
// request does not specify a locale.
Bio maniflex.LocaleString `json:"bio" mfx:"locale,default_locale:ar"`
// Model-level default: all locale fields on this model use French by default,
// unless overridden by a field's `default_locale` tag.
server.MustRegister(Article{}, maniflex.ModelConfig{DefaultLocale: "fr"})
Requiring a locale key
Use validate.RequireLocale to enforce that specific locale keys are present
and non-empty on create (or update) requests:
server.Pipeline.Validate.Register(
validate.RequireLocale("name", "en"),
maniflex.ForModel("Department"),
maniflex.ForOperation(maniflex.OpCreate),
)
A request that omits the "en" key, or supplies an empty string for it, is
rejected with HTTP 422 MISSING_LOCALE. Pass multiple keys to require several
locales at once:
validate.RequireLocale("name", "en", "ar")
Filtering and sorting locale fields
LocaleString fields tagged filterable and sortable work with the
standard query string grammar. In split and resolve mode the framework
automatically targets the effective locale’s JSON key in the database query.
GET /departments?filter=name:ilike:%25fin%25&sort=name:asc
In the example above, when the effective locale is "en", the adapter runs:
- SQLite:
json_extract("departments"."name", '$.en') LIKE '%fin%' - Postgres:
"departments"."name"->>'en' ILIKE '%fin%'
You can also filter a specific locale key explicitly:
GET /departments?filter=name.ar:contains:مال
In dynamic mode without an explicit ?locale= the filter hits the raw JSON
column, which typically returns no results for plain-string comparisons — this
is intentional: in dynamic mode the field’s meaning depends on request context.
Searching localized content
The mfx:"searchable" full-text directive (the ?q= endpoint) indexes plain
string columns only — tagging a LocaleString field searchable fails at
model registration, because full-text search has no single text column to index:
maniflex: model "Department" field "Name" is mfx:"searchable" but its type is
maniflex.LocaleString; full-text search only indexes text (string) columns
Two ways to make localized content searchable:
- Substring filter (simplest). Tag the
LocaleStringfieldfilterableand use?filter=name:ilike:%25term%25. Because the value is stored as one JSON object, anilikeagainst the raw column matches across every locale’s text at once — no extra schema. - Denormalized plaintext column. Add a plain
stringcolumn (e.g.SearchText) taggedsearchable, and populate it on create/update from a Service-step middleware that flattens the localized strings. Use this when you need true?q=full-text ranking rather than substring matching.
RTL meta
When the resolved locale is in LocaleOptions.RTL, every response envelope
gains a meta object with "_dir": "rtl" — for both list responses
(which already carry pagination in meta) and single-record responses
(read / create / update):
{
"data": {
"name": "مالية",
"name_i18n": { "en": "Finance", "ar": "مالية" }
},
"meta": { "_dir": "rtl" }
}
List responses include pagination alongside the direction flag:
{ "data": [...],
"meta": { "total": 5, "page": 1, "limit": 20, "pages": 1, "_dir": "rtl" } }
Clients can use meta._dir to switch text direction without needing to know
which locale is active.
Model-level mode override
Set a uniform mode for all LocaleString fields on a model without tagging
each one individually:
server.MustRegister(LegacyArticle{}, maniflex.ModelConfig{
DefaultLocaleMode: maniflex.LocaleModeDynamic,
})
Field-level tags take precedence over the model setting, which in turn takes
precedence over the app-level LocaleOptions.DefaultLocaleMode.
Full example
type Product struct {
maniflex.BaseModel
Name maniflex.LocaleString `json:"name" mfx:"locale,filterable,sortable"`
Description maniflex.LocaleString `json:"description" mfx:"locale,resolve"`
SKU string `json:"sku" mfx:"required,unique"`
}
func main() {
server := maniflex.New(maniflex.Config{
// application settings
})
server.Pipeline.Deserialize.Register(maniflex.LocaleResolver(maniflex.LocaleOptions{
Supported: []string{"en", "ar"},
Default: "en",
FromHeader: true,
RTL: []string{"ar"},
}))
server.MustRegister(Product{})
db, _ := sqlite.Open("store.db", server.Registry())
server.SetDB(db)
server.Start()
}
With this setup:
GET /productsreturnsnameas the resolved English string plusname_i18nwith all translations;descriptionis always a resolved English string.GET /products?locale=arresolves both fields to Arabic.POST /productswith{"name":{"en":"Laptop"},"sku":"LAP-01"}succeeds.
Pipeline Overview
Every HTTP request handled by a generated model route flows through the same six-step pipeline. Each step has a default behaviour supplied by the framework and a registry of user middleware that can run before, after, or in place of it. This page describes what each step is responsible for; later pages cover how to register middleware on them and how state flows between them.
The six steps
Auth → Deserialize → Validate → Service → DB → Response
| Step | Default behaviour |
|---|---|
| Auth | Pass-through. Populates nothing by default. User middleware sets ctx.Auth here. |
| Deserialize | Parses URL query parameters (page, limit, filter, sort, include) into ctx.Query. On POST/PATCH, reads the JSON body into ctx.ParsedBody (limit: 4 MB), or parses multipart/form-data into ctx.ParsedBody and ctx.Files. |
| Validate | For create and update, enforces the mfx: tag rules: strips readonly and id, strips immutable on update, checks required, enum, min, max, and that each value is one the field’s type can hold. |
| Service | Pass-through. Reserved for business logic supplied by user middleware. |
| DB | Dispatches to the configured adapter for the current operation — FindMany, FindByID, Create, Update, or Delete. Routes through ctx.Tx when a transaction is active. |
| Response | Builds the JSON envelope from ctx.DBResult and writes it to the http.ResponseWriter. |
The OpenAPI endpoint (GET /openapi.json) has its own three-step pipeline —
Auth → Generate → Response — accessible via server.Pipeline.OpenAPI. The
model-route pipeline described here is the one used for everything else.
Operations
The CRUD operation a request performs is identified by an Operation value
that is stable across all six steps:
| Operation | Triggered by |
|---|---|
OpList | GET /<table> |
OpRead | GET /<table>/{id} |
OpCreate | POST /<table> |
OpUpdate | PATCH /<table>/{id} |
OpDelete | DELETE /<table>/{id} |
OpOptions | OPTIONS /<table> or OPTIONS /<table>/{id} |
OpReadAttachment | GET /<table>/{id}/<file_field> — per-model attachment download (see Files) |
OpAction | a custom action endpoint registered with server.Action() |
A HEAD request has no operation of its own: it is GET with the body
suppressed, so it runs as OpList (collection) or OpRead (item) — same
middleware, same status code, no body. The OpHead constant still exists but is
never set on a request; scope HEAD-aware middleware with OpRead / OpList.
ctx.Operation is the value middleware uses to branch behaviour. OpAction
requests follow a trimmed pipeline (Auth → action handler → Response); the
Deserialize, Validate, Service, and DB steps are skipped for them.
Per-step responsibilities
Auth
The Auth step is the place to verify a token, look up a user, and set
ctx.Auth. The default handler does nothing; an unauthenticated request reaches
the DB layer with ctx.Auth == nil. Add a middleware here to reject anonymous
callers, populate identity, or check scopes.
Deserialize
The Deserialize step assembles request input from three sources:
- The URL query string becomes a
*QueryParamsonctx.Query. Filter and sort references are validated against the model’s tag-derived field lists. - A JSON body becomes
ctx.ParsedBody(a read-only*RequestBody, JSON-keyed) and is bound to the typed recordctx.Record. Bodies over 4 MB are rejected as413 BODY_TOO_LARGE; a genuine I/O failure reading the body is400 BODY_READ_ERROR. - A multipart body populates both
ctx.ParsedBody(the form fields) andctx.Files(the file parts). The form-field-to-file-field mapping is by name. A form carries only strings, so a value bound for a number, boolean or timestamp column is converted here — see How uploads work.
Reads carry no body, so only ctx.Query is populated for OpList / OpRead.
Validate
The Validate step runs only on OpCreate and OpUpdate. It applies the rules
declared by mfx: tags to ctx.ParsedBody:
readonlyfields and theidcolumn are silently stripped.immutablefields are stripped on update.requiredfields must be present on create.enum,min,maxare checked when the value is present.- Each value must be one the field’s Go type can hold:
{"age": "abc"}for anintcolumn is422, as are1.5for anint, a number for astring, and a timestamp that is not RFC 3339.
The type check applies to the columns whose type accepts one shape. A type that
brings its own — a LocaleString, or a SQLTyper such as money.Amount, which
takes both {"amount", "currency"} and a bare 12.34 — is asked whether it can
read the value rather than measured against a rule this step invented.
Validation failures abort the pipeline with 422 Unprocessable Entity and a
details payload listing every offending field.
Service
The Service step has no default behaviour — it exists for application logic.
Hashing a password before persistence, charging a payment, recomputing a
derived total, calling an external API: all of these belong here. A Service
middleware that needs to short-circuit the request calls ctx.Abort(...) and
returns without invoking next().
DB
The DB step is the only step with side effects on the database. It selects the
operation matching ctx.Operation, builds the column-keyed write set from
ctx.Record (falling back to ctx.ParsedBody), calls the adapter, and writes
the result into ctx.DBResult — a *ListResult for lists, otherwise the record
(a typed *T on reads). When ctx.Tx is set, the call is routed through the
transaction; otherwise the bare adapter is used.
Two error classes are normalised at this step:
maniflex.ErrNotFoundbecomes404 NOT_FOUND.*maniflex.ErrConstraintis split by kind: a unique or foreign-key violation becomes409 CONFLICT, while a NOT NULL violation becomes422 VALIDATION_ERROR(a missing-required-value problem, not a conflict).
A cancelled context becomes 504 TIMEOUT — unless the cancellation came from the
client hanging up, which is 499 with no body (see
Error Handling). All other adapter errors surface as
500 DB_ERROR.
Response
The Response step serialises ctx.DBResult into an APIResponse and writes it
to the wire. List responses include a meta block with total, page,
limit, and pages; single-record responses do not. The standard envelope is
{"data": ...} for success and {"error": {...}} for failure.
Short-circuiting
Any middleware can stop the pipeline by setting ctx.Response (typically via
ctx.Abort(status, code, message)) and returning without calling next().
Subsequent steps are skipped and the Response step writes the prepared error
envelope. This is the standard mechanism for unauthorised requests, validation
failures inside Service middleware, and any other refusal that should not
reach the database.
Per-step middleware
Each step exposes a *StepRegistry on server.Pipeline:
server.Pipeline.Auth.Register(jwtAuth)
server.Pipeline.Service.Register(hashPassword,
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate))
Registered middleware can run Before the default (the default position), After it, or Replace it entirely. Scoping by model and operation is covered in Writing Middleware.
Next
- ServerContext — the object threaded through every step.
- Writing Middleware —
Register, options, positions. - Transactions — wrapping the DB step in a transaction.
- Error Handling — the error envelope and sentinel errors.
ServerContext
*maniflex.ServerContext is the object threaded through every pipeline step for one
HTTP request. Steps read from it, write to it, and call next() to proceed.
Middleware does the same. This page documents the fields and methods middleware
will commonly touch.
Lifecycle
A new ServerContext is constructed by the handler for every request, populated
incrementally by the pipeline, and discarded once the response is written. It
is not safe to share across requests or goroutines.
The fields populated by each step are:
| Step | Sets |
|---|---|
| handler (before Auth) | Request, Writer, Ctx, Model, Operation, ResourceID, RequestID, TraceID |
| Auth | Auth (when user middleware populates it) |
| Deserialize | RawBody, ParsedBody, Query, Files |
| Service | (whatever user middleware sets) |
| DB | DBResult, possibly Tx |
| Response | Response, then writes it to Writer |
Routing context
Set by the handler before Auth runs; safe to read in any step.
| Field | Meaning |
|---|---|
Request | the original *http.Request |
Writer | the underlying http.ResponseWriter |
Ctx | the request context.Context; cancellation propagates from here |
Model | the *ModelMeta for the resource — name, table, fields, relations |
Operation | the Operation being performed (OpCreate, OpList, …) |
ResourceID | the {id} path parameter, empty for list and create |
RequestID | chi’s request ID, echoed in X-Request-Id |
TraceID | the W3C traceparent header, when present |
A Singleton has no {id} to read, so ResourceID is pinned to the
SingletonID placeholder. For an unscoped singleton that placeholder is the
row’s real id. For a scoped one it is not — there is one row per scope, and
the DB step swaps in the caller’s real id once the forced filters are known.
Middleware running before that swap — anything at the DB pipeline’s Before
position — therefore sees a value that addresses no row. Call
ctx.ResolveResourceID() instead of reading the field directly when you need an
id you can query with:
id := ctx.ResolveResourceID() // "" if the scoped row is not provisioned yet
It returns ctx.ResourceID unchanged for every non-singleton request, and never
provisions a row.
For it to resolve anything, the scope must already be established — register the
scope middleware with maniflex.ProvidesScope():
server.Pipeline.DB.Register(
db.Tenancy("org_id", tenantFromAuth),
maniflex.ProvidesScope(),
)
That hoists it to run right after Deserialize instead of at the DB step, so Validate, Service and the DB step all see the same scope. Without it the scope still applies to the query, but nothing before the DB step can ask what it is.
Step outputs
Populated in order by the pipeline.
| Field | Populated by | Type |
|---|---|---|
RawBody | Deserialize | []byte — the raw request bytes |
ParsedBody | Deserialize | *RequestBody — read-only JSON-keyed body (mutate via SetField) |
Record | Deserialize | the typed record carrier (*T for ctx.Model) bound from the body |
Query | Deserialize | *QueryParams — pagination, filters, sorts, includes |
Files | Deserialize (multipart only) | map[string]*UploadedFile |
DBResult | DB | *ListResult for lists; the record otherwise (a typed *T on reads) |
Response | Response | *APIResponse — the envelope written to the wire |
Setting Response from any step causes the remaining steps to skip and the
prepared envelope to be written. See Abort below.
Auth
Auth *AuthInfo is populated by Auth middleware. When nil, the request is
anonymous.
type AuthInfo struct {
UserID string
Roles []string
Claims map[string]any
TenantID string
IdentityType AuthIdentityType // human, service_account, anonymous
Scopes []string
SessionID string
AuthMethod string // "jwt", "api_key", "session", …
}
ctx.HasRole(role string) bool is a convenience wrapper that returns false
when Auth is nil.
Transactions
Tx Tx carries the active transaction, if any. When set, the default DB step
routes through it. ctx.BeginTx(ctx.Ctx, opts) returns a Tx and is the
standard way for middleware to start one. See Transactions
for the full pattern.
Aborting the pipeline
ctx.Abort(status int, code, message string) populates ctx.Response with an
error envelope. The current middleware must then return nil without calling
next(). Subsequent steps are skipped; the Response step writes the prepared
error. For 5xx statuses, the supplied message is logged with the request ID
but replaced by generic status text on the wire; 4xx messages are preserved.
if header == "" {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing token")
return nil
}
Calling next() after Abort
Abort does not stop the pipeline — it only populates ctx.Response. If the
middleware calls next() afterwards, the chain continues exactly as if the
abort had not happened:
- The remaining steps still execute, with all their side effects. The DB step will still issue its query and possibly modify the database; a Service middleware will still call out to external services.
- Any of those steps may overwrite
ctx.Response— for example, the DB step replaces it with a404 NOT_FOUNDif the record is missing, or the default Response step builds a200 OKenvelope fromctx.DBResult. Whichever step writes last wins. - If nothing downstream touches
ctx.Response, the original abort envelope is preserved and sent to the client — but the side effects have already happened.
The result is almost always a bug: either the client sees a misleading status
(a write succeeded but the response claims it was rejected), or the database
is mutated by a request that was meant to be refused. Always return without
next() after Abort.
Reading input
Three helpers wrap common request reads:
| Method | Purpose |
|---|---|
BindJSON(v any) error | decode the body into v, enforcing the 4 MB limit |
EnsureRawBody() ([]byte, error) | return RawBody, reading/buffering/restoring the body if an earlier step hasn’t (for middleware that needs the raw bytes in a trimmed action/search pipeline) |
URLParam(name string) string | read a chi URL parameter |
QueryParam(name string) string | read a URL query parameter |
BindJSON calls Abort internally on error and returns a non-nil error so the
caller can return nil immediately.
The request body
ctx.ParsedBody holds the deserialized JSON (or multipart form) body as a
*RequestBody. It is read-only: there is no exported way to index or assign
it, so a stray ctx.ParsedBody["x"] = y is a compile error. This is deliberate.
The body is mirrored onto a typed record (ctx.Record), and the only mutators —
ctx.SetField / ctx.DeleteField — keep both in sync; writing the map directly
would update one and not the other, and the change could be silently dropped at
the DB step.
Reading
| Call | Returns |
|---|---|
ctx.Field(name string) (any, bool) | one field by its JSON name |
ctx.ParsedBody.Has(name) bool | whether a key is present (an explicit null counts) |
ctx.ParsedBody.Keys() []string / .Len() int | the top-level key set |
ctx.ParsedBody.Map() map[string]any | a copy of the body, for read-only consumers |
All are nil-safe: ctx.ParsedBody is nil for body-less requests (GET, DELETE)
and the readers return zero values rather than panicking.
Numbers read from the body are float64
The body is JSON decoded into a map, so every number in it is a float64 —
including one written as an integer. A float64 represents every integer only
up to 2^53 (9007199254740992), so a larger one read back through ctx.Field,
ParsedBody.Map(), a validation callback or an ABAC policy has been rounded,
and can differ from what the client sent in either direction.
The stored value has not been. Writes source their columns from the typed record
(ctx.Record), where an int64 field decodes from the JSON number exactly, so a
large id or amount reaches the database intact — the body map is the fallback and
a typed write does not use it. That is the trap worth knowing: above 2^53 the
value a policy inspects and the value the row receives can disagree.
If a middleware needs an integer larger than 2^53 exactly, read it from the typed
record (maniflex.For[T](ctx)) rather than from the body map.
For typed access, read the whole body as the concrete model struct:
u, ok := maniflex.For[User](ctx) // (*User, bool) — false if no User body is bound
u, err := maniflex.Bind[User](ctx) // (*User, error) — errors when absent
// or adapt a typed handler straight into middleware:
server.Pipeline.Service.Register(
maniflex.Handle(func(ctx *maniflex.ServerContext, u *User) error {
if u.Age < 18 {
ctx.Abort(http.StatusUnprocessableEntity, "TOO_YOUNG", "must be 18+")
}
return nil
}),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
Writing
Middleware that injects or rewrites a field must go through these setters so the value reaches both the body and the typed record (and so the DB step persists it):
| Call | Effect |
|---|---|
ctx.SetField(name string, value any) | set a field by its JSON name |
ctx.DeleteField(name string) | remove a field (e.g. strip an input-only field) |
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
ctx.SetField("owner_id", ctx.Auth.UserID) // force the owner server-side
return next()
}, maniflex.ForOperation(maniflex.OpCreate))
Cross-step storage
For state that one middleware needs to pass to another:
ctx.Set("invoiceID", inv.ID)
// later, in another middleware:
id, ok := ctx.Get("invoiceID")
The store is per-request and discarded with the context.
Direct database access
Middleware that needs to reach beyond ctx.Model — to read another model, run
a raw query, or take a row lock — has four entry points, all routed through
ctx.Tx when one is active:
| Method | Purpose |
|---|---|
GetModel(name string) *ModelAccessor | CRUD on any registered model (.List / .Read / .Create / .Update / .Delete) |
RawQuery(sql string, args ...any) ([]map[string]any, error) | parameterised SELECT, CTE-SELECT, or a data-modifying statement with RETURNING (e.g. UPDATE … RETURNING id) |
RawExec(sql string, args ...any) (int64, error) | parameterised non-SELECT (returns rows affected) |
LockForUpdate(modelName, id string) (map[string]any, error) | pessimistic row lock; requires ctx.Tx |
GetModel returns an accessor whose methods route through ctx.Tx when set,
so middleware in a transaction does not have to thread the Tx manually.
Typed cross-model helpers
GetModel(name) is dynamic — string-named, exchanging map[string]any. For
compile-time types use the generic free functions, which resolve the model from
the type parameter and route through ctx.Tx the same way (so they also
participate in maniflex.Batch):
u, err := maniflex.Read[User](ctx, id) // *User
all, err := maniflex.List[User](ctx, nil) // []*User
created, err := maniflex.Create(ctx, &User{Name: "Jane"})
maniflex.Update(ctx, id, &User{ /* full record */ })
maniflex.Delete[User](ctx, id)
Results that are not a registered model — raw SQL, aggregates, recursive
queries — use maniflex.Row (an alias for map[string]any); RawQuery,
Aggregate, and RecursiveQuery return []maniflex.Row.
Placeholders in raw SQL are rebound to the adapter’s dialect, so ? works on
both SQLite and Postgres ($N). Always pass values as args — never interpolate
them into the query string.
Logging
ctx.Logger() *slog.Logger returns a slog logger pre-seeded with
request_id, trace_id, and service attributes, so log lines emitted from
middleware are correlated automatically.
ctx.Logger().Info("payment captured",
slog.String("invoice_id", inv.ID),
slog.Float64("amount", inv.Total),
)
Service name
ctx.ServiceName() returns the Config.ServiceName configured on the server.
Middleware uses this to enrich audit records or outgoing requests without
holding a reference to the framework Config.
Next
- Writing Middleware — composing middleware on these fields.
- Transactions —
ctx.Tx,BeginTx,LockForUpdate. - Error Handling —
Abortand the response envelope.
Writing Middleware
A middleware is a function that runs as part of one of the six pipeline steps. It can inspect and modify the request context, call into the database, decide whether to proceed, and inject behaviour before or after the step’s default handler.
Signature
type MiddlewareFunc func(ctx *maniflex.ServerContext, next func() error) error
A middleware does one of two things:
- Continue the pipeline — perform its work, then call
next()and return the result. The chain executes the remaining middleware in the step and then the steps after it. - Short-circuit — set
ctx.Response(typically viactx.Abort(...)) and returnnilwithout callingnext(). The remaining steps are skipped and the prepared response is written to the wire.
func bearerToken(ctx *maniflex.ServerContext, next func() error) error {
header := ctx.Request.Header.Get("Authorization")
if !strings.HasPrefix(header, "Bearer ") {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing bearer token")
return nil
}
ctx.Auth = &maniflex.AuthInfo{UserID: parseSubject(header)}
return next()
}
Typed middleware
For middleware that works with the request body, maniflex.Handle[T] adapts a
typed handler into a MiddlewareFunc. It hands you the bound record as a
concrete *T (the same value ctx.Record holds) instead of a map, runs only
when a *T body is bound, and is skipped on body-less operations:
server.Pipeline.Service.Register(
maniflex.Handle(func(ctx *maniflex.ServerContext, u *User) error {
if u.Age < 18 {
ctx.Abort(http.StatusUnprocessableEntity, "TOO_YOUNG", "must be 18+")
}
return nil
}),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
To change a field, call ctx.SetField(name, value) rather than mutating the
struct, so the write reaches both the body and the record (see
ServerContext › The request body). For an ad-hoc typed read inside
a plain middleware, use maniflex.For[T](ctx) / maniflex.Bind[T](ctx).
Where the record is bound matters. Deserialize decodes the client’s body into
*T before Validate runs, so a middleware registered on Deserialize sees the
raw request: readonly, hidden and immutable fields still hold whatever the
client sent, because nothing has refused them yet. From Validate onwards —
which is where the example above sits — those fields have been reset, so u.Role
is the empty string rather than the "admin" a client tried to send. Register
body-inspecting middleware on Validate or later unless you specifically want to
see what was attempted.
Registration
Each pipeline step exposes a *StepRegistry on server.Pipeline. Register a
middleware on the step where its work belongs:
server.Pipeline.Auth.Register(bearerToken)
Without options, the middleware applies to every model and every operation.
Register must be called before Start() or Handler(). Building the router
closes the registration window: each step’s chain is composed once per
(model, operation) and cached from then on, rather than rebuilt on every request,
so the middleware set has to stop changing. A Register after that point panics —
it could otherwise only apply to some requests and not others, and it would be
mutating a slice live requests are reading.
server.Pipeline.Auth.Register(bearerToken) // fine
server.Start() // window closes
server.Pipeline.DB.Register(audit) // panics
Scoping
Two functional options narrow the scope. They are independent and may be combined.
ForModel(names ...string)
Restrict to one or more models, by struct name:
server.Pipeline.Service.Register(hashPassword, maniflex.ForModel("User"))
ForOperation(ops ...Operation)
Restrict to specific operations:
server.Pipeline.Auth.Register(requireToken,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Operation values: OpList, OpRead, OpCreate, OpUpdate, OpDelete,
OpOptions, OpAction. Registering on Validate/Service/DB with OpAction has
no effect — those steps are skipped for action endpoints. A HEAD request runs
as the GET it mirrors (OpRead / OpList), so scope it with those.
Combining
server.Pipeline.Service.Register(chargePayment,
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpCreate),
)
Position
By default, a middleware runs before the step’s default handler. Use
AtPosition to change that.
| Position | When the middleware runs |
|---|---|
maniflex.Before (default) | before the default handler |
maniflex.After | after the default handler |
maniflex.Replace | instead of the default handler — the step’s built-in behaviour is skipped |
// Run after the DB step succeeds — useful for audit logs.
server.Pipeline.DB.Register(auditLog,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
// Replace the default DB step for one model entirely.
server.Pipeline.DB.Register(customDispatch,
maniflex.ForModel("LegacyOrder"),
maniflex.AtPosition(maniflex.Replace),
)
Within a step, all matching Before middlewares run in registration order,
then the core handler (default or Replace), then all matching After
middlewares in registration order. If multiple Replace middlewares match,
the last one registered wins.
A Replace on the DB step takes over feeding the Response step, so it must
leave ctx.DBResult in the shape that step expects: a *maniflex.ListResult for
a list, and a record for a read, create, or update. A record is a
map[string]any or a pointer to this model’s struct, and the same goes for
each of a ListResult’s Items. Anything else is rejected with
500 INVALID_DB_RESULT naming the type it got — including a pointer to another
model’s struct, or to a type defined from this one, since the response is built
by walking this model’s fields through whatever it is handed. A read, create or
update that leaves ctx.DBResult unset is rejected the same way; a list that
does is answered with an empty page. On a ListResult you need only set Items
and Total — a missing or partial Query is filled in with the default page
and limit.
Naming for traces
maniflex.WithName("name") attaches a human label to a middleware for use in
pipeline trace logs (enabled via Config.Trace). It does not change runtime
behaviour:
server.Pipeline.Auth.Register(rateLimit, maniflex.WithName("rate-limiter"))
Step-specific guidance
| Step | What middleware here typically does |
|---|---|
| Auth | Verify a token, populate ctx.Auth, reject unauthenticated requests. |
| Deserialize | Rarely customised. After middleware can rewrite the body via ctx.SetField / ctx.DeleteField. |
| Validate | Custom validation that goes beyond mfx: tags. Abort with 422 on failure. |
| Service | Business logic — derive fields, call external services, start transactions (maniflex.WithTransaction). |
| DB | Hooks around the database call. After middleware sees ctx.DBResult; Replace substitutes a different backend. |
| Response | After middleware can add headers; Replace lets you write a non-envelope response. |
After-middleware error handling
An After middleware sees ctx.Response when the default step has populated
it. Inspect it to decide whether to act:
func auditLog(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil {
return err
}
// don't audit failed writes
if ctx.Response != nil && ctx.Response.StatusCode < 400 {
record(ctx)
}
return nil
}
Per-model middleware at registration
For middleware that belongs to exactly one model, ModelConfig.Middleware
attaches hooks scoped to that model at registration time, avoiding the separate
Register call:
server.MustRegister(
Order{}, maniflex.ModelConfig{
Middleware: &maniflex.ModelMiddleware{
Validate: []maniflex.MiddlewareFunc{checkStock},
Service: []maniflex.MiddlewareFunc{chargePayment},
},
},
)
Both forms are equivalent; choose whichever keeps the declaration close to the code that depends on it.
Built-in middleware
Several middleware functions ship with the framework or its satellite modules — JWT auth, password hashing, audit logging, CORS, and more. They are documented in Middleware Catalogue.
Next
- ServerContext — the fields a middleware reads and writes.
- Transactions —
maniflex.WithTransactionas a Service-step middleware. - Error Handling — what
Abortproduces and how it propagates.
Transactions
A transaction wraps one or more database operations so they either all commit or all roll back. In maniflex the unit of transactional work is normally a single request: every database call performed by its pipeline runs in the same transaction, and the transaction commits if and only if the request produces a successful response.
Enabling transactions
The shipped middleware maniflex.WithTransaction wraps the DB step in a
transaction. Register it on the Service step, scoped to the operations that
should be transactional:
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil), // nil = default isolation level
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Once registered, every matching request executes the DB step (and any subsequent After-DB middleware) inside a transaction. The middleware:
- Begins a transaction before calling
next(). - Stores it on
ctx.Txand the underlyingctx.Ctx, so all downstream code can join it. - Commits if
next()returns nil andctx.Responseis a 2xx. - Rolls back if
next()returns an error, ifctx.Response.StatusCode >= 400, or if anything panics.
The same middleware can be registered on the DB step at Replace position if
you want to substitute the default DB step entirely.
Customising isolation
maniflex.TxOptions is an alias for sql.TxOptions:
server.Pipeline.Service.Register(
maniflex.WithTransaction(&maniflex.TxOptions{
Isolation: sql.LevelSerializable,
ReadOnly: false,
}),
maniflex.ForModel("Invoice"),
)
SQLite ignores most isolation levels. Write-locking does not depend on them:
sqlite.Open gives its write connections _txlock=immediate, so every
transaction takes the write lock at BEGIN rather than on its first write.
Joining the transaction from middleware
When ctx.Tx is set, every CRUD call made through ctx.GetModel(...),
ctx.RawQuery, ctx.RawExec, and the default DB step routes through it
automatically:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
// This Update participates in the same transaction as the DB step that follows.
if _, err := ctx.GetModel("Inventory").Update(itemID, map[string]any{
"reserved": true,
}); err != nil {
return err // triggers rollback
}
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
There is no separate “transactional” API — calling ctx.GetModel is enough.
Starting a transaction manually
When WithTransaction does not fit — for example, when only part of a request
should be transactional, or when the transaction must span an action endpoint
— begin one yourself:
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
ctx.Tx = tx
defer tx.Rollback() // no-op after Commit
// ... transactional work via ctx.GetModel / ctx.RawExec ...
if err := tx.Commit(); err != nil {
return err
}
ctx.Tx = nil // clear so post-commit code uses the bare adapter
tx.Rollback() after a successful commit is safe — it returns
sql.ErrTxDone, which the framework swallows. Always defer it.
Pessimistic locking
Inside a transaction, ctx.LockForUpdate(modelName, id) acquires a row-level
write lock and returns the current row:
row, err := ctx.LockForUpdate("StockBalance", stockID)
if err != nil {
return err
}
if row["quantity"].(int64) < 1 {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "no inventory remaining")
return nil
}
On Postgres this appends FOR UPDATE to the SELECT. On SQLite the lock is at
the transaction level — the row is protected because the transaction itself is
write-locked.
LockForUpdate returns an error if ctx.Tx is nil; the lock is meaningless
outside a transaction.
Joining from outside a ServerContext
Code without access to *ServerContext — for example, a job-queue helper, or an
outbox writer — can retrieve the active transaction from ctx.Ctx using
maniflex.TxFromContext:
func enqueue(ctx context.Context, job Job) error {
if tx := maniflex.TxFromContext(ctx); tx != nil {
return enqueueInTx(tx, job)
}
return enqueueDirect(job)
}
WithTransaction stores the active transaction on the context.Context for
exactly this purpose.
The transaction is reachable only while it is open. Once WithTransaction
commits or rolls back, both ctx.Tx and TxFromContext go back to nil, so a
middleware that resumes after its next() — an audit hook, an outbox enqueue —
takes the tx == nil branch and writes through the bare adapter, rather than
into a finished transaction.
Adapter scope
A transaction lives on a single DBAdapter. ctx.BeginTx opens the
transaction on the request’s model adapter (its ModelConfig.Adapter if
set, otherwise Config.DB). All operations on that ctx.Tx must target
models routed to the same adapter.
maniflex.Batch enforces this at runtime: calling b.Create("X", ...) for a
model that routes to a different adapter than the batch transaction
returns an error suggesting pkg/saga for cross-adapter coordination.
See Per-model adapter routing.
Nesting
WithTransaction is idempotent: if ctx.Tx is already set when it runs, it
simply calls next() without starting a new transaction. The outer transaction
remains the unit of commit.
SQLite does not support nested transactions; calling ctx.BeginTx while one
is already active returns an error.
Failure semantics
A transaction is rolled back when:
- the chain returns a non-nil error from any step;
ctx.Responseis set to a status>= 400(e.g. viactx.Abort);- a panic occurs (the framework’s panic recoverer ensures rollback).
WithTransaction is committed when:
- the chain completes without error and
ctx.Responseis a 2xx (or unset).
A commit failure is reported as 500 TX_COMMIT_ERROR; a begin failure as
500 TX_BEGIN_ERROR.
Next
- Writing Middleware — how to register
WithTransactionor write your own transactional middleware. - Error Handling — how a rollback surfaces to the client.
Error Handling
Every error returned by the pipeline is delivered to the client as the same JSON envelope. This page describes that envelope, the sentinel errors the framework recognises, and how to produce errors from middleware.
The error envelope
A failing request writes:
{
"error": {
"code": "VALIDATION_ERROR",
"message": "field \"email\" is required",
"details": { /* optional, per-error */ }
}
}
with the HTTP status code from the underlying failure. The code is a short
machine-readable identifier; message is a human-readable summary; details
is optional and may carry per-field errors or other structured context.
A successful request uses the {"data": ...} envelope instead; the two are
mutually exclusive.
Built-in error responses
The default pipeline produces the following errors without any user code:
| Status | Code | Source |
|---|---|---|
400 | INVALID_JSON | malformed JSON body |
400 | EMPTY_BODY | empty body on POST / PATCH |
400 | BODY_READ_ERROR | an I/O failure while reading the request body |
400 | INVALID_QUERY | unknown filter/sort field, malformed ?include, etc. |
400 | MULTIPART_ERROR | malformed multipart/form-data |
404 | NOT_FOUND | record does not exist (or is soft-deleted) |
409 | CONFLICT | unique or check constraint violated |
413 | BODY_TOO_LARGE | request body exceeded the 4 MB read limit |
422 | VALIDATION_ERROR | one or more mfx: tag rules failed |
500 | DB_ERROR | unclassified adapter error |
500 | TX_BEGIN_ERROR / TX_COMMIT_ERROR | transaction lifecycle failure |
499 | (no body) | the client disconnected before the response was written |
501 | NO_STORAGE | file endpoint hit with no FileStorage configured |
504 | TIMEOUT | Config.QueryTimeout (or another server-side deadline) expired |
A panic anywhere in the pipeline is caught and reported as 500 PANIC by the
framework’s recoverer — with one deliberate exception. A handler that panics with
http.ErrAbortHandler is abandoning its response on purpose (this is what
httputil.ReverseProxy does when an upstream dies mid-stream), so the recoverer
passes it on and net/http closes the connection silently. It is not logged as a
panic, and no error envelope is appended to whatever was already written.
499 is nginx’s non-standard “Client Closed Request” (maniflex.StatusClientClosedRequest).
Nothing is written to the client — the connection is already gone — but the status
is what your access log, metrics, and any After middleware reading the response
status will see, so a caller who hangs up is not counted as a server timeout.
Only a genuine server-side deadline produces 504 TIMEOUT. The disconnect itself
is logged at DEBUG, not as an error.
Aborting from middleware
The standard way to produce an error from middleware is ctx.Abort:
ctx.Abort(statusCode, code, message)
It populates ctx.Response with an error envelope. The caller must then return
nil (or an error) without calling next():
if header == "" {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing token")
return nil
}
Subsequent steps are skipped; the Response step writes the prepared envelope.
Calling next() after Abort allows downstream steps to overwrite the
response — usually not what you want.
For a 5xx status, message is a private diagnostic: Maniflex logs it once
through Config.Logger, correlated by request_id, but sends only generic
status text such as internal server error or bad gateway. The HTTP status
and stable error code remain public. Any details on a directly assigned
5xx APIResponse are also removed at the wire boundary. Validated 4xx
messages and details are unchanged. Every routed response echoes the
correlation value in X-Request-Id.
Returning structured details
For per-field errors and similar payloads, set ctx.Response directly:
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusUnprocessableEntity,
Error: &maniflex.APIError{
Code: "VALIDATION_ERROR",
Message: "one or more fields failed validation",
Details: []map[string]string{
{"field": "email", "message": "must be a valid email"},
{"field": "password", "message": "must be at least 8 characters"},
},
},
}
return nil
This is the shape used by the default Validate step.
Sentinel errors from the adapter
Adapter methods return errors that the DB step maps to HTTP responses. The two that user code most often interacts with are exported as sentinels.
maniflex.ErrNotFound
var ErrNotFound = errors.New("record not found")
Returned by FindByID, Update, and Delete when the row does not exist (or
is soft-deleted). Detect it with errors.Is:
row, err := ctx.GetModel("Invoice").Read(id)
if errors.Is(err, maniflex.ErrNotFound) {
ctx.Abort(http.StatusNotFound, "INVOICE_NOT_FOUND",
fmt.Sprintf("invoice %s does not exist", id))
return nil
}
The default DB step does this conversion automatically; you only need it when you are calling the adapter yourself from a Service middleware.
*maniflex.ErrConstraint
type ErrConstraint struct {
Kind ConstraintKind // unique, foreign_key, or not_null
Table string
Column string // may be empty when the driver does not expose it
Detail string // raw driver message
}
Returned by Create and Update on unique or check constraint violations.
Both SQLite and Postgres errors are normalised into this type, so middleware
need not inspect driver-specific codes.
row, err := ctx.GetModel("User").Create(data)
var ec *maniflex.ErrConstraint
if errors.As(err, &ec) {
ctx.Abort(http.StatusConflict, "DUPLICATE",
fmt.Sprintf("%s already exists", ec.Column))
return nil
}
Every exported sentinel
ErrNotFound and ErrConstraint above are the two you match in a handler. The
rest are listed here because they are part of the compatibility contract: a
sentinel keeps its identity for all of v1, so errors.Is against any of them
goes on working. Match on these values, never on message text — the text is
explicitly not covered (see
Stability & Compatibility).
Core module, github.com/xaleel/maniflex:
| Sentinel | Returned when |
|---|---|
ErrNotFound | the row does not exist, or is soft-deleted |
*ErrConstraint | a unique or check constraint was violated — a type, not a value; use errors.As |
ErrNoAdapter | BeginTx is called with no database adapter configured |
ErrRawNotSupportedInTx | RawQuery/RawExec run inside a transaction whose Tx cannot execute raw SQL. Refused rather than silently run outside the transaction |
ErrIncrementOutOfBounds | Increment would take a column past its mfx:"min:"/"max:" bound. Distinct from ErrNotFound: only this one may succeed on a retry |
ErrIncrementNotSupported | the adapter does not implement Incrementer |
ErrFileNotFound | FileStorage.Retrieve is given a key that does not exist |
ErrPresignUnsupported | the storage backend cannot mint a presigned upload; the upload-url route answers 501 |
ErrAlreadyStarted | a start method is called while the server already has an active startup owner |
ErrStopped | a start method is called after the server stopped or failed. A server is not restartable — build a new one |
ErrRegistrationClosed | a route or spec contributor is registered after Start/Handler sealed the server |
Subpackages:
| Sentinel | Package | Returned when |
|---|---|---|
ErrBusClosed | events/inproc | Publish after Close — the event was never accepted |
ErrQueueFull | events/inproc | a matching subscription’s queue is full. The bus is behind; the caller still holds the event |
ErrDrainIncomplete | events/inproc | Close gave up with deliveries still running |
ErrUnauthorized | realtime | an Authenticator rejected a connection — a type, carrying Reason |
ErrResponseTooLarge | pkg/integration | an upstream response exceeded MaxResponseBytes |
ErrCrossOriginRedirect | pkg/integration | the redirect policy refused to forward the request, and its headers, to another origin |
ErrHTTPStatus | pkg/integration | the upstream answered non-2xx — a type, carrying the status and decoded body |
ErrWebhookReplay | pkg/integration | a signed webhook was already accepted; the receiver maps it to 409 |
ErrImbalanced | pkg/ledger | debits do not equal credits for some currency in the entry |
ErrNoLines | pkg/ledger | Post was called with fewer than two lines |
ErrCurrencyMismatch | pkg/money | Add/Sub were given amounts in different currencies |
TestErrorsDocListsEveryExportedSentinel fails when an exported Err*
identifier is missing from this page, so a new one cannot ship undocumented.
Errors and transactions
When a request runs inside a transaction (see Transactions)
and any step returns an error or sets ctx.Response to status >= 400, the
transaction is rolled back before the response is written. The client sees the
error envelope; the database sees no change.
Logging errors
ctx.Logger() returns a *slog.Logger pre-seeded with request_id and
trace_id, so a single log line correlates with the request that produced it:
ctx.Logger().Error("payment provider rejected charge",
slog.String("provider", "stripe"),
slog.String("error_code", resp.Code),
)
ctx.Abort(http.StatusBadGateway, "PAYMENT_PROVIDER_ERROR", resp.Message)
return nil
Abort logs the supplied 5xx message automatically. The explicit log is still
useful when you have structured provider fields that should not be flattened
into the diagnostic string. Neither resp.Message nor those fields reach the
client.
Next
- ServerContext — the full set of fields available to error-producing middleware.
- Transactions — rollback semantics.
- Writing Middleware — where to attach error-producing logic.
Example 2: B2B SaaS API
This example builds a small multi-tenant SaaS backend. Compared to Example 1, it exercises every concept introduced in the “Defining Your API” and “The Request Pipeline” sections:
- Multiple related models with
BelongsToandHasManyrelations. - Soft delete on the records that matter for an audit trail.
- A bearer-token Auth middleware that populates
ctx.Auth. - A Service-step middleware that scopes every query to the caller’s tenant.
- A Service-step middleware that runs inside a transaction.
- Custom error envelopes with
ctx.Abort.
The goal is to show how the pieces compose; the auth and tenancy code is deliberately minimal so the example fits on one page.
Domain
A SaaS platform with three resources:
- Organization — the tenant. Every other record belongs to one.
- Member — a user belonging to an organization, with a role.
- Project — a unit of work owned by a member.
type Organization struct {
maniflex.BaseModel
maniflex.WithDeletedAt
Name string `json:"name" mfx:"required,filterable,sortable"`
Plan string `json:"plan" mfx:"required,enum:free|pro|enterprise,default:free"`
Members []Member `json:"members,omitempty"`
Projects []Project `json:"projects,omitempty"`
}
type Member struct {
maniflex.BaseModel
maniflex.WithDeletedAt
OrganizationID string `json:"organization_id" mfx:"required,filterable,immutable"`
Email string `json:"email" mfx:"required,filterable,unique"`
Role string `json:"role" mfx:"required,enum:owner|admin|editor|viewer,default:viewer,filterable"`
Projects []Project `json:"projects,omitempty"`
}
type Project struct {
maniflex.BaseModel
maniflex.WithDeletedAt
OrganizationID string `json:"organization_id" mfx:"required,filterable,immutable"`
OwnerID string `json:"owner_id" mfx:"required,filterable,relation:Owner"`
Owner Member `json:"owner,omitempty"`
Name string `json:"name" mfx:"required,filterable,sortable"`
Status string `json:"status" mfx:"required,enum:active|paused|archived,default:active,filterable,sortable"`
}
Owner is a companion field; the explicit relation:Owner tag is required
because the FK name (OwnerID) does not match the model name (Member).
OrganizationID follows the convention, so no companion is needed there.
Wiring
A single main.go registers the models, installs three middlewares, and
starts the server.
func main() {
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
server.MustRegister(Organization{}, Member{}, Project{})
db, err := sqlite.Open("./saas.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
registerMiddleware(server)
if err := server.Start(); err != nil {
log.Fatal(err)
}
}
Auth — populating ctx.Auth
A real deployment would verify a JWT; this example resolves a bearer token
against an in-memory map to keep the focus on ctx.Auth:
var tokens = map[string]maniflex.AuthInfo{
"alice-token": {UserID: "user-alice", TenantID: "org-acme", Roles: []string{"owner"}},
"bob-token": {UserID: "user-bob", TenantID: "org-acme", Roles: []string{"editor"}},
"carol-token": {UserID: "user-carol", TenantID: "org-globex", Roles: []string{"admin"}},
}
func bearerAuth(ctx *maniflex.ServerContext, next func() error) error {
header := ctx.Request.Header.Get("Authorization")
token := strings.TrimPrefix(header, "Bearer ")
info, ok := tokens[token]
if !ok {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "invalid or missing token")
return nil
}
ctx.Auth = &info
return next()
}
Tenant scoping — filtering by ctx.Auth.TenantID
A B2B API must never leak data across tenants. A Service-step middleware inspects every list and read, and rejects writes that would assign records to a foreign organization:
func enforceTenant(ctx *maniflex.ServerContext, next func() error) error {
tenant := ctx.Auth.TenantID
switch ctx.Operation {
case maniflex.OpList:
// Inject a filter so users only see their organization's rows.
ctx.Query.Filters = append(ctx.Query.Filters, maniflex.Filter{
Field: "organization_id", Op: "eq", Value: tenant,
})
case maniflex.OpCreate, maniflex.OpUpdate:
if v, ok := ctx.Field("organization_id"); ok && v != tenant {
ctx.Abort(http.StatusForbidden, "TENANT_MISMATCH",
"organization_id does not match the authenticated tenant")
return nil
}
ctx.SetField("organization_id", tenant)
}
return next()
}
The middleware applies to the two tenanted models — Member and Project —
and to every operation. Organization itself is not scoped, because the auth
middleware already binds each token to exactly one organization.
Transactions — creating a project atomically
A new project should fail entirely if the member quota check fails. A Service
middleware wraps the operation in a transaction and uses
ctx.LockForUpdate to read the organization with a write lock:
func enforceProjectQuota(ctx *maniflex.ServerContext, next func() error) error {
org, err := ctx.LockForUpdate("Organization", ctx.Auth.TenantID)
if err != nil {
return err
}
rows, err := ctx.RawQuery(
`SELECT COUNT(*) AS n FROM projects WHERE organization_id = ? AND deleted_at IS NULL`,
ctx.Auth.TenantID,
)
if err != nil {
return err
}
count := rows[0]["n"].(int64)
limit := planLimit(org["plan"].(string))
if count >= limit {
ctx.Abort(http.StatusPaymentRequired, "PROJECT_LIMIT",
fmt.Sprintf("plan %q allows %d projects; upgrade to add more", org["plan"], limit))
return nil
}
return next()
}
func planLimit(plan string) int64 {
switch plan {
case "enterprise":
return 1000
case "pro":
return 25
default:
return 3
}
}
Registering the middleware
All three middlewares are registered in one place:
func registerMiddleware(s *maniflex.Server) {
// Auth on every write — reads are public within the tenant once they
// pass enforceTenant below; tighten or relax to taste.
s.Pipeline.Auth.Register(bearerAuth,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete, maniflex.OpList, maniflex.OpRead),
)
// Tenant scoping for the two tenanted models.
s.Pipeline.Service.Register(enforceTenant,
maniflex.ForModel("Member", "Project"),
)
// The DB step is wrapped in a transaction for project creation, and the
// quota check runs inside it before next() reaches DB.
s.Pipeline.Service.Register(maniflex.WithTransaction(nil),
maniflex.ForModel("Project"), maniflex.ForOperation(maniflex.OpCreate),
)
s.Pipeline.Service.Register(enforceProjectQuota,
maniflex.ForModel("Project"), maniflex.ForOperation(maniflex.OpCreate),
)
}
Order matters: WithTransaction is registered before enforceProjectQuota, so
the transaction is open by the time the quota check calls ctx.LockForUpdate.
A request, end to end
# Alice (org-acme, owner) creates a project.
curl -X POST localhost:8080/api/projects \
-H 'Authorization: Bearer alice-token' \
-H 'Content-Type: application/json' \
-d '{"name":"Atlas","owner_id":"user-alice"}'
What happens:
- Auth —
bearerAuthresolvesalice-tokenand setsctx.Auth. - Deserialize — JSON body parsed into
ctx.ParsedBody. - Validate —
mfx:tag rules pass;organization_idis missing but the next step injects it. - Service:
enforceTenantwritesorganization_id = "org-acme"into the body.WithTransactionbegins a transaction.enforceProjectQuotalocks the organization row, counts existing projects, and either aborts with402 PROJECT_LIMITor proceeds.
- DB —
Createruns through the transaction;ctx.DBResultholds the inserted row. - Response — the envelope is written; the transaction commits.
Carol’s carol-token belongs to org-globex, so the same payload from her is
rejected before it reaches the DB step:
curl 'localhost:8080/api/projects?filter=organization_id:eq:org-acme' \
-H 'Authorization: Bearer carol-token'
# → list filtered to org-globex only; the requested filter is ignored
What this example showed
- Relations declared with both the convention (
OrganizationID) and the explicitrelation:Ownerform. WithDeletedAton every audited model.ctx.Authpopulated by an Auth middleware, then read by Service middleware to scope queries.ctx.Query.Filtersmodified to enforce a tenant invariant.maniflex.WithTransactionplusctx.LockForUpdatefor a check-and-act write.- Custom error codes (
TENANT_MISMATCH,PROJECT_LIMIT) emitted withctx.Abort.
Where to go next
The next section covers the ready-made middleware that ships with maniflex — the production-quality versions of the auth and validation helpers sketched here.
- Middleware Catalogue — JWT auth, password hashing, unique validation, audit logging, and more.
- Querying — the full filter, sort, and
includegrammar used in this example.
Middleware Catalogue
The catalogue is a set of ready-made middleware packages that cover the common
needs of every production API — authentication, validation, password hashing,
audit logging, caching, CORS, and so on. Most return an ordinary
maniflex.MiddlewareFunc you register on the appropriate pipeline step, with
the same scoping options as any other middleware. CORS returns
maniflex.HTTPMiddleware because browser preflight must run before Auth.
The packages live under maniflex/middleware/. Most are part of the root module,
so they need no extra require to use. Only the two with heavy third-party
dependencies are broken out into their own Go modules — middleware/db/redis
(the Redis rate-limit/cache backend) and middleware/service/bcrypt (password
hashing) — so a project pulls those dependencies in only when it uses them:
| Package | Step | What it ships |
|---|---|---|
middleware/auth | Auth | JWT, API key, role gates, public-read helpers |
middleware/body | Deserialize / Validate | body size limits, unknown-field stripping, type coercion |
middleware/idempotency | Deserialize | idempotency-key replay for safely retried POSTs |
middleware/validate | Validate | uniqueness, regex, cross-field rules, numeric precision, date ranges, conditional required |
middleware/workflow | Validate | state-machine transitions with role-gated guards |
middleware/service | Service / DB-After | password hashing, slugify, derived fields |
middleware/db | DB | tenancy, forced filters, rate limiting, audit log, cache invalidation |
middleware/response | HTTP router / Response | CORS, caching, transforms, redaction, envelopes, metrics |
middleware/openapi | OpenAPI.Generate | security schemes, servers, titles, custom extensions |
How to use the catalogue
Import the package you need and register the returned middleware on the matching pipeline step:
import (
"github.com/xaleel/maniflex/middleware/auth"
"github.com/xaleel/maniflex/middleware/service"
)
server.Pipeline.Auth.Register(
auth.JWTAuth("my-signing-secret", auth.JWTOptions{Issuer: "my-app"}),
)
server.Pipeline.Service.Register(
service.HashField("password", svcbcrypt.Hasher()), // svcbcrypt = middleware/service/bcrypt
maniflex.ForModel("User"),
)
Each middleware factory returns a maniflex.MiddlewareFunc, so the standard
options — ForModel, ForOperation, AtPosition, WithName — apply
verbatim.
Composition
Catalogue middleware is designed to be composable. The expected stack for a typical REST API is roughly:
- Auth —
JWTAuthorAPIKeyAuthpopulatesctx.Auth;RequireRolegates sensitive operations. - Body —
MaxBodySizeandStripUnknownFieldsshape input early. - Validate — built-in tag rules plus
UniqueFieldand friends. - Service —
HashField,SetField,SlugifyField, then any custom business logic, then themaniflex/eventshelpers (events.Emit/events.Webhook/events.SendEmail) on the After side. - DB —
TenancyorForceFilterenforces row-level scoping;AuditLogandInvalidaterun After. - Response —
Cache,RedactField, thenLogging/Metricson the After side.
Install CORSHeaders separately in Config.HTTPMiddlewares; it must wrap the
router rather than wait for the Response step.
Mix and match freely; nothing in the catalogue is required.
Writing your own
The catalogue is just an applied form of Writing Middleware.
If a built-in does not match your needs, write your own — the contract is the
same func(ctx *maniflex.ServerContext, next func() error) error signature.
Auth Middleware
The maniflex/middleware/auth package supplies authentication and authorisation
middleware. Each function returns a maniflex.MiddlewareFunc that either populates
ctx.Auth on success or aborts with 401/403 on failure. Most register on the
Auth step; the exceptions are called out where they occur — Enforce
(attribute policies) runs on the DB step so the record is available, and
ReadAudit runs on the Response step after a 2xx.
JWTAuth
Verifies a bearer JWT and populates ctx.Auth from its claims.
import "github.com/xaleel/maniflex/middleware/auth"
server.Pipeline.Auth.Register(
auth.JWTAuth("my-signing-secret", auth.JWTOptions{
Issuer: "my-app",
Audience: "api",
TenantClaim: "org_id", // copied into AuthInfo.TenantID
ScopesClaim: "scope", // copied into AuthInfo.Scopes
}),
)
Supports HMAC (HS256/384/512) when the secret is a string and asymmetric
algorithms (RS256/384/512) when JWTOptions.PublicKey is set — useful with
external identity providers (Auth0, Okta, Cognito, etc.). AuthMethod on
ctx.Auth is set to "jwt".
Where the token is read from
By default the token comes from Authorization, which must carry the
Bearer scheme — a bare token there is malformed, not merely unadorned. Point
JWTOptions.Header at another header to read it from there instead:
auth.JWTAuth(secret, auth.JWTOptions{Header: "X-Auth-Token"})
A custom header accepts the token with or without the Bearer prefix, so a
client that sends it out of habit is not punished for it. The scheme name is
matched case-insensitively on both, per RFC 7235 — bearer <token> is valid.
Tokens must carry an exp claim: one with no expiry is rejected
(401 TOKEN_MISSING_EXPIRY), since it would otherwise be valid forever. Set
JWTOptions.AllowNoExpiry to accept non-expiring tokens from issuers that
deliberately mint them. On the HMAC path the signing secret must be non-empty (an
empty secret panics at startup) and should be at least 32 bytes — a shorter
secret is allowed but logs a warning.
JWKSAuth
Verifies asymmetric JWTs against a rotating JWK Set published by an identity
provider (e.g. an issuer’s /.well-known/jwks.json), instead of pinning a single
static key. Signing keys are fetched, cached, and selected by the token header’s
kid; an unknown kid triggers a rate-limited refetch, so a rotated key is
picked up without a redeploy. RSA (RS256/384/512) and EC (ES256/384/512) are
supported.
server.Pipeline.Auth.Register(auth.JWKSAuth(
"https://issuer.example.com/.well-known/jwks.json",
auth.JWTOptions{Issuer: "https://issuer.example.com", Audience: "api"},
))
All JWTOptions (Issuer, Audience, claim mappings, ClockSkew) apply exactly
as with JWTAuth — reach for the static-key JWTAuth only when the key is fixed
or for offline tests. See
Auth & Security Hardening for the
production checklist.
The JWKS URL must be https://
The JWK Set is the entire root of trust here. There is no shared secret: a
token is accepted if its signature verifies against a key from that URL, and
Issuer and Audience are claims inside the token, checked only once the
signature already has. Anyone who can control the bytes that URL returns can
therefore serve their own public key and mint tokens for any identity they like.
A plaintext URL logs a warning at construction. Two things make a moment of interception last much longer than the moment:
- a fetch replaces the whole key map and caches it for an hour, so one intercepted response buys an hour with nobody on the wire;
- when a later refresh fails, a cached key of any age is still used — so breaking the endpoint afterwards keeps an injected key alive indefinitely.
http://localhost and other loopback addresses are exempt and log nothing: a
local Keycloak or dex is an ordinary development setup and there is no wire to
intercept. A private LAN address such as 192.168.1.10 is not exempt — it is
still another host, reached over a network someone may be sitting on.
Redirects are policed as well as the configured URL. Go’s default HTTP client
follows https:// → http:// without complaint, so an issuer that redirects
could move key material onto plaintext while your configured https:// URL
looks untouched. Such a redirect is refused and the fetch fails.
APIKeyAuth
Validates a static API key from a request header. Each entry maps one key to
the AuthInfo it grants.
server.Pipeline.Auth.Register(auth.APIKeyAuth("X-API-Key",
auth.APIKeyEntry{Key: "svc-abc", Auth: maniflex.AuthInfo{
UserID: "svc-1", Roles: []string{"admin"},
}},
auth.APIKeyEntry{Key: "svc-xyz", Auth: maniflex.AuthInfo{
UserID: "svc-2", Roles: []string{"reader"},
}},
))
AuthMethod on ctx.Auth is set to "api_key". Combine with JWTAuth on
the same step to accept either credential — the first match wins.
Keys are indexed by their SHA-256 digest rather than by the key itself, so
lookup timing is a function of the digest and not of the secret, and the raw
keys are not retained in the index. Lookup stays O(1) — comparing every entry
with subtle.ConstantTimeCompare would close the same gap but scan every
configured key on every authenticated request.
These are static keys held in memory: they are as good as the deployment’s
secret handling, they cannot be revoked without a restart, and they do not
expire. For rotation, revocation, or per-user credentials, use JWTAuth with an
auth.Revoker.
RequireRole
Rejects the request unless ctx.Auth.Roles contains the named role. Typically
registered with ForModel / ForOperation to scope where the check applies.
server.Pipeline.Auth.Register(
auth.RequireRole("admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpDelete),
)
Both failure cases return 403 FORBIDDEN: an anonymous request (ctx.Auth == nil) and an authenticated request lacking the role are treated alike. This
differs from RequireOwner, which answers 401 to an anonymous caller.
RequireScope and RequireAnyScope
Reject the request unless ctx.Auth.Scopes carries the OAuth2 grants the route
needs. JWTAuth fills that slice from JWTOptions.ScopesClaim (default
"scope"), accepting both the JSON array form and RFC 6749’s space-delimited
string.
RequireScope requires every listed scope:
server.Pipeline.Auth.Register(
auth.RequireScope("posts:read", "posts:write"),
maniflex.ForModel("Post"), maniflex.ForOperation(maniflex.OpUpdate),
)
That is the opposite of RequireRole, deliberately. A role names who the caller
is, so holding one of several is the usual question; a scope names a grant that
was issued, so an endpoint that both reads and writes needs both grants — not
either. Where several grants really are each sufficient, ask for that explicitly:
server.Pipeline.Auth.Register(auth.RequireAnyScope("posts:write", "admin"))
The refusal names only the scopes the caller is missing, so a client granted three of four is told which one to go and ask for.
Scopes match exactly. A read:* grant does not satisfy read:posts — the
framework cannot know whether :, /, or nothing at all delimits your issuer’s
hierarchy. Express one with Enforce and a Policy.
Both constructors panic when given no scopes. Under all-of semantics an empty requirement is vacuously satisfied, so such a guard would admit every request while reading as though it protected the route.
Like RequireRole, both answer 403 FORBIDDEN to an anonymous caller and to an
authenticated one lacking the grant.
RequireOwner
Enforces that the authenticated user owns the record being read or written. On
create it stamps ownerField = ctx.Auth.UserID automatically; on read, update,
and delete it fetches the target and compares its ownerField to the caller —
answering 404 (not 403, so the endpoint never reveals that a record it doesn’t
own exists). ownerField may be given as the JSON or the DB column name. Callers
holding any role in adminRoles bypass the check.
server.Pipeline.Auth.Register(auth.RequireOwner("user_id", "admin"))
RequireOwner scopes single-resource operations only — a collection GET
still returns every row. Constrain list reads with db.ForceFilter or
db.Tenancy on the DB step.
Enforce — attribute-based policies
RequireRole and RequireOwner answer who is calling; Enforce answers may
this principal touch this record, evaluating a Policy against the affected
row’s fields. A Policy is a plain function:
type Policy func(ctx *maniflex.ServerContext, resource map[string]any) (allow bool, err error)
Return (false, nil) for a 403 FORBIDDEN; a non-nil error becomes a 500.
Register on the DB step (not Auth), so the row is available:
sameTenant := func(ctx *maniflex.ServerContext, r map[string]any) (bool, error) {
return r["tenant_id"] == ctx.Auth.TenantID, nil
}
server.Pipeline.DB.Register(auth.Enforce(sameTenant), maniflex.ForModel("Patient"))
Which record the policy sees depends on the operation:
| Operation | Record checked | When |
|---|---|---|
OpCreate | the proposed body (ctx.ParsedBody) | before the insert |
OpUpdate / OpDelete | the current stored record | fetched before the write |
OpRead | the fetched record | after the DB step |
OpList | each row in turn | after the DB step; denied rows are dropped |
Compose policies with AllOf, AnyOf, and Not:
auth.Enforce(auth.AllOf(sameTenant, auth.AnyOf(isOwner, auth.Not(isArchived))))
For lists the policy runs per row after the query, so pagination totals
reflect the pre-filter count. When the rule can be expressed as a WHERE clause,
prefer db.ForceFilter — it scopes in SQL, keeps totals accurate, and
never fetches rows the caller can’t see.
AllowAnonymous
Marks the routes it is registered for as tolerating a request that carries no
credential at all. Register it before the authenticator and scope it with the
usual filters; JWTAuth and JWKSAuth honour it.
server.Pipeline.Auth.Register(auth.AllowAnonymous(),
maniflex.ForModel("Post"),
maniflex.ForOperation(maniflex.OpList, maniflex.OpRead),
)
server.Pipeline.Auth.Register(auth.JWTAuth(secret)) // everything else, unchanged
Exactly one thing changes: a request with no token is served with ctx.Auth left
nil instead of answering 401. A credential that was presented and failed —
expired, wrongly signed, revoked, malformed, or missing its Bearer scheme — is
still 401 here, as it is everywhere else. Degrading a bad token to anonymous
would turn an expired session into a silent change of permissions.
Two consequences worth planning around:
- The exemption is what you enumerate, not the coverage. The authenticator
stays registered globally, so a model added next month is protected by default.
Forgetting an entry in the exemption list produces a
401on the first request rather than an endpoint that quietly opens. - A token that is presented still authenticates. This is what scoping
JWTAuthaway from a route cannot do — there the middleware never runs, so even a valid token leavesctx.Authnil. UseAllowAnonymouswhen a route shows more to a signed-in visitor than to a stranger (published posts to everyone, drafts to their author).
ctx.Auth stays nil for the anonymous caller rather than being filled with a
blank principal, so every ctx.Auth == nil test keeps meaning what it means.
Ordering is by registration within a step: an AllowAnonymous registered after
the authenticator sets its marker too late to be read, which fails closed.
AllowPublicRead
A passthrough on OpRead and OpList; on every other operation it requires
ctx.Auth to be populated.
It does not, on its own, make reads reachable. An authenticator that aborts on
a missing credential has already answered 401 and returned without calling
next(), so an AllowPublicRead registered after it never runs — and registered
before it, the passthrough calls next() straight into the authenticator, which
aborts anyway. Pair it with AllowAnonymous:
server.Pipeline.Auth.Register(auth.AllowAnonymous(),
maniflex.ForOperation(maniflex.OpList, maniflex.OpRead),
)
server.Pipeline.Auth.Register(auth.JWTAuth("..."))
server.Pipeline.Auth.Register(auth.AllowPublicRead())
Here it is the belt to AllowAnonymous’s braces: one registration, covering every
model, re-asserting that nothing without a principal may write — a check no
per-route exemption list can drift away from.
BlockOperation
Refuses the listed operations outright, regardless of identity. Useful for making a model effectively read-only at the HTTP layer.
server.Pipeline.Auth.Register(
auth.BlockOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.ForModel("AuditLog"),
)
The model’s routes remain mounted but always return 405 METHOD_NOT_ALLOWED.
Token revocation and logout
A JWT is valid until its exp and nothing the server does can take it back.
That is fine until a user logs out, changes their password, or has their account
compromised — at which point “valid until it expires” is exactly wrong. Set
JWTOptions.Revoker to give the server its say back, at the cost of a lookup
per request.
rev := auth.NewMemoryRevoker()
server.Pipeline.Auth.Register(auth.JWTAuth(secret, auth.JWTOptions{Revoker: rev}))
server.Action(auth.Logout(rev, "")) // POST /logout
server.Action(auth.LogoutAll(rev, "", 24*time.Hour)) // POST /logout-all
Logout revokes the token the caller authenticated with — logout on this device.
LogoutAll revokes every token belonging to the caller, including sessions whose
jti the server has never seen; it is the one to call after a password change.
Both respond 204, and both must be mounted behind the same auth middleware as
everything else (they read ctx.Auth).
The two granularities
| Call | Effect |
|---|---|
RevokeToken(ctx, jti, exp) | blocks one token; the entry may be dropped once exp passes, since the token is refused for being expired anyway |
RevokeUser(ctx, userID, cutoff, retainUntil) | blocks every token for the user issued before cutoff; keep the record until retainUntil, which must be past the exp of the longest-lived token you mint |
The per-user cutoff is what makes “log out everywhere” possible: the outstanding
jti values are unknown and usually unknowable, so a per-token blocklist alone
cannot express it. Logging in again works immediately — the cutoff kills tokens
issued before it, not the account.
What changes when it is on
- A
jtibecomes mandatory. A token without one cannot be revoked, so it is refused with401 TOKEN_NOT_REVOCABLErather than handed a permanent exemption from the blocklist. Mint a uniquejtiper token. - An
iatbecomes required for users who have a cutoff. A token that cannot be placed relative to the cutoff is refused. Tokens withoutiatkeep working for every user who has never calledLogoutAll. - A store outage refuses requests — see below.
Error codes:
| Code | Status | Meaning |
|---|---|---|
TOKEN_REVOKED | 401 | explicitly revoked; the client should discard it and log in again |
TOKEN_NOT_REVOCABLE | 401 | no jti, so it cannot be revoked — a minting bug, not a client error |
REVOCATION_UNAVAILABLE | 503 | the blocklist could not be consulted |
Failing closed
Every Revoker method returns an error, and the middleware refuses the request
when a lookup fails rather than reading “I could not check” as “not revoked”. A
blocklist that fails open silently un-revokes every logged-out token for the
duration of an outage — precisely when it matters most. This is why Revoker is
its own interface rather than a reuse of maniflex.CacheStore, whose Get
reports a miss and an outage identically.
The refusal is 503, not 401: the credential is fine, the server is not, and
answering 401 would push every healthy client into a re-login storm during an
incident.
Backends
NewMemoryRevoker() keeps the blocklist in-process. Its limitation is
structural: a second replica does not see a logout performed by the first, and a
restart loses every entry — which un-revokes every still-unexpired token. It
suits single-replica deployments, development, and tests.
Behind more than one replica, use a shared store. Two ship.
Redis — server-side key expiry, so nothing ever has to be swept:
import authredis "github.com/xaleel/maniflex/middleware/auth/redis"
rev := authredis.NewRevoker(redisClient, "myapp:revoked")
SQL — the database you already run, with no second piece of infrastructure
and no extra dependency (the package imports nothing but database/sql):
import authsql "github.com/xaleel/maniflex/middleware/auth/sql"
if err := authsql.Migrate(ctx, db, "postgres"); err != nil { // or "sqlite"
log.Fatal(err)
}
rev := authsql.NewRevoker(db)
Migrate creates revoked_token and revoked_user and is safe to run on every
boot and from every replica — every statement is IF NOT EXISTS, and none
rewrites an existing column. Use WithTablePrefix("auth_") if those names are
taken; pass the same option to both calls.
SQL has no TTL, so expired rows are swept rather than expiring themselves. This
never affects correctness — both reads filter on the deadline, so a row the sweep
has not reached yet stops being honoured at exactly the right moment — it only
bounds the table. The Revoker sweeps every 128 writes on its own; call
Prune(ctx) to do it on a schedule and see the error, or WithPruneEvery(0) to
turn the automatic sweep off and take over entirely:
tokens, users, err := rev.Prune(ctx)
One behavioural note when comparing the two: RevokeUser moves a cutoff forward
in the INSERT … ON CONFLICT itself, so concurrent revocations cannot resurrect
tokens by landing out of order. The Redis backend reads then writes and documents
that race as benign.
Or implement auth.Revoker yourself over any store — four methods, and the
interface is in terms of context.Context and time.Time only.
VerifyToken hook
JWTOptions.VerifyToken is the final say on a token that has passed every other
check — signature, registered claims, and revocation. Return an error to refuse
the request with 401 INVALID_TOKEN carrying that message.
auth.JWTAuth(secret, auth.JWTOptions{
VerifyToken: func(ctx *maniflex.ServerContext, claims map[string]any, info *maniflex.AuthInfo) error {
if claims["tier"] != "paid" {
return errors.New("subscription required")
}
info.Roles = append(info.Roles, "subscriber") // may enrich the principal
return nil
},
})
Use it for issuer-specific rules the framework does not model: a required custom
claim, a tenant allowlist, a per-user check against your own store. For
revocation specifically, prefer Revoker — it is the same hook point with the
blocklist and its failure semantics already written.
CSRF
Protects unsafe HTTP methods (POST/PUT/PATCH/DELETE) against cross-site
request forgery. Two strategies, selected by CSRFOptions.Mode:
CSRFDoubleSubmit(default) — issues a random token in a non-HttpOnly cookie on safe requests and requires the client to echo it in a header on unsafe ones.CSRFSignedToken— derives the expected token asHMAC(SessionID, Secret); stateless, no cookie issued. Register it after the JWT step soctx.Auth.SessionIDis populated (e.g. from the token’sjticlaim).
server.Pipeline.Auth.Register(auth.CSRF(auth.CSRFOptions{
AllowedOrigins: []string{"https://admin.example.com", "*.example.com"},
Secure: true,
}))
Bearer-authenticated requests are exempt by default — a bearer token read
from JS is not an ambient credential, so it isn’t CSRF-vulnerable. Set
EnforceBearer: true only if your bearer flow also rides browser-managed cookies.
The optional AllowedOrigins allowlist (exact origins or *.host wildcards) is
checked first on unsafe methods. Failures abort with a 403 carrying one of
CSRF_ORIGIN_REJECTED, CSRF_TOKEN_MISSING, CSRF_COOKIE_MISSING,
CSRF_NO_SESSION, or CSRF_TOKEN_MISMATCH.
From a login Action, hand the token to the SPA with auth.IssueCSRFCookie(w, opts) (double-submit) or auth.SignedCSRFToken(sessionID, secret) (signed mode).
The admin panel does not use this middleware. It ships its own always-on double-submit check over its own forms — see Admin Panel → CSRF protection. Registering
auth.CSRFneither configures nor disables it, and leavingauth.CSRFoff does not leave the panel unprotected.
ReadAudit
Writes a structured audit record after every successful read or list — the
read-side counterpart to db.AuditLog, for data where who looked is itself the
compliance control (clinical, financial). Implement a ReadAuditSink and register
ReadAudit at the After position on the Response step, so it fires only on
a 2xx:
server.Pipeline.Response.Register(
auth.ReadAudit(mySink),
maniflex.ForModel("Patient", "LabResult"),
maniflex.ForOperation(maniflex.OpRead, maniflex.OpList),
maniflex.AtPosition(maniflex.After),
)
Each ReadAuditRecord carries the actor, roles, tenant, session, request/trace
IDs, client IP, and either the accessed RecordID (read) or the RecordCount
(list). Writes are fire-and-forget — a goroutine with a 5 s timeout, so a slow
sink never delays the response — and sink errors are discarded, so back a lossless
requirement with a durable queue.
Scoping patterns
The middleware in this package combines with ForModel / ForOperation to
build per-route policy without writing custom Auth code:
// Public reads, JWT writes, admin-only deletes
server.Pipeline.Auth.Register(auth.AllowAnonymous(),
maniflex.ForOperation(maniflex.OpList, maniflex.OpRead))
server.Pipeline.Auth.Register(auth.JWTAuth("..."))
server.Pipeline.Auth.Register(auth.RequireRole("admin"),
maniflex.ForOperation(maniflex.OpDelete))
Note which registration carries the filter. Scoping the authenticator to the
write operations would work today and fail later: ForModel/ForOperation are
inclusion-only, so anything they do not name is covered by no auth registration at
all. Scoping the exemption instead leaves the authenticator global, so the
default for a route nobody listed is refusal.
Body Middleware
The maniflex/middleware/body package shapes the request body during the
Deserialize and Validate steps — before the framework’s tag rules run.
MaxBodySize
Overrides the default 4 MB body limit for the current request. Register on the Deserialize step, scoped to the model that needs the larger limit:
import "github.com/xaleel/maniflex/middleware/body"
server.Pipeline.Deserialize.Register(
body.MaxBodySize(16 << 20), // 16 MB
maniflex.ForModel("Article"),
)
Requests over the limit are aborted with 413 BODY_TOO_LARGE before the JSON
parser runs — as is any request over the 4 MB default when this middleware is
not registered. An oversized body is never truncated to fit.
StripUnknownFields
Removes keys from ctx.ParsedBody that do not correspond to a model field.
Register on the Validate step (or Deserialize After-position) so the cleanup
happens before tag validation and the DB step:
server.Pipeline.Validate.Register(body.StripUnknownFields())
The default behaviour is to accept and silently ignore unknown fields. Use this middleware to enforce a stricter contract when desired.
CoerceTypes
Coerces string values in ctx.ParsedBody into the Go type declared on the
model — "42" → 42 (int), "3.14" → 3.14 (float64), "true" → true
(bool). Helps when the client sends form-encoded or query-string-shaped payloads.
Only string→int/float64/bool is performed; other types are left as-is.
server.Pipeline.Validate.Register(body.CoerceTypes())
Coercion happens before the framework’s min / max / enum checks, so
numeric ranges and enums work against the coerced values.
Validate Middleware
The maniflex/middleware/validate package supplies validators that go beyond what
mfx: tags can express. Each one runs on the Validate step alongside the
built-in tag enforcement, and most abort with 422 VALIDATION_ERROR on rejection.
The exceptions:
RestrictField/FieldRoleanswer403 FIELD_FORBIDDEN, because they gate on who is asking, not on whether the value is valid.RequireLocaleanswers422 MISSING_LOCALE— still a 422, but with its own code so a missing translation is distinguishable from a plain validation miss.UniqueFieldanswers409 CONFLICT, not a422— a duplicate is a conflict with existing state rather than malformed input, and this is the same answer the database’s own constraint gives for the identical error. If the underlying count query itself fails it answers500 UNIQUE_CHECK_FAILEDrather than letting a duplicate slip through.
UniqueField
Rejects a create or update whose value collides with an existing row.
import "github.com/xaleel/maniflex/middleware/validate"
server.Pipeline.Validate.Register(
validate.UniqueField(sqlDB, maniflex.Postgres, "email"),
maniflex.ForModel("User"),
)
The middleware runs a count query against the underlying database before the DB
step. It answers the same 409 CONFLICT the mfx:"unique" constraint produces —
identical status, code, message and details — so a client handles one response
whichever mechanism caught the duplicate. What it buys is when: the collision is
reported before the write is attempted, and named by JSON field.
It is not a substitute for mfx:"unique". Count-then-write is not atomic, so two
concurrent requests can both pass the count; the database constraint remains the
actual guarantee.
The driver argument selects the placeholder dialect (maniflex.Postgres →
$N, maniflex.SQLite → ?) and must match the driver used to open the
adapter. The third argument is the JSON field name; it is resolved to the
underlying DB column via ctx.Model.FieldByJSONName.
RegexField
Validates that a string field matches a regular expression:
server.Pipeline.Validate.Register(
validate.RegexField("phone", `^\+?[0-9]{7,15}$`),
maniflex.ForModel("Contact"),
)
A non-matching value aborts with 422 and the field name in details.
ForbiddenValues
Rejects writes that contain any of the listed values for a field. Use it for defence-in-depth on enum-like fields where the mfx enum tag would still allow a privileged value:
server.Pipeline.Validate.Register(
validate.ForbiddenValues("role", "superadmin", "root"),
maniflex.ForModel("User"),
)
RequireAtLeastOne
Ensures at least one of the named fields is present in the request body. Most
useful on OpUpdate, where every field is otherwise optional:
server.Pipeline.Validate.Register(
validate.RequireAtLeastOne("name", "email"),
maniflex.ForOperation(maniflex.OpUpdate),
)
RequireLocale
Ensures a localised (LocaleString) field carries a non-empty value for each of
the required locale keys. Use it for translatable fields where certain languages
are mandatory:
server.Pipeline.Validate.Register(
validate.RequireLocale("name", "en", "ar"),
maniflex.ForModel("Department"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
The field value may arrive as a JSON object (map[string]any) or a Go-native
map[string]string. A locale key that is absent, null, or empty fails the
check. When any required locale is missing the request is aborted with
422 MISSING_LOCALE, and details lists every offending locale keyed by field.
If the field is absent from the body, null, or not a locale map at all, the
rule passes silently — pair it with required when presence itself is mandatory.
NumericPrecision
Enforces decimal precision and scale on a numeric field. The check is a
string-parse, so it works regardless of how the column is stored
(INTEGER, NUMERIC(p,s), TEXT, custom SQLTyper):
server.Pipeline.Validate.Register(
validate.NumericPrecision("amount", 19, 4), // up to 19 total digits, max 4 after the point
maniflex.ForModel("Invoice"),
)
precision— maximum total significant digits (integer + fractional). Pass0to disable.scale— maximum digits after the decimal point. Pass0to disable.
Sign (+/-), leading zeros, and trailing fractional zeros do not count
toward either limit. Scientific notation (1e3) is rejected because its
implied precision is ambiguous — supply financial values in plain form.
Absent and null values are skipped; pair with required when presence
matters.
CrossFieldValidate
A general-purpose escape hatch for rules that span multiple fields:
server.Pipeline.Validate.Register(
validate.CrossFieldValidate(func(body map[string]any) error {
if body["status"] == "scheduled" && body["scheduled_at"] == nil {
return fmt.Errorf("scheduled_at is required when status is scheduled")
}
return nil
}),
maniflex.ForModel("Post"),
)
The returned error becomes the message of a 422 VALIDATION_ERROR response.
DateRange
Ensures an end field is not before a start field. Accepts RFC3339 timestamps
("2026-05-01T08:00:00Z") and YYYY-MM-DD date strings. If either field is
absent, null, or unparseable, the rule passes silently — pair with required
or another rule when presence matters.
server.Pipeline.Validate.Register(
validate.DateRange("start_date", "end_date"),
maniflex.ForModel("Booking"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
Equal dates are accepted (start == end). A 422 VALIDATION_ERROR is returned
with the end field named in details when the end precedes the start.
RequireWhen
Makes a field required only when other fields satisfy all listed conditions.
Each condition is a "field:op:value" string; op is one of eq, ne, gt,
gte, lt, lte. Multiple conditions are ANDed. Invalid syntax panics at
startup so misconfiguration is caught before the first request.
// Require rejection_reason whenever status is "rejected"
server.Pipeline.Validate.Register(
validate.RequireWhen("rejection_reason", "status:eq:rejected"),
maniflex.ForModel("Claim"),
)
// Require shipping_address only for physical orders with priority >= 3
server.Pipeline.Validate.Register(
validate.RequireWhen("shipping_address", "order_type:eq:physical", "priority:gte:3"),
maniflex.ForModel("Order"),
)
When the conditions are met but the target field is absent, null, or empty
string, the request is rejected with 422 VALIDATION_ERROR. When the
conditions are not all met, the rule passes regardless of the target field’s
value — it does not prevent the field from being supplied.
Numeric comparisons (gt, gte, lt, lte) coerce both the body value and
the condition value to float64. Non-numeric body values cause the condition to
evaluate as false (the target field stays optional).
FieldRole / RestrictField
Gate a field on who is writing it. These are the write-side twin of
response.RedactField — the same predicate shape, on
the opposite step.
// Only a superuser may write status; everyone else may write the rest of the row
server.Pipeline.Validate.Register(
validate.FieldRole("subscription_expires_at", "superuser"),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
RestrictField takes a predicate instead of a role list, for gates roles cannot
express (ownership, tenant, plan tier):
server.Pipeline.Validate.Register(
validate.RestrictField("document_quota_bytes",
func(ctx *maniflex.ServerContext) bool {
return ctx.HasRole("superuser") || isBillingAdmin(ctx)
}),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
FieldRole(field, roles...) is exactly RestrictField with a ctx.HasRole
predicate (OR-semantics). With no roles passed it rejects every write of the
field, so an accidentally empty list fails closed — matching auth.RequireRole.
Declare the field so a typo can’t fail open
Misspell the field name and the gate is inert: it watches for a body key nothing
sends, the real field keeps its name, and nothing gates it. Add
maniflex.RequiresField and that becomes a startup failure:
server.Pipeline.Validate.Register(
validate.FieldRole("subscription_expires_at", "superuser"),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
maniflex.RequiresField("subscription_expires_at"), // ← checked at startup
)
The declared name is checked against the model, so writing it twice catches a misspelling rather than duplicating one. Do this on every gate that protects a field you care about.
Without it, a mismatch is only a warning on the first request that reaches the model — no help for an endpoint nobody exercises, and indistinguishable from a gate deliberately registered across models where only some carry the field. See Startup Validation.
Why this isn’t readonly
The mfx: write controls are static: readonly, immutable, hidden apply to
every caller identically. They cover “no client ever writes this”. They cannot
express “only a superuser may set subscription_expires_at, while the owner
writes the rest of their own row” — which, without this, costs a separate
endpoint per privileged field.
Refused, not stripped
A caller without permission gets 403 FIELD_FORBIDDEN naming the field.
This deliberately differs from readonly, which silently drops the field:
readonly | FieldRole | |
|---|---|---|
| meaning | nobody writes this, ever | someone writes this — not you |
| client sending it | confused about the schema | making a privilege error |
| on violation | field dropped, 200 | whole write refused, 403 |
Answering 200 to a write that did not happen, echoing the old value back, is
indistinguishable from success. The write is refused whole — a mixed
PATCH {"title": …, "status": …} from a non-holder changes neither field.
Details
- Only a field present in the body is gated. A PATCH that does not mention it
passes untouched. An explicit
nullis a write, and is gated. - Create and update both, when you scope it with
ForOperation. fieldis the JSON name.- Scope it with
maniflex.ForModel. Registered without one it applies to every model, and a gate naming a field the model does not have can never fire. It warns once per model in that case, since a typo would otherwise leave the real field ungated in silence.
Workflow Middleware
maniflex/middleware/workflow enforces a state-machine on a model’s status
field. Declare the permitted transitions once; the middleware rejects writes
that would move a record between states the workflow does not allow, or that
fail a guard (e.g. role check).
import "github.com/xaleel/maniflex/middleware/workflow"
sm := workflow.New("status",
workflow.Allow("draft", "submitted"),
workflow.Allow("submitted", "approved", workflow.RequireRole("manager")),
workflow.Allow("submitted", "rejected", workflow.RequireRole("manager")),
workflow.Allow("approved", "paid", workflow.RequireRole("finance")),
workflow.AllowAny(workflow.RequireRole("admin")), // admin escape hatch
workflow.AllowInitial("draft", "submitted"), // legal seed states on Create
)
server.Pipeline.Validate.Register(
sm.Middleware(),
maniflex.ForModel("Invoice"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
How it runs
The middleware lives on the Validate step. (A machine that declares
OnTransition hooks registers on the DB step instead — see
Transition hooks.)
- On
OpCreate— ifAllowInitialis declared, the value of the chosen field must be in the set; otherwise any initial value passes. No guards apply on Create (the Create itself is its own authorisation surface). - On
OpUpdate— if the body does not include the status field, the middleware is a no-op. Otherwise it reads the current record viactx.GetModel(modelName).Read(id)(so reads participate inctx.Txwhen active), extractsfrom, compares to the body’sto, and:- same-state writes (
from == to) pass silently; - the first matching rule wins; its guards run in order;
- the first guard error rejects the transition with
422 INVALID_TRANSITION.
- same-state writes (
A PATCH that triggers the read also costs one round-trip. Stash the loaded
record on ctx.Set if you need it again later in the request.
Rules
| Option | Effect |
|---|---|
Allow(from, to, guards...) | permit from → to; both sides may be "*" for “any” |
AllowAny(guards...) | shorthand for Allow("*", "*", guards...) |
AllowInitial(states...) | restrict Create to the listed initial states |
OnTransition(from, to, fn) | run fn inside the write’s transaction when from → to is taken |
Rule matching is a linear scan in declaration order — write narrow rules before broad ones if you want them to take precedence.
A machine with no Allow rules permits nothing — every transition is
rejected. If you want hooks without restricting transitions, say so explicitly
with AllowAny().
Guards
type Guard interface {
Check(ctx *maniflex.ServerContext, from, to string) error
}
RequireRole(roles ...string)— pass if the caller holds any one of the listed roles. OR-semantics; passing zero roles always rejects (a defensive choice against accidentally unguarded “require” rules).GuardFunc— adapt anyfunc(ctx, from, to) errorto the interface.
A non-nil guard error becomes the response message:
{
"error": {
"code": "INVALID_TRANSITION",
"message": "role [manager] required for transition \"submitted\" → \"approved\"",
"details": [{"field": "status", "from": "submitted", "to": "approved"}]
},
"status": 422
}
Transition hooks (OnTransition)
OnTransition(from, to, fn) attaches a side effect to a transition. The hook
runs inside the write’s transaction, so it commits with the transition or
not at all:
sm := workflow.New("status",
workflow.Allow("pending", "confirmed", workflow.RequireRole("store_owner")),
workflow.Allow("confirmed", "pending", workflow.RequireRole("store_owner")),
workflow.AllowAny(workflow.RequireRole("admin")),
workflow.OnTransition("pending", "confirmed", applyStoreCredit),
workflow.OnTransition("confirmed", "pending", reverseStoreCredit),
workflow.OnTransition("*", "delivered", emitReviewRequested),
)
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpUpdate),
)
server.Pipeline.DB.Register(
sm.Hooks(),
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpUpdate),
)
type Hook func(ctx *maniflex.ServerContext, from, to string) error
Hooks() replaces Middleware() — it does not supplement it
A machine with hooks registers sm.Hooks() on the DB step, and that is the
only registration it needs: Hooks() enforces the same rules and guards
Middleware() does, and more strictly. Declaring a hook makes Middleware()
panic rather than let it be registered on Validate where the hooks could never
fire.
| Machine | Register | Transaction |
|---|---|---|
| guards only | sm.Middleware() on Validate | not required |
any OnTransition | sm.Hooks() on DB | required — WithTransaction on Service |
The split exists because the two cannot run at the same point. A guard must run
before anything is written; a hook must run inside the write’s transaction — and
that transaction does not exist until WithTransaction opens it on the Service
step, after Validate.
That is also why Hooks() re-reads from rather than trusting the Validate
step’s verdict. The Validate-step read takes no lock, so two concurrent PATCHes
can both observe from="pending", both pass, and both apply the store
credit. Hooks() re-reads from through ctx.LockForUpdate inside the
transaction: the loser waits, then sees confirmed and is either a no-op
(same-state) or re-checked as the transition it has really become. Because the
re-read can name a different rule than the one that passed on the stale value,
the matching rule’s guards re-run too.
With no active transaction, Hooks() refuses the request with
500 WORKFLOW_NO_TX rather than take no lock and silently reintroduce the race
— the same fail-loudly stance as mfx:"lock_scope".
Semantics
- Fire-all-matching, in declaration order — unlike
Allow, which is first-match-wins.OnTransition("pending", "confirmed", …)andOnTransition("*", "delivered", …)are independent side effects, and a rule ordering that silenced one of them would be a bug rather than a policy. OpUpdateonly. A Create seeds an initial state (AllowInitialgoverns it); it is not a transition. Use a Service-step middleware for create-time side effects.- Same-state writes fire nothing — a no-op is not a transition.
- A returned error rolls the whole request back, transition included, and
answers
500 WORKFLOW_HOOK_ERROR. To choose your own status, callctx.Abort(…)and returnnil; the write still rolls back and your response is what the client sees. - The hook runs after the write lands, so a read through
ctx.Txsees the new state.
Status field type
Values are compared as strings via fmt.Sprintf("%v", v), matching
validate.ForbiddenValues. This covers string, int, and custom enum
types.
Service Middleware
The maniflex/middleware/service package supplies business-logic helpers for the
Service step — field transforms, derived values, and owner-scoping.
Side-effect helpers (events, webhooks, email) live in the separate
maniflex/events package, not here — see
Side effects below.
Field transforms
HashField
Replaces a plaintext field with its hash before the DB step, using a Hasher
you supply. The bcrypt hasher lives in the satellite package
maniflex/middleware/service/bcrypt (kept separate so the core has no bcrypt
dependency):
import (
"github.com/xaleel/maniflex/middleware/service"
svcbcrypt "github.com/xaleel/maniflex/middleware/service/bcrypt"
)
server.Pipeline.Service.Register(
service.HashField("password", svcbcrypt.Hasher()),
maniflex.ForModel("User"),
)
svcbcrypt.Hasher() takes an optional cost (svcbcrypt.Hasher(12)); the default
is suitable for production. A Hasher is just func(plaintext string) (string, error), so you can supply argon2 or any other implementation.
SlugifyField
Derives a slug field from a source field on create:
server.Pipeline.Service.Register(
service.SlugifyField("title", "slug"),
maniflex.ForModel("Post"), maniflex.ForOperation(maniflex.OpCreate),
)
Punctuation is stripped, spaces become hyphens, and the result is lowercased.
SetField
Sets a field on every create or update based on context — typically pulling
identity from ctx.Auth:
server.Pipeline.Service.Register(
service.SetField("user_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.UserID
}),
maniflex.ForOperation(maniflex.OpCreate),
)
StripField
Removes a field from the request body (and the typed record) before the DB step. Useful for input-only confirmation fields that should never reach the database:
server.Pipeline.Service.Register(service.StripField("password_confirm"))
TimestampWhen
Sets a timestamp column when another field transitions to a specific value —
for example, recording published_at the moment status becomes
"published":
server.Pipeline.Service.Register(
service.TimestampWhen("published_at", "status", "published"),
maniflex.ForModel("Post"),
)
Timestamp
Unconditionally sets a timestamp column to the current time on every write it
runs for — use ForOperation to scope it (e.g. a last_seen_at touched on
update):
server.Pipeline.Service.Register(
service.Timestamp("last_seen_at"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpUpdate),
)
CopyField
Copies one field’s value into another before the DB step — for denormalising a value the client shouldn’t set directly:
server.Pipeline.Service.Register(
service.CopyField("email", "billing_email"),
maniflex.ForModel("Account"), maniflex.ForOperation(maniflex.OpCreate),
)
Authorisation
OwnerScope
Forces a user-id field to the authenticated caller on create. It is exactly
SetField("user_id", ctx.Auth.UserID) — an unconditional overwrite, so any value
the client sent for that field is replaced (not rejected):
server.Pipeline.Service.Register(
service.OwnerScope("user_id"),
maniflex.ForOperation(maniflex.OpCreate),
)
Side effects
Side effects — events, webhooks, email — are not in this package. They live
in maniflex/events, and only events.Emit
is pipeline middleware; events.Webhook and events.SendEmail are event-bus
subscribers, not middleware, so they are wired with bus.Subscribe, not
Pipeline.Register.
events.Emit publishes a domain event on the DB step at maniflex.After,
after the write succeeds:
import "github.com/xaleel/maniflex/events"
server.Pipeline.DB.Register(
events.Emit(bus),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
Webhooks and email then react to those events as subscribers:
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: events.Webhook(events.WebhookConfig{URL: "https://hooks.example.com/orders", Secret: "whsec_…"}),
})
bus is any events.Bus — events/inproc for a single binary, or the Kafka,
NATS, RabbitMQ, and Redis adapters. See
Events & Background Jobs for the full API,
including the transactional outbox that makes events.Emit commit atomically
with the write (without it, a rolled-back transaction leaves an already-sent
side effect — outbound emails do not unsend themselves).
DB Middleware
The maniflex/middleware/db package wraps the DB step with row-level
scoping, request budgeting, post-write hooks, and result caching.
Row-level scoping
ForceFilter
Injects a filter on every list, read, update, and delete, regardless of what the client requested. Used to enforce invariants the client cannot override:
import "github.com/xaleel/maniflex/middleware/db"
server.Pipeline.DB.Register(
db.ForceFilter("org_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.Claims["org_id"]
}),
maniflex.ProvidesScope(),
)
On a list or a read the filter goes into the query. On an update or a delete
there is nowhere to put it — the adapter’s Update and Delete are keyed by id
alone — so the DB step reads the record back through the filter first and
answers 404 if it does not match, indistinguishably from a record that is
genuinely absent. The check and the write share one transaction, so the row
cannot leave scope in between.
On a create or an update the scope value is also stamped onto the row. An
equality on a plain column says two things at once — “only rows where field =
value” and “a row you create has field = value” — and both are honoured.
Without the stamp a create stored whatever the client sent in the scope column,
so the row landed outside its own author’s scope: invisible to them on their next
read, and, since that column is the client’s to send, plantable straight into
someone else’s scope. The stamp is skipped on update when the column is
immutable, which cannot change anyway.
A foreign key the write names is checked too. The stamp makes the row look
correctly owned whatever its foreign keys say, so POST {"order_id": "<another tenant's>"} used to land a child under their parent. The child is hidden from
that tenant on every read — see the include rule below — but the framework itself
still writes to the parent: a rollup recomputes
the parent’s denormalised column over every child naming it, so a planted row
moved a total in a row its author could neither read nor reach. So every
BelongsTo key a create or update sets is read through the scope first, and a
miss is the same 404 a scoped read of that parent gives.
Three shapes are deliberately left alone. A key the write doesn’t set can’t move
the row anywhere, and the row itself was already read back through the scope. A
key written empty is legal — the child carries the scope column itself, so a
child with no parent is perfectly describable (unlike ForceFilterVia below,
where the parent is the scope and a null key is a 422). And a parent carrying
no column the scope names is shared rather than partitioned — a currency
table, a plan catalogue — so every tenant may still reference it, the same rule
the includes below apply. A key naming a model you never registered — the
microservice case of storing a foreign id by design — has no parent table to
read, so nothing is checked.
That read is the only cost, and only a request carrying a forced filter pays it:
a write with nothing scoped goes straight to the adapter as before. A client’s
own ?filter= never constrains a write — only filters the server imposed do.
The field takes either spelling — the DB column name or the json name — and resolves to the same column on every path the scope reaches: the query, the write-back read, the value stamped onto the row, and the relation includes. Pick one and stay with it; a scope that resolved on some paths and not others would be a scope with a hole in it, which is what audit O2 found when the include gate accepted only the DB spelling.
If you build a maniflex.FilterExpr by hand and it expresses who may touch the
row rather than which rows were asked for, set Forced: true on it; that is
what carries it onto updates and deletes. ForceFilter and Tenancy set it for
you.
Forced filters also travel into ?include=: a related model carrying the scope
column is fetched through it, so a child that reached another tenant’s parent by
some other road — an unscoped back-office write, an import, a row predating the
key check above — does not surface in that tenant’s include. A many-to-many
junction carrying the column is read through it too, so a link another tenant
wrote between two of this tenant’s rows stays out as well. A
related model with no such column — a shared lookup table like currencies or
categories — is deliberately left unscoped, since it is not partitioned and has
nothing to scope by.
Forced: true also lets a scope be imposed before the DB step. The
Deserialize step rebuilds ctx.Query from the request, which discards a plain
filter an earlier step appended — but a Forced filter survives that rebuild.
So an Auth-step middleware may append a Forced filter to ctx.Query.Filters
and have it reach the query; a non-forced one set that early is dropped. The DB
step remains the idiomatic home for scoping (ForceFilter, Tenancy), and is
required when the scope must also cover writes end-to-end within one transaction.
ProvidesScope — running the scope before Validate
Registering a scoper on the DB step puts it after Validate. The scope reaches
the query, but nothing earlier in the chain can ask what it is. Declare
maniflex.ProvidesScope() to hoist it to run right after Deserialize instead:
server.Pipeline.DB.Register(
db.Tenancy("org_id", tenantFromAuth),
maniflex.ProvidesScope(), // ← hoists it ahead of Validate
)
Add it to every ForceFilter, ForceFilterVia and Tenancy registration unless
you have a reason not to. It changes when the middleware runs, never what it
does: the same filter is appended, the same rows are scoped, and an operation
that skips the DB step still skips it. Hoisted middleware runs exactly once — it
is removed from its own step, not duplicated.
It is opt-in, and forgetting it fails quietly. The framework cannot tell a
scope provider from any other middleware, so it will not infer this, and there is
no startup error for a registration that omits it. What you get instead is a
scope that works for the query and is invisible to everything before the write.
The sharpest symptom is on a scoped Singleton, whose row id is not known until
its scope is: validate.UniqueField then excludes the record under edit by a
placeholder that matches nothing, the row collides with itself, and every PATCH
is refused with 422 "<field> is already taken" (audit 13.12).
lock_when is the other case, and it degrades rather than breaking: without the
hoist the guard cannot read the row through your scope, so it moves to the DB
step and refuses the update there instead — correctly, one step later. Hoisting
restores the early abort, before the Service step’s transaction and business
logic run.
Anything that needs the record a request addresses should ask
ctx.ResolveResourceID() rather than reading ctx.ResourceID directly — see
the context reference.
Use the maniflex.Op* constants for Operator. It’s a bare string type, and a
hand-built filter is never parsed — only filters arriving over HTTP are — so
Operator: "equals" compiles and boots, and until v0.2.3 it produced a scope that
silently matched every row. An operator no adapter implements is now refused with
an error naming it, and the adapters render it as a false predicate rather than a
true one, so a filter reaching one by some other path matches nothing instead of
everything. maniflex.FilterOperator.Valid() reports whether an operator is one
the query builder implements.
ForceFilterVia
ForceFilter maps a field to a value, which needs the model to carry that field.
A child table often doesn’t: a DamagedItem has an item_id and nothing else,
and whether it’s yours is a fact about its Item. ForceFilterVia scopes such a
model through the column its parent carries:
// A DamagedItem is the caller's if its Item is
server.Pipeline.DB.Register(
db.ForceFilterVia("item", "owner_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.Claims["owner_id"]
}),
maniflex.ForModel("DamagedItem"),
maniflex.ProvidesScope(),
)
The first argument names the relation, in the same vocabulary a nested ?filter=
uses (?filter=author.status:neq:banned → author); the second names a column on
the parent, by JSON or DB name. The relation must be a BelongsTo — the join needs
a foreign key on this row pointing at the one row that owns it, which is what a
HasMany doesn’t have.
The alternative is to denormalise owner_id onto every child — a schema change
plus a standing obligation to keep it in step, on exactly the tables whose scoping
is easiest to get wrong — or to hand-write the predicate, which is what
declarative scoping exists to replace.
Reads join the parent and apply the predicate. Updates and deletes read the row
back through it and answer 404 on a miss, exactly as they do for ForceFilter.
Creates are scoped too, and they have to be. The foreign key is what the whole
scope hangs from, and it’s the one part of it the client supplies. On a create
there’s no row to read back, so without a check nothing looks at item_id at all
and POST {"item_id": "<another tenant's>"} lands a row under their parent — where
they can see it and you can’t. On an update the row read back is the one the old
item_id points at, so a PATCH that rewrites the key passes a check of where the
row used to be and then moves it somewhere else. So the parent a write names is
read through the scope’s own predicate first, and a miss is the same 404 a scoped
read of that parent gives. A create that names no parent at all is refused with
422: the join would find nothing, so the row would be invisible to whoever
created it.
That parent read is the only added cost, and only a scope that runs through a parent pays it.
There’s no TenancyVia, because Tenancy is ForceFilter plus stamping the
tenant column onto writes — and the whole premise here is a model with no such
column to stamp. Checking the parent is what takes its place.
Actions:
ForceFilterViaregisters on the DB step, so like every other DB-step middleware it doesn’t run for a customAction— see Scoping Actions. There’s noForceFilterViaAction: anActionScope’s filters apply to whatever model the handler touches, while aViascope is resolved against one specific model’s relations, and an action runs on a synthetic model with none. For a single-model action, build the nestedFilterExprby hand (IsNested,RelationKey,RelationTable,RelationFK,NestedField,Forced) and pass it toctx.SetActionScope.ctx.ViaFilter— the resolverForceFilterViais built on — reports this rather than guessing if you call it from an action.
Tenancy
A specialised ForceFilter for the common multi-tenant case. Reads the tenant
id from ctx.Auth and pins every query to it:
server.Pipeline.DB.Register(
db.Tenancy("org_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.Claims["org_id"].(string)
}),
maniflex.ProvidesScope(),
)
Tenancy also rewrites the org_id field on creates and updates so a tenant
cannot place rows into another tenant’s bucket. That rewrite is why the scoping
on updates matters twice over: it stamps the caller’s tenant onto whatever row
the update reaches, so an update that reached the wrong row would not merely
overwrite it — it would move it into the caller’s tenant and leave the owner
unable to see it at all.
That stamp is written with ctx.SetField, which marks the field as set by the
server. A tenant column is normally readonly — a client must not choose its own
tenant — and readonly means not from a client, so the Validate step keeps a
value the server stamped and strips only one that arrived in the request body.
Without that distinction a hoisted Tenancy would have its stamp discarded and
write the row with an empty tenant: invisible to the tenant that created it, and
visible to every caller whose scope is also empty.
Scoping Actions: TenancyAction / ForceFilterAction
Tenancy and ForceFilter register on the DB step, and a custom Action
does not run it — its pipeline is Auth → middleware → handler → Response. Their
only output is a filter on ctx.Query, which nothing in that chain reads, so
registering them on the DB step does nothing at all for an Action, silently.
Use the Action variants instead, in the action’s own Middleware list, where
they run after Auth and can read ctx.Auth:
server.Action(maniflex.ActionConfig{
Method: "POST", Path: "/orders/{id}/refund",
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
db.TenancyAction("org_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.Claims["org_id"].(string)
}),
},
Handler: refund,
})
Inside that handler every DB path either applies the scope or refuses to run:
| Path | Under a scope |
|---|---|
ctx.GetModel(name) — List/Read/Create/Update/Delete | scoped |
maniflex.List/Read/Create/Update/Delete[T] | scoped |
ctx.Aggregate | scoped (AND-ed into WHERE) |
ctx.LockForUpdate | scoped |
ctx.BeginTx — and the Tx it returns | scoped |
ctx.RawQuery, ctx.RawExec | refuses |
ctx.Search, ctx.RecursiveQuery | refuses |
Reads see only matching rows. A create is stamped with the scope’s values,
overwriting whatever the caller supplied — a row created outside the scope would
be invisible to the caller that created it. An update or delete of a record
outside the scope returns ErrNotFound, the same answer the scoped read gives.
Transactions work normally. Tx mirrors DBAdapter — its FindByID/FindMany
take a *QueryParams, its Update/Delete are keyed by id — so the Tx that
ctx.BeginTx returns is scoped the same way the accessor is, and that matters
because ctx.Tx is a public field: anything downstream that picks the
transaction up is scoped too. maniflex.WithTransaction and maniflex.Batch
both call ctx.BeginTx, so both work on a scoped action:
tx, err := ctx.BeginTx(ctx.Ctx, nil) // scoped
defer tx.Rollback()
// tx.FindByID / tx.Update / tx.Delete all honour the scope
return tx.Commit()
The refusals are the point. Raw SQL is opaque to the framework: a SELECT
string cannot be scoped without rewriting it. Scoping the convenient paths and
letting those leak in silence would put the guarantee in this page and not in the
code — worse than no guarantee, because it would be trusted. Refusing means an
action either honours the scope or fails at the first request that exercises it.
When a path genuinely must step outside the scope — an audit query across tenants, a migration — say so:
rows, err := ctx.Unscoped().RawQuery("SELECT ... FROM orders")
ctx.Unscoped() exposes RawQuery, RawExec, BeginTx, GetModel and
Search with no scope applied. It is a distinct call rather than a flag so the
bypass shows up at the call site, in the diff, and in a grep — which only works
while it stays rare, so it is deliberately not needed for ordinary work.
The scope applies only to Actions. A generated CRUD route gets its scoping from
the DB step, where ctx.RawQuery from an After-DB middleware is a normal thing
to do and is not refused.
Action variants: RateLimitAction / AuditLogAction
RateLimit and AuditLog register on the DB step, which a custom Action
never runs, so they never fire for one. Their action-list counterparts run in the
action’s own Middleware list — after the global Auth step, so ctx.Auth is
available:
server.Action(maniflex.ActionConfig{
Method: "POST", Path: "/reports/{id}/evidence",
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
db.RateLimitAction(db.RateLimitConfig{RequestsPerMinute: 10}),
db.AuditLogAction(mySink),
},
Handler: uploadEvidence,
})
RateLimitAction takes the same RateLimitConfig as RateLimit but keys on the
caller (its KeyFunc, else the authenticated user id, else the remote IP) plus
the request method and path, since an action has no model/operation to key on. It
rejects over-limit requests with 429 RATE_LIMITED and a Retry-After header.
AuditLogAction writes one audit record after a successful action (a >= 400
response is skipped), sourcing the actor and tenant from ctx.Auth, the resource
id from the {id} URL param, the result from ctx.Response.Data, and
Operation = OpAction. Writes are fire-and-forget, so a sink error never fails
the request; change diffing (WithChanges) does not apply to actions.
Request budgeting
Paginate
Caps the maximum ?limit= accepted on list responses. Per-model overrides are
common for tables whose rows are expensive to render:
server.Pipeline.DB.Register(db.Paginate(50), maniflex.ForModel("AuditLog"))
RateLimit
A token-bucket rate limiter scoped by IP or by authenticated user:
server.Pipeline.DB.Register(
db.RateLimit(db.RateLimitConfig{
RequestsPerMinute: 10,
KeyFunc: func(ctx *maniflex.ServerContext) string {
if ctx.Auth != nil {
return ctx.Auth.UserID
}
return ctx.Request.RemoteAddr
},
}),
maniflex.ForModel("PasswordReset"),
)
Rejected requests receive 429 RATE_LIMITED. The counter is in-process by
default; for a limit shared across replicas set RateLimitConfig.Backend (see
middleware/db/redis
for the Redis implementation).
When the key falls back to the client IP (ctx.Request.RemoteAddr), that address
is the direct TCP peer unless Config.TrustProxyHeaders is enabled — set it
(only behind a trusted proxy) so per-IP limits see the real client instead of the
load balancer. See Security.
RateLimitField
Rate-limits by the value of a request-body field rather than by caller identity — cap password-reset or OTP requests per email address, for instance, so one address cannot be flooded regardless of which client sends the requests:
server.Pipeline.DB.Register(
db.RateLimitField("email", 3, time.Hour),
maniflex.ForModel("PasswordReset"),
maniflex.ForOperation(maniflex.OpCreate),
)
The arguments are the field name (its parsed-body value becomes the bucket key),
the request cap, and the window. If the field is absent from the parsed body the
limit is not applied, so a legitimate request is never blocked by a missing
field. Rejected requests receive 429 RATE_LIMITED. Pass
db.WithRateLimitBackend to share the window across replicas, or
db.WithRateLimitErrorMessage to override the 429 message.
Post-write hooks
These run at maniflex.After position so they only fire when the database write
succeeded.
AuditLog
Writes one audit record per mutating operation to a configured sink:
server.Pipeline.DB.Register(
db.AuditLog(mySink),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
mySink is anything implementing the audit interface — a logger, a database
table, an external SIEM. The record carries the operation, model name, actor
(from ctx.Auth), and a JSON diff of the affected row.
Invalidate
Invalidates cache keys when a row changes. The key list is computed per request:
server.Pipeline.DB.Register(
db.Invalidate(redisCache, func(ctx *maniflex.ServerContext) []string {
return []string{
"posts:list",
fmt.Sprintf("post:%s", ctx.ResourceID),
}
}),
maniflex.ForModel("Post"),
maniflex.AtPosition(maniflex.After),
)
CacheQuery
Memoises read results (OpRead, OpList) in a CacheStore — the read-side
complement to Invalidate. On a cache hit it sets ctx.DBResult and the
adapter read is skipped; the Response step renders the cached result. On a miss
it runs the read and stores the result for TTL. Pair it with Invalidate on
writes to evict stale entries.
cache := maniflex.NewMemoryCache() // or a Redis-backed CacheStore
server.Pipeline.DB.Register(
db.CacheQuery(cache, db.CacheConfig{
TTL: 5 * time.Minute,
KeyFunc: func(ctx *maniflex.ServerContext) string {
// Only cache the common, high-traffic query shapes. Requests that
// filter on `name` or carry a ?q= search are typically long-tail,
// ad-hoc lookups — caching them floods the store with low-value
// entries that are rarely read back, so skip them by returning "".
if q := ctx.Query; q != nil {
if q.Search != "" {
return ""
}
for _, f := range q.Filters {
if f.Field == "name" {
return ""
}
}
}
return "products:list:" + ctx.Request.URL.RawQuery
},
}),
maniflex.ForModel("Product"),
maniflex.ForOperation(maniflex.OpList, maniflex.OpRead),
)
server.Pipeline.DB.Register(
db.Invalidate(cache, func(*maniflex.ServerContext) []string {
return []string{"products:list:..."}
}),
maniflex.ForModel("Product"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
KeyFunc must capture every input that changes the result (model, tenant,
filters, sort, pagination, includes); returning "" skips the cache for that
request. The value stored is ctx.DBResult, so a distributed CacheStore must
round-trip a *maniflex.ListResult for lists, and records — alone or as list
rows — as a map[string]any or a pointer to the model’s own struct. A store
that decodes into anything else is treated as a miss and the request reads the
database, rather than failing. Avoid
caching mfx:"encrypted" models, since the decrypted result would live in the
cache.
Ordering
Row-level scopers (ForceFilter, ForceFilterVia, Tenancy) must run Before
the default DB step — the default position — so the filter is in place when the
SELECT or UPDATE runs. Post-write hooks must run After, so they observe the result.
The package’s defaults follow this; only change them if you know why.
Before is the latest a scoper can run, not the earliest it should. Pair it with
maniflex.ProvidesScope()
to hoist the scoper ahead of the Validate step, so the steps between Deserialize
and the write see the same scope the query does.
Response Middleware
The maniflex/middleware/response package shapes the outgoing response —
headers, body transforms, redactions, and observability hooks — on the
Response step. CORS is the exception: it is HTTP middleware because
preflight must run before route dispatch and authentication.
Cross-cutting headers
CORSHeaders
Adds CORS headers to allowed cross-origin responses and validates browser
preflight before route dispatch. At least one origin is required — pass
explicit origins (recommended) or "*" to allow any origin. Calling it with no
origins panics at startup, so a permissive wildcard is never applied by accident.
import "github.com/xaleel/maniflex/middleware/response"
// Configure before calling maniflex.New(cfg).
// Explicit origins (recommended)
cfg.HTTPMiddlewares = append(cfg.HTTPMiddlewares,
response.CORSHeaders("https://app.example.com"))
// Public API: opt in to any origin explicitly
cfg.HTTPMiddlewares = append(cfg.HTTPMiddlewares, response.CORSHeaders("*"))
For credentials or custom allowed headers/methods/max-age, use
CORSHeadersWithConfig. AllowCredentials cannot be combined with a "*"
origin (browsers reject that combination) and panics if you try:
cfg.HTTPMiddlewares = append(cfg.HTTPMiddlewares,
response.CORSHeadersWithConfig(response.CORSConfig{
AllowOrigins: []string{"https://app.example.com"},
AllowCredentials: true,
}))
CORSHeaders returns router-level middleware. A true preflight is OPTIONS
with both Origin and Access-Control-Request-Method. Allowed preflights
return 204 No Content with an empty body before Auth runs. Disallowed origins
and request headers return 403; disallowed methods return 405. Plain
OPTIONS requests continue to the generated route and retain its Allow
response.
The default allowed request headers include Authorization, Content-Type,
If-Match, and If-None-Match. ETag and X-Request-ID are exposed by
default so browser code can perform optimistic-locking updates and correlate
failures. Override AllowHeaders, AllowMethods, or ExposeHeaders when the
application needs a narrower or broader policy.
AddHeader
Sets one static header on every response:
server.Pipeline.Response.Register(
response.AddHeader("Strict-Transport-Security", "max-age=63072000"),
)
Caching
Cache
Sets an explicit cache policy and ETag on reads. The zero policy is
private, max-age=0; shared caches are never enabled implicitly. Register at
maniflex.After so the framework’s own headers do not override yours:
server.Pipeline.Response.Register(
response.Cache(response.CacheConfig{
MaxAge: 300, // private, 5 minutes
Vary: []string{"Authorization"},
}),
maniflex.ForOperation(maniflex.OpRead, maniflex.OpList),
maniflex.AtPosition(maniflex.After),
)
CacheConfig supports:
Private: true— only a browser/user-agent cache may store the response. This is the default when no storage mode is selected.NoStore: true— no cache may store the response. This also disables ETags andIf-None-Matchhandling.Public: true— explicitly permits shared proxy/CDN storage.MaxAge— freshness lifetime in seconds.Vary— request-header names that form part of the representation’s cache key; values are canonicalized, deduplicated, and merged with existing entries such as CORS’sVary: Origin.
Public, Private, and NoStore are mutually exclusive. Public caching is a
security decision, not a performance default:
server.Pipeline.Response.Register(
response.Cache(response.CacheConfig{
Public: true,
MaxAge: 60,
Vary: []string{"Accept-Encoding"},
}),
maniflex.ForModel("PublicArticle"),
maniflex.ForOperation(maniflex.OpRead, maniflex.OpList),
maniflex.AtPosition(maniflex.After),
)
Do not set Public on authenticated, tenant-filtered, role-dependent, or
dynamically redacted responses unless every authorization input is included in
the shared cache key and the proxy/CDN is verified to honor it. Common inputs
include Authorization, Cookie, tenant headers, locale, and negotiated
representation headers. When that guarantee is difficult to prove, use the
private default or NoStore.
Body transforms
TransformField
Rewrites a single field value before serialisation. Common use: rebasing a stored relative path onto a CDN host.
server.Pipeline.Response.Register(
response.TransformField("avatar_url", func(v any) any {
return cdnBase + v.(string)
}),
)
RedactField
Hides a field from the response conditionally. The predicate decides per
request, often based on ctx.Auth:
server.Pipeline.Response.Register(
response.RedactField("phone", func(ctx *maniflex.ServerContext) bool {
return !ctx.HasRole("support")
}),
)
RedactField is the right tool for view-time access control on individual
columns. For all-or-nothing exclusion across an entire model, the hidden or
writeonly field tag is simpler.
For the write side — “only a superuser may set this field” — use
validate.FieldRole / validate.RestrictField,
which takes the same predicate on the Validate step. Note the two differ on
purpose when the predicate fails: a redacted read returns the record without the
field, while a refused write returns 403 rather than quietly dropping it.
It covers exports too: GET /:model/export masks the same field for the
same callers, and drops it from the CSV/XLSX header rather than emitting an
empty column.
Writing your own masking middleware
A Response-step middleware normally masks by editing ctx.Response after
next() returns. That is not enough on its own, because an export has no
ctx.Response — it streams its bytes during next() — so a middleware that
only edits one masks the JSON and leaves the export in full.
Declare the field instead, before calling next():
func maskSalary(ctx *maniflex.ServerContext, next func() error) error {
if !ctx.HasRole("admin") {
ctx.RedactResponseField("salary") // before next(), so the export sees it
}
return next()
}
The declaration is honoured by every read path — list, single read, create and
update echoes, and both export formats. response.RedactField does this for
you.
Before v0.2.5 a masking middleware applied to JSON responses only. An app that hid a column from non-admins served it in full at
/:model/export.
Envelope
Replaces the default {"data": ...} envelope with one of your own:
server.Pipeline.Response.Register(
response.Envelope(func(ctx *maniflex.ServerContext, data any, meta *maniflex.ResponseMeta) any {
return map[string]any{
"result": data,
"paging": meta,
"trace_id": ctx.TraceID,
}
}),
)
Useful when integrating with a frontend or API gateway that expects a different shape. Error responses are unaffected; only success responses are re-enveloped.
Observability
Logging
Writes a structured access log after the complete HTTP request:
server.ObserveRequests(
response.Logging(slog.Default()),
)
The line carries request ID, method, path, model, operation, final status, full router-to-response duration, and the authenticated user when set. Router-level observation also records requests rejected during Auth, before they can reach the Response step.
Metrics
Records per-request metrics — count, latency, and exact status — into a configured collector:
server.ObserveRequests(
response.Metrics(myCollector),
)
Any sink implementing MetricsCollector works. Counters and histograms retain
model, operation, and status labels; non-model routes use empty model and
operation labels.
Wiring Prometheus
maniflex ships no exporter. Metrics leave through this interface, so the framework depends on no metrics library and you can use any — but that leaves the other side of the interface to you, so here is a complete one.
The two sides disagree about when labels are fixed: MetricsCollector passes a
label map with every observation, while a Prometheus vector binds its label
names when it is constructed. The adapter builds a vector on first sight of a
metric name and projects every later observation onto those names, which is what
keeps the exposition valid — Prometheus rejects a metric whose label set varies
between samples.
// promCollector adapts prometheus/client_golang to response.MetricsCollector.
//
// The two sides disagree about when labels are fixed: MetricsCollector passes a
// label map with every observation, while a Prometheus vector binds its label
// names when it is constructed. So a vector is built on first sight of a metric
// name, its label names taken from that first observation, and every later
// observation is projected onto those names — missing keys become empty, extra
// keys are dropped. Prometheus rejects a metric whose label set varies between
// samples, so projecting is what keeps the exposition valid.
type promCollector struct {
reg *prometheus.Registry
mu sync.Mutex
counters map[string]*prometheus.CounterVec
histograms map[string]*prometheus.HistogramVec
labelNames map[string][]string
}
func newPromCollector(reg *prometheus.Registry) *promCollector {
return &promCollector{
reg: reg,
counters: map[string]*prometheus.CounterVec{},
histograms: map[string]*prometheus.HistogramVec{},
labelNames: map[string][]string{},
}
}
// namesFor returns the label names bound to metric on its first observation.
func (c *promCollector) namesFor(metric string, labels map[string]string) []string {
if names, ok := c.labelNames[metric]; ok {
return names
}
names := make([]string, 0, len(labels))
for k := range labels {
names = append(names, k)
}
sort.Strings(names) // stable order, so the values line up on every call
c.labelNames[metric] = names
return names
}
// valuesFor projects labels onto the names this metric was created with.
func valuesFor(names []string, labels map[string]string) []string {
values := make([]string, len(names))
for i, n := range names {
values[i] = labels[n]
}
return values
}
func (c *promCollector) IncCounter(name string, labels map[string]string) {
c.mu.Lock()
defer c.mu.Unlock()
names := c.namesFor(name, labels)
vec, ok := c.counters[name]
if !ok {
vec = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: name, Help: "maniflex " + name}, names)
c.reg.MustRegister(vec)
c.counters[name] = vec
}
vec.WithLabelValues(valuesFor(names, labels)...).Inc()
}
func (c *promCollector) ObserveHistogram(name string, value float64, labels map[string]string) {
c.mu.Lock()
defer c.mu.Unlock()
names := c.namesFor(name, labels)
vec, ok := c.histograms[name]
if !ok {
vec = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: name,
Help: "maniflex " + name,
// response.Metrics records seconds, which is what DefBuckets covers
// (5ms to 10s). Widen it for an API that streams or exports.
Buckets: prometheus.DefBuckets,
}, names)
c.reg.MustRegister(vec)
c.histograms[name] = vec
}
vec.WithLabelValues(valuesFor(names, labels)...).Observe(value)
}
/metrics is not a maniflex route. Mount the API under your own router with
maniflex.Mount and
register the scrape endpoint beside it:
func main() {
registry := prometheus.NewRegistry()
collector := newPromCollector(registry)
server := maniflex.New(maniflex.Config{
PathPrefix: "/api",
MaxConcurrentRequests: 64,
})
server.MustRegister(Order{})
// response.Metrics observes at the router level, so it also counts requests
// rejected during Auth — the ones that never reach a model.
server.ObserveRequests(response.Metrics(collector))
// /metrics is not a maniflex route: mount the API under your own router and
// register the scrape endpoint beside it. Keep it off the public listener,
// or put an auth middleware in front — the label set names every model and
// operation your API exposes.
r := chi.NewRouter()
r.Handle("/metrics", promhttp.HandlerFor(registry, promhttp.HandlerOpts{}))
maniflex.Mount(r, server)
log.Println("API on /api, metrics on /metrics")
if err := http.ListenAndServe(":8080", r); err != nil {
log.Fatal(err)
}
}
Keep that endpoint off the public listener, or put an auth middleware in front of it: the label set names every model and operation your API exposes.
This example is compiled in CI, so an API change cannot leave it broken.
OpenAPI Middleware
The maniflex/middleware/openapi package customises the auto-generated OpenAPI
3.1 specification served at GET /openapi.json. Every middleware here is
registered on the Pipeline.OpenAPI.Generate step at maniflex.After
position, so it sees the framework’s generated spec and can mutate it before
the Response step serialises it.
SetTitle, SetDescription
Override the default title and description, which are derived from
Config.ServiceName:
import "github.com/xaleel/maniflex/middleware/openapi"
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetTitle("Orders API"),
maniflex.After,
)
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetDescription("# Orders API\nProduction endpoints for the orders service."),
maniflex.After,
)
SetVersion
Overrides the spec’s info.version, which is otherwise the framework default.
Use it to surface your service’s own release version in the published spec:
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetVersion("2.1.0"),
maniflex.After,
)
AddServer
Declares a server URL in servers[]. Repeat for multiple environments:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddServer("https://api.example.com", "Production"), maniflex.After)
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddServer("https://staging.example.com", "Staging"), maniflex.After)
Without AddServer the spec carries no servers array, leaving clients to
resolve URLs against the host that served the spec.
AddSecurityScheme
Adds a security scheme to components.securitySchemes and applies it to
every operation generated by the framework:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddSecurityScheme("bearerAuth", maniflex.OASSecurityScheme{
Type: "http",
Scheme: "bearer",
BearerFormat: "JWT",
}),
maniflex.After,
)
Pair with auth.JWTAuth on the runtime side. For API keys,
use Type: "apiKey" and set In and Name.
AddTag
Appends a Tag Object to the spec. Tags render as collapsible groups in Swagger
UI; maniflex already emits one tag per model, so use AddTag for cross-cutting
or documentation-only groupings. A tag whose name already exists is not
duplicated.
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddTag("Authentication", "Endpoints related to user authentication"),
maniflex.After,
)
InjectRequestExample
Attaches an example request body to a specific operation so Swagger UI’s “Try it out” pre-populates with realistic values. The operation is named by path and lower-case method:
server.Pipeline.OpenAPI.Generate.Register(
openapi.InjectRequestExample(
openapi.OperationTarget{Path: "/posts", Method: "post"},
"Example post", map[string]any{
"title": "Hello World",
"body": "My first post.",
"status": "draft",
}),
maniflex.After,
)
The example is only applied when the target operation and its request body exist in the generated spec; an unmatched target is a no-op.
AddExtension
A general-purpose escape hatch — receives the full *maniflex.OpenAPISpec and lets
you mutate any part of it:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddExtension(func(spec *maniflex.OpenAPISpec) {
spec.Info.Description += "\n\nContact the API team at [email protected]."
}),
maniflex.After,
)
Use sparingly — anything you can express through the typed helpers is easier to read.
Securing the spec itself
Generated specifications are not mounted by default. Prefer one router-level policy for both OpenAPI and AsyncAPI:
cfg.Documentation = maniflex.DocumentationConfig{
Middleware: []maniflex.HTTPMiddleware{
maniflex.AdaptAuth(
auth.JWTAuth(jwtSecret),
auth.RequireRole("internal"),
),
},
}
Pipeline.OpenAPI.Auth remains available for OpenAPI-specific middleware, but
its function type is OpenAPIMiddlewareFunc; model-route helpers such as
auth.JWTAuth and auth.RequireRole must be used through AdaptAuth as shown
above.
Querying
Every generated list and read endpoint accepts the same query parameters —
page, limit, filter, sort, include, and select. This page
documents their grammar and the fields that opt in to each.
Query complexity limits
Client-controlled query shapes are bounded before SQL is built. Defaults allow
at most 8 KiB for the complete request URI, 32 filter clauses, 8 bracketed OR
groups with 8 clauses each, 8 sort fields, 64 selected fields, and 8 include
paths. Exceeding the URI ceiling returns 414 URI_TOO_LONG; exceeding a shape
limit returns 400 INVALID_QUERY.
Applications can change these through Config.QueryLimits, and can override
individual fields for one model with ModelConfig.QueryLimits. A zero field
inherits; a negative field explicitly disables that limit. The router-level
global URI ceiling cannot be loosened per model. See
Configuration for every field and default.
page and limit
Standard offset pagination.
?page=2&limit=20
| Parameter | Default | Maximum |
|---|---|---|
page | 1 | 1,000,000 |
limit | 20 | 200 |
Limits above the maximum are clamped silently. Pages above the maximum, values
whose pagination arithmetic cannot fit, and negative or non-numeric values are
rejected with 400 INVALID_QUERY.
The response carries meta.total, meta.page, meta.limit, and meta.pages
— see Response Envelope. The total is a second query; see count to decline it.
cursor (keyset pagination)
Offset pagination skips or duplicates rows when the dataset changes between page fetches — delete a row on page 1 and page 2 silently jumps a record. Keyset (cursor) pagination walks the data by a stable ordering key instead, so the window never shifts. Opt a model in by naming a sortable, effectively monotonic cursor column:
type Event struct {
maniflex.BaseModel `mfx:"cursor_field:created_at"` // created_at is sortable on BaseModel
Name string `json:"name" db:"name"`
}
Equivalently, set ModelConfig.CursorField: "created_at" at registration, or
put mfx:"...,cursor_field:<name>" on any of the model’s own fields.
The cursor column must be mfx:"sortable", not nullable, and a supported
scalar type (string, bool, an integer, a float, or time.Time). Pointer,
collection, and structured fields are rejected at registration. Keyset pagination
needs a total order, and NULL has no place in one: the boundary comparison is never
true for it, and Postgres and SQLite don’t even agree where NULLs sort. A nullable
cursor column would drop or repeat rows across pages, so the model fails to
register rather than paginating wrongly.
When the cursor column is one of BaseModel’s — created_at is the usual
choice — sortable is not a default and cursor_field does not grant it
implicitly. Declare both:
type Event struct {
maniflex.BaseModel `mfx:"cursor_field:created_at"`
Name string `json:"name" mfx:"required"`
}
server.MustRegister(Event{}, maniflex.ModelConfig{
BaseModelTags: map[string]string{"created_at": "sortable,index"},
})
Writing one half without the other fails registration with an error naming the missing piece.
The presence of ?cursor= switches the request into keyset mode (it supersedes
?page). Send an empty value for the first page, then the meta.next_cursor
from each response to fetch the next:
GET /events?cursor=&limit=20 → first page
GET /events?cursor=<next_cursor>&limit=20 → following page
The walk is ordered by (cursor_field, id) — id is the implicit tiebreaker so
the order is total even when the cursor column ties. The default direction is
ascending; sort on the cursor field to reverse it:
GET /events?cursor=&sort=created_at:desc
Any ?sort= on a different field is rejected with 400 in cursor mode, since
the keyset order is fixed to the cursor column.
Cursor responses carry a different meta shape — no total/page/pages
(the count is skipped, which is the point on large tables):
{ "data": [ ... ], "meta": { "limit": 20, "next_cursor": "eyJ2Ijoi...", "has_more": true } }
has_more is false and next_cursor is omitted on the last page. The token is
opaque — treat it as a string and pass it back verbatim. Tokens carry the cursor
value’s type and are checked against the model field; a token for a different
field type is rejected with 400 INVALID_QUERY. Missing IDs, null or non-scalar
values, trailing JSON, and values outside the field or database driver’s supported
range are rejected the same way before query execution. Timestamp cursors use a
fixed-width UTC representation so ordering is identical on SQLite and Postgres.
Valid scalar unversioned tokens issued by earlier Maniflex releases remain
accepted during upgrades.
count
meta.total comes from a second query — a COUNT over the whole filtered set,
run again on every page. On a large or heavily filtered table it is the
more expensive half of a list request.
?count=false
No count query runs, and the response omits total and pages, carrying
has_more in their place:
{ "data": [ ... ], "meta": { "page": 2, "limit": 20, "has_more": true } }
The keys are absent rather than zero, to avoid reading “not counted” as
“no rows”. has_more comes from reading one row past the page — the same
over-fetch cursor mode uses, and the cost of one row rather than a scan. For a
“load more” button that is the whole of what total was for.
count=true is the default spelled out. 1 and 0 are accepted for each; any
other value is rejected with 400 INVALID_QUERY rather than guessed at.
It can be used with other query parameters (?filter=, ?q=, ?sort=, ?include=)
since it only changes the meta. The exception is ?cursor=, which never counts, and
?count=false in this case is a no-op. Combining ?cursor=..&count=true is rejected with
400 INVALID_QUERY.
filter
Each filter is a colon-separated triple — field, operator, value:
?filter=status:eq:published
?filter=views:gt:100
?filter=created_at:gte:2025-01-01
Multiple filters combine with AND:
?filter=status:eq:published&filter=views:gt:100
Filters reference a field by its json name. Only fields tagged
mfx:"filterable" may be used; unknown or non-filterable references abort the
request with 400 INVALID_QUERY.
Combining filters with OR
Add a bracketed index to put filters in the same OR group. Filters sharing an index are OR-ed together:
?filter[0]=status:eq:draft&filter[0]=status:eq:published
Different indexes are separate groups, and groups combine with AND. A bare
?filter= has no group and is its own AND clause, so the two spellings mix
freely:
# (owner = u1 OR owner = u2) AND amount >= 20
?filter[0]=owner:eq:u1&filter[0]=owner:eq:u2&filter[1]=amount:gte:20
# resolved = false AND (owner = u1 OR owner = u2)
?filter=resolved:eq:false&filter[0]=owner:eq:u1&filter[0]=owner:eq:u2
That is the whole expressible shape: an AND of ORs. There is no way to OR
across groups, and no nesting — (a AND b) OR (c AND d) cannot be written as a
query string. When you need it, put the query behind a
custom action and write it with
ctx.RawQuery, where the shape is yours to
choose and is not client-controlled.
The index is a label, not an ordering: filter[7] and filter[2] are simply
two groups, and gaps are fine. It must be a non-negative integer — filter[recent]
is refused with 400 INVALID_QUERY naming the requirement rather than being
ignored.
Group counts are bounded by Config.QueryLimits — by default 8 groups of 8
clauses, inside an overall ceiling of 32 filter clauses. See
Query complexity limits above.
The /aggregate endpoint reads ?filter= with exactly these semantics, so a
grouped filter counts the rows the same filter lists.
Go callers: the equivalent is
FilterExpr.Group. Any value ≥ 1 is a group;0is the zero value and means ungrouped, so groups start at 1 rather than 0 and the numbering does not line up with the URL’s —filter[0]parses toGroup: 1.
filters := []*maniflex.FilterExpr{
// (owner = u1 OR owner = u2) AND amount >= 20
{Field: "owner", Operator: maniflex.OpEq, Value: "u1", Group: 1},
{Field: "owner", Operator: maniflex.OpEq, Value: "u2", Group: 1},
{Field: "amount", Operator: maniflex.OpGte, Value: 20}, // ungrouped → AND
}
Values are read against the column’s type
A filter value arrives as text — a URL has no types — and is coerced to the form its column compares against, so the same filter means the same thing on every driver.
A boolean column accepts true/false and 1/0, in any case:
?filter=resolved:eq:false
?filter=resolved:eq:0 # identical
?filter=resolved:in:true,false
This matters more than it looks. A caller interpolating a boolean into a URL
(`resolved:eq:${showResolved}`) sends the word, and a bound string is not
the same thing as a SQL literal: on SQLite, where booleans are stored in an
INTEGER column, the word false stays TEXT and can never compare equal, so
the filter used to return zero rows with no error while the identical
request against Postgres worked. Both drivers now agree.
A timestamp column takes any full RFC3339 value, with or without a fraction
and in any zone; it is normalised to UTC in a fixed-width form so that string
comparison on SQLite orders the same way instants do. A date-only bound
(2026-01-01) is left exactly as written and keeps its meaning.
A value that is not a recognised spelling for its column is passed through untouched rather than guessed at.
BaseModel’s id, created_at and updated_at are not filterable by
default — the columns are readonly and nothing more. A model opts them in at
registration, since BaseModel lives in the framework and its struct tags
cannot be edited:
server.MustRegister(Post{}, maniflex.ModelConfig{
BaseModelTags: map[string]string{"created_at": "filterable,sortable"},
})
See BaseModel for the per-column allowlist.
Operators
| Operator | Effect | Value |
|---|---|---|
eq | field = value | one value |
neq | field ≠ value | one value |
gt, gte, lt, lte | numeric and date comparisons | one value |
like | SQL LIKE, case-sensitive | one pattern — % and _ are wildcards |
ilike | SQL ILIKE, case-insensitive | one pattern — % and _ are wildcards |
contains | field contains the value, case-insensitive | one literal value |
starts_with | field starts with the value, case-insensitive | one literal value |
ends_with | field ends with the value, case-insensitive | one literal value |
has | JSON column holds this element / key=value pair | one value, or key=value |
not_has | JSON column does not hold it | one value, or key=value |
in | field IN (…) | at least one comma-separated value |
not_in | field NOT IN (…) | at least one comma-separated value |
between | field ≥ lo AND ≤ hi (inclusive) | exactly two comma-separated values lo,hi |
is_null | field IS NULL | no value |
not_null | field IS NOT NULL | no value |
eq_field, neq_field | field = / ≠ another column | the name of another column |
gt_field, gte_field, lt_field, lte_field | comparisons against another column | the name of another column |
?filter=tag:in:go,rust,zig
?filter=amount:between:100,500
?filter=created_at:between:2025-01-01,2025-03-31
?filter=archived_at:is_null
?filter=title:ilike:%intro%
?filter=title:contains:intro
?filter=paid_amount:gte_field:amount_due
?filter=category_ids:has:cat-1
?filter=meta:has:tier=gold
Patterns vs. literals
like and ilike take a pattern: % matches any run of characters and _
matches exactly one. That is what makes ?filter=title:ilike:%intro% work — but
it also means a value the user typed is interpreted rather than matched.
?filter=label:like:50% finds 500 units and 50 off as readily as the 50%
you were looking for, and there is no portable way to escape a % in a pattern
(SQLite has no escape character by default; Postgres has a backslash).
contains, starts_with, and ends_with take a literal: % and _ in the
value are escaped for you and match themselves, so ?filter=label:contains:50%
finds exactly the labels containing 50%. They are case-insensitive on both
backends. Use them for anything a user typed — a search box, a filename, an SKU —
and reach for like/ilike only when the caller genuinely is writing a pattern.
Note that % must still be percent-encoded in a URL (%25), as in any query
string:
?filter=label:contains:50%25 → matches the literal "50%"
?filter=label:like:50%25 → matches "50%", "500 units", "50 off", …
Filtering inside a JSON column
has asks whether a JSON column holds a value. It needs the column to say which
kind of document it holds, with mfx:"json_array" or mfx:"json_object":
type Merchant struct {
maniflex.BaseModel
CategoryIDs JSONArray `json:"categoryIds" mfx:"filterable,json_array"`
Meta JSONObject `json:"meta" mfx:"filterable,json_object"`
}
?filter=categoryIds:has:cat-1 # the array holds the element "cat-1"
?filter=categoryIds:not_has:cat-1 # …and its negation
?filter=meta:has:tier=gold # the object holds "tier": "gold"
The tag is required rather than inferred because nothing else can tell: a JSON
column is JSONB on Postgres but plain TEXT on SQLite — the same SQL type a
string column has — so the running driver cannot answer the question for you.
Both sides are compared as JSON, not as text. ?filter=tags:has:5 matches an
array holding the number 5, and does not match one holding the string "5";
the two are different values, and they stay different on both backends.
On Postgres both forms compile to @>, so a GIN index on the column serves them.
Only a top-level key is supported on an object — meta:has:a.b=x is refused
rather than quietly reading a.b as a single key.
containsis refused on a JSON column. It compiles to a substringLIKEover the serialised document, which matched across element boundaries on SQLite — a search forcat-1returning a row holding["cat-10"]— and could not run at all on Postgres, whereLIKEhas noJSONBoverload. The same filter was quietly wrong in development and a hard error in production, so it now returns400naminghasinstead. Text columns are unaffected.
Comparing two columns
The *_field operators compare one column against another column of the same
record, instead of against a value you supply:
?filter=paid_amount:gte_field:amount_due # settled orders
?filter=paid_amount:lt_field:amount_due # orders still owing
The value is a field name, never a literal — that is why these are separate
operators rather than a marker on the value. ?filter=note:eq:status compares
the note column against the text "status"; ?filter=note:eq_field:status
compares it against the status column. Neither spelling can be mistaken for
the other.
Both sides must:
- be columns on the model you are listing — a relation (
customer.credit) or a locale sub-key is rejected with400 INVALID_QUERY; - be marked
mfx:"filterable", the same tag the left side of any filter needs; - hold the same kind of value. Numbers compare with numbers, strings with
strings, booleans with booleans, timestamps with timestamps. Mixing them —
?filter=paid_amount:gte_field:note— is rejected rather than left to the database, because SQLite and PostgreSQL would not agree on what it means.
Encrypted columns cannot be compared, for the same reason they cannot be filtered: their stored ordering is not their plaintext ordering.
If either column is NULL on a row, the comparison is NULL rather than true,
so the row is excluded — from neq_field as well as from eq_field. Add an
explicit ?filter=credit:not_null when you need those rows counted.
Arithmetic is not supported: there is no way to write
paid_amount >= amount_due + delivery_fee. Maintain the total you want to
compare against as its own column.
Filtering on related fields
When a relation is declared on the model, you can filter by a field on the related table using dot notation:
?filter=user.role:eq:admin
?filter=posts.status:eq:published
The related field must itself be filterable. The framework joins the related
table for the query; no separate ?include= is required to filter on it (but
you still need ?include= to return the related row).
q (full-text search)
?q= runs a native full-text search over every field tagged mfx:"searchable"
and orders the results by match relevance:
?q=hello world
?q=postgres&filter=tag:eq:db
This is distinct from filter: full-text search uses the database’s own
ranking, stemming, and tokenisation rather than literal comparison, so ?q=run
also matches running, and the densest match ranks first. The backend’s native
machinery does the work — a tsvector column and GIN index on PostgreSQL, an
FTS5 index on SQLite — both provisioned automatically during migration.
- Only models with at least one
mfx:"searchable"field accept?q=; on any other model it aborts with400 INVALID_QUERY. Searchable fields must be text columns. ?q=combines with?filter=(ANDed) and the usual?page=/?limit=offset pagination. It cannot be combined with?cursor=, since keyset order and relevance order are mutually exclusive.- An empty value (
?q=) is ignored — the list is returned unfiltered. - On PostgreSQL the text-search configuration defaults to
english; override it per model withModelConfig.SearchLanguage.
type Article struct {
maniflex.BaseModel
Title string `json:"title" db:"title" mfx:"required,searchable"`
Body string `json:"body" db:"body" mfx:"searchable"`
}
// GET /articles?q=keyset+pagination → relevance-ranked matches
sort
Each sort is field:direction:
?sort=created_at:desc
?sort=title:asc
Multiple sorts compose left-to-right (primary, secondary, …):
?sort=status:asc&sort=created_at:desc
Only fields tagged mfx:"sortable" may be used. BaseModel’s id,
created_at and updated_at are not sortable by default — opt in with
ModelConfig.BaseModelTags as shown under filter above. A sort on
a column that has not opted in returns 400, and the error names
BaseModelTags as the fix.
Sorting on a relation field
Use relation.field to sort by a column on a BelongsTo parent. The server
adds the LEFT JOIN automatically — no filter or include on that relation is
required:
?sort=user.name:asc
?sort=vendor.name:desc&filter=status:eq:open
The related field must be tagged mfx:"sortable" on the parent model. Only
BelongsTo relations are supported; relation.field on a HasMany or
ManyToMany returns 400, as does an unknown relation or a non-sortable
related field.
include
Loads related records inline. The value is a comma-separated list of relation keys:
?include=user
?include=user,comments
Each key becomes a nested object (for BelongsTo) or array (for HasMany and
ManyToMany) on the returned row. See Relations for how
relation keys are derived.
Includes are populated by separate queries after the main query — they do not multiply rows or affect pagination.
One level of nesting
A key may carry a single dot to load a relation of the related model:
?include=author.company
?include=author.company,comments
?include=author.company implies author — the parent is what the child hangs
off — so you do not need to name both.
Two segments is the limit. ?include=a.b.c is refused with 400 INVALID_QUERY. Each level is one more batched query, and the tree comes from the
client, so leaving it uncapped would let a caller choose how much work a request
costs.
Every segment must name a real relation; a typo is a 400, not a silently
missing key. Nested rows are scoped, decrypted and field-filtered exactly as the
first level is — a forced filter (db.Tenancy, db.ForceFilter) applies at every
level, and hidden / writeonly fields on the nested model stay out.
Go callers: nesting applies to the JSON response. The typed relation structs (
post.Author) are still populated one level deep, sopost.Author.Companyis not filled in by a typed read. Use the JSON path, or a secondmaniflex.Read.
select
Request a subset of fields instead of the full row. Useful for wide tables (payroll, product catalogues with 40+ attributes) where most columns are irrelevant to the caller.
?select=id,name,department
?select=id,amount,status
The value is a comma-separated list of JSON field names. Unknown names
abort the request with 400 INVALID_QUERY. Fields tagged mfx:"hidden" or
mfx:"writeonly" are still stripped from the response even if explicitly
selected — the projection happens at the database layer, not as an ACL bypass.
?select= applies to both list (GET /:model) and read
(GET /:model/:id) endpoints. It can be combined freely with filter, sort,
and include.
Putting it together
A complete request that exercises all parameters:
GET /api/posts
?filter=status:eq:published
&filter=views:gte:100
&sort=created_at:desc
&include=user,comments
&select=id,title,views,status
&page=1
&limit=20
The framework parses the query string once in the Deserialize step into
ctx.Query (a *QueryParams), which middleware can read and modify before
the DB step. Tenant-scoping middleware, for example, appends a filter to
ctx.Query.Filters to enforce row-level access — see
Example 2.
Searching
maniflex has two layers of full-text search:
- Per-model search — the
?q=parameter on a model’s list endpoint, over itsmfx:"searchable"fields. Covered in Querying. - Cross-model search — search several models at once and merge the hits into
one relevance-ranked list. That is what this page documents: the
ctx.Searchprimitive and the built-inGET /searchendpoint.
Both use the database’s native full-text engine (PostgreSQL tsvector /
ts_rank, SQLite FTS5 / bm25), provisioned automatically for every model that
declares mfx:"searchable" fields. Free-form input is sanitised, so a query can
never be a syntax error.
The ctx.Search primitive
ctx.Search runs a cross-model search and returns the merged, relevance-ranked
hits. Use it from a custom Action to build search
endpoints scoped to exactly the models you choose, with your own authorisation:
server.Action(maniflex.ActionConfig{
Method: "GET", Path: "/search-community",
Middleware: []maniflex.MiddlewareFunc{communityAuth},
Handler: func(ctx *maniflex.ServerContext) error {
hits, err := ctx.Search(maniflex.SearchOptions{
Query: ctx.QueryParam("q"),
Models: []string{"Post", "Comment"}, // explicit, app-authorised set
Limit: 20,
})
if err != nil {
ctx.Abort(400, "SEARCH_ERROR", err.Error())
return nil
}
ctx.Response = &maniflex.APIResponse{StatusCode: 200, Data: hits}
return nil
},
})
type SearchOptions struct {
Query string // the search text; blank → no-op (no results)
Models []string // models to search; empty → all GlobalSearchable models
Limit int // max merged results; <= 0 → 20
PerModelLimit int // fairness cap (see Merge order); <= 0 → pure relevance
}
type SearchResult struct {
Model string `json:"model"` // the model the hit came from
ID string `json:"id"` // primary key of the matched row
Snippet string `json:"snippet"` // excerpt of the matched text
Score float64 `json:"score"` // relevance, higher = more relevant
}
func (c *ServerContext) Search(opts SearchOptions) ([]SearchResult, error)
With an explicit Models list each named model only needs mfx:"searchable"
fields — it does not need GlobalSearchable. That flag governs only the
built-in endpoint below; the Action path is yours to authorise. ctx.Search
participates in ctx.Tx when one is active and excludes soft-deleted rows.
The built-in GET /search endpoint
Enable it explicitly, then opt models in with ModelConfig.GlobalSearchable:
server.EnableGlobalSearch() // mounts GET {PathPrefix}/search
server.MustRegister(
Post{}, maniflex.ModelConfig{GlobalSearchable: true},
Comment{}, maniflex.ModelConfig{GlobalSearchable: true},
Product{}, maniflex.ModelConfig{GlobalSearchable: true},
)
GlobalSearchable requires the model to declare at least one mfx:"searchable"
field; registration fails otherwise.
GET /api/search?q=wireless+headphones
GET /api/search?q=invoice&limit=10&models=Product
| Parameter | Default | Notes |
|---|---|---|
q | — | Required. Blank → 400 INVALID_QUERY. |
limit | 20 | Clamped to the configured maximum (default 100). |
models | all | Comma-separated subset; each name must be a GlobalSearchable model, else 400. |
The response is the standard envelope with a flat array of hits, ordered by score descending:
{
"data": [
{"model": "Product", "id": "9f8…", "snippet": "wireless …", "score": 0.61},
{"model": "Post", "id": "1a2…", "snippet": "… wireless", "score": 0.18}
]
}
Configure the route and limits via EnableGlobalSearch:
server.EnableGlobalSearch(maniflex.GlobalSearchConfig{
Path: "/search",
DefaultLimit: 20,
MaxLimit: 100,
})
Authorization
The endpoint runs only the global Auth pipeline step — it does not apply
per-model auth or tenancy middleware. Gate it with Pipeline.Auth middleware,
either globally or scoped to the search operation:
server.Pipeline.Auth.Register(requireLogin, maniflex.ForOperation(maniflex.OpSearch))
An intentionally public search endpoint can set
GlobalSearchConfig.AllowPublic; this is an explicit declaration for
ValidateProduction, not an authorization middleware. Production validation
also requires MaxLimit to remain positive.
Because per-model row-level rules are not applied, only set GlobalSearchable on
models that are safe to expose this way. When you need per-model authorisation,
build a scoped Action with ctx.Search instead (see above) and attach your own
middleware. Middleware registered for OpSearch on the Deserialize, Validate,
Service, or DB steps never runs (the endpoint skips them) and is reported with a
startup warning.
Merge order
A deployment uses one database driver, so every model shares one ranking function and the scores are directly comparable; results merge by score descending.
By default the merge is pure relevance — if one model’s hits dominate, the
result can be entirely from that model. Set PerModelLimit to give each model a
fair share: the merge first takes up to PerModelLimit of each model’s
top-scoring hits, then backfills any remaining slots up to Limit from the
leftovers (best score first, regardless of model). It is a fair-chance floor, not
a hard ceiling — the result still fills to Limit when some models have fewer
hits.
Cross-model scores are a heuristic: bm25 and
ts_rankdepend on each table’s own corpus statistics, so a common term can score higher in a table where it is rarer. UsePerModelLimitwhen you want guaranteed representation across models rather than a pure score ranking.
Response Envelope
Every response from a generated route follows one of two shapes — the data envelope or the error envelope. This page documents both, along with the status codes the framework emits.
Success envelope
A successful single-row response — OpRead, OpCreate, OpUpdate:
{
"data": {
"id": "8c1a…",
"title": "First post",
"created_at": "2026-05-19T12:34:56Z",
"updated_at": "2026-05-19T12:34:56Z"
}
}
A successful list response carries the same data key plus a meta block:
{
"data": [
{ "id": "8c1a…", "title": "First post", ... },
{ "id": "9d2b…", "title": "Second post", ... }
],
"meta": {
"total": 137,
"page": 1,
"limit": 20,
"pages": 7
}
}
meta field | Meaning |
|---|---|
total | total matching rows across all pages |
page | page number returned (1-based) |
limit | rows per page |
pages | total page count, computed as ceil(total/limit) |
When a request uses cursor (keyset) pagination
(?cursor=), the meta block takes a different shape — no total/page/pages
(the count is skipped):
{ "data": [ ... ], "meta": { "limit": 20, "next_cursor": "eyJ2Ijoi...", "has_more": true } }
With ?count=false - total and pages are absent since nothing is counted, replaced with has_more:
{ "data": [ ... ], "meta": { "page": 2, "limit": 20, "has_more": true } }
So meta is a union of three shapes and a client should read its keys rather
than assume them: total is present only when a count ran.
DELETE returns 204 No Content with no body.
Error envelope
Every error response uses:
{
"error": {
"code": "VALIDATION_ERROR",
"message": "one or more fields failed validation",
"details": [
{ "field": "email", "message": "field \"email\" is required" },
{ "field": "password", "message": "must be at least 8 characters" }
]
}
}
| Field | Meaning |
|---|---|
code | machine-readable identifier (e.g. NOT_FOUND, CONFLICT) |
message | human-readable summary |
details | optional structured payload — an array of {field, message} objects for per-field errors |
details is an array wherever it is present, including on a 409 CONFLICT from
a unique violation and on the 422 a database-side NOT NULL violation raises.
Each was a bare object once — the 409 until v0.3.0, so a duplicate value answered
in two shapes depending on whether the database or validate.UniqueField caught
it; a client that ranged over details had to type-switch first.
The field on that 422 names the JSON field the client sent, as the Validate
step’s own required check does, not the database column the driver reported — so
a model with db:"headline_col" json:"headline" answers with headline.
A composite unique constraint contributes one entry per column, so a form can highlight every input involved:
{
"error": {
"code": "CONFLICT",
"message": "unique constraint violation",
"details": [
{
"field": "phone_number",
"message": "the combination of phone_number, owner_id is already taken"
},
{
"field": "owner_id",
"message": "the combination of phone_number, owner_id is already taken"
}
]
}
}
The message names the combination rather than a column because neither column is in violation on its own — that phone number is fine, the pair is not.
The catalogue of built-in codes is in Error Handling.
Status codes
| Operation | Success | Notable errors |
|---|---|---|
OpList | 200 OK | 400 INVALID_QUERY |
OpRead | 200 OK | 404 NOT_FOUND |
OpCreate | 201 Created | 400 INVALID_JSON, 409 CONFLICT, 422 VALIDATION_ERROR |
OpUpdate | 200 OK | 404 NOT_FOUND, 409 CONFLICT, 422 VALIDATION_ERROR |
OpDelete | 204 No Content | 404 NOT_FOUND |
HEAD mirrors the GET for the same URL with the body suppressed: same status
(including 404 for a record that does not exist), same headers, same middleware
— just no body.
OPTIONS returns 204 No Content with an Allow header listing the methods the
route accepts.
Headers
Every response carries:
| Header | Source |
|---|---|
Content-Type: application/json | always |
X-Request-Id | echoed from chi’s RequestID middleware |
X-Service-Name | when Config.ServiceName is set |
Custom middleware can add more — see Response Middleware
for AddHeader, CORSHeaders, Cache, and friends.
X-Request-Id
A request that arrives with one keeps it, so an id assigned upstream stays the same value through the logs, the audit record and a versioned model’s history row. A request without one is given a generated id.
An incoming id is adopted only if it is at most 128 characters of alphanumerics
and - _ . : / + = — which covers a UUID, plain hex, base64url, AWS X-Ray’s
Root=1-… and the framework’s own host/base64-000001. Anything else is
replaced with a generated id, because the value is echoed, logged on every line
the request produces, and stored; an unbounded one is a client-chosen string in
each of those places. The response header always says which id was actually used.
It is a correlation hint, not an identity. Any client may send any acceptable id, including one it saw elsewhere, so two unrelated requests can report the same value. Never treat it as unique or as evidence of who made the call.
Computed (virtual) fields
Server.AddComputedField registers a derived field that appears in every
read response (create echo, single read, update echo, list rows) without
being stored:
server.MustAddComputedField("Product", "stock_level",
func(ctx *maniflex.ServerContext, row map[string]any) (any, error) {
return stockService.CurrentLevel(ctx.Ctx, row["id"].(string))
})
The function runs in the Response step after the DB row has been converted
to JSON keys, so row’s keys are JSON field names. It receives the
*ServerContext, so it can reach ctx.Tx, ctx.GetModel and ctx.Auth.
Computed fields:
- Cannot be filtered or sorted — they’re materialised on output only.
- Are name-collision-checked at registration: a name that matches a real model field, or that’s already registered as computed, is rejected.
- Tolerate errors per-row — a non-nil error from the function is logged and the field is omitted from that row; the rest of the response is unaffected.
- Tolerate a panic the same way — a panicking callback is logged
(with its stack, at
ERROR) and its field omitted, exactly as a returned error is. It costs that one field on that one row: the record is still returned and the model’s other computed fields still resolve. This holds for batch callbacks too, so converting a per-row field to a batch one does not change what a bad row costs. - Run on every read path that goes through the default Response step, including the create and update echoes.
- Must be goroutine-safe. On a multi-row page the per-row callbacks run concurrently (bounded to 8 at a time), so a callback that writes to a captured variable, a shared map, or any other state must synchronise it. Batch callbacks are called once for the whole page and are not affected.
Do not use
ctx.Txfrom a per-row callback. A*sql.Txis not safe for concurrent use, and the callbacks for one page share the sameServerContext. Reads throughctx.GetModelormaniflex.Readenlistctx.Txwhen one is open, so a computed field that reads while the request is in a transaction is exactly this case. Resolve such a field withAddBatchComputedField, which runs once and sequentially.
- Appear in the OpenAPI spec as read-only properties of the model’s response schema (never in a create or update body).
Use them for derived values that change too often to denormalise (stock level, leave balance, account balance) or that depend on external systems.
Batch resolution (AddBatchComputedField)
AddComputedField runs once per row, so a resolver that queries is an
N+1: a 50-row page costs 50 round-trips. AddBatchComputedField resolves
the whole page in one call instead — this is what lets a generated
GET /store-sites return an item_count without a hand-written action:
server.MustAddBatchComputedField("StoreSite", "item_count",
func(ctx *maniflex.ServerContext, rows []map[string]any) ([]any, error) {
ids := make([]any, len(rows))
for i, r := range rows {
ids[i] = r["id"]
}
counts, err := itemCountsBySite(ctx, ids) // ONE query for the page
if err != nil {
return nil, err
}
out := make([]any, len(rows))
for i, r := range rows {
out[i] = counts[r["id"].(string)]
}
return out, nil
},
maniflex.ComputedSchema(&maniflex.OASSchema{Type: "integer"}))
The callback must return exactly one value per row, positionally aligned
to rows. A length mismatch is logged and the field is omitted from the
whole response rather than landed on the wrong records — an absent column is
diagnosable, a misaligned one is not.
One registration serves every read path: a single read and the create/update echo call it with a one-row slice, and an export calls it once per chunk of rows (so a batch field costs one call per 500 records there, not one per record).
Prefer the batch form for anything that touches a database. Per-row
resolvers run concurrently across a page, but bounded at 8 at a time — the
fan-out used to be one goroutine per row with no ceiling, so a 100-row page
fired 100 concurrent round-trips and the load scaled as page-size ×
concurrent-requests. The bound stops that from being unbounded; it does not
stop it from being an N+1. Note too that work through ctx.Tx is serialised
by the transaction’s single connection, so per-row parallelism buys nothing
there.
Declaring the type
Both callbacks return any, so the framework cannot infer a computed
field’s type. Without ComputedSchema the field still appears in the spec
(read-only) but carries no type — a generated client knows it exists but not
what it holds. maniflex.ComputedSchema(&maniflex.OASSchema{…}) declares it.
Typed variants
maniflex.AddComputedField[T] and maniflex.AddBatchComputedField[T] take
the loaded record(s) as *T / []*T instead of JSON maps:
maniflex.AddBatchComputedField(server, "StoreSite", "item_count",
func(ctx *maniflex.ServerContext, sites []*StoreSite) ([]any, error) {
// …one query, one value per site
})
Replacing the envelope
The default shape is good enough for most APIs, but if you integrate with a
client that expects a different layout, register response.Envelope from the
catalogue:
import "github.com/xaleel/maniflex/middleware/response"
server.Pipeline.Response.Register(
response.Envelope(func(ctx *maniflex.ServerContext, data any, meta *maniflex.ResponseMeta) any {
return map[string]any{
"result": data,
"paging": meta,
"trace_id": ctx.TraceID,
}
}),
)
Error responses are unaffected — they always use the {"error": {…}} shape so
clients can distinguish success from failure with a single key check.
OpenAPI Spec
maniflex generates an OpenAPI 3.1 specification from the registered models. The HTTP endpoint is private-by-default: the zero-value configuration does not mount it. Explicitly publish it or place it behind a shared documentation access policy. The spec is derived from the same struct tags that drive validation and querying, so it cannot drift from the actual behaviour of the API.
The endpoint
To publish the spec intentionally, opt in when constructing the server:
server := maniflex.New(maniflex.Config{
Documentation: maniflex.DocumentationConfig{Public: true},
})
It then lives at /openapi.json under the configured PathPrefix:
curl localhost:8080/api/openapi.json
The response is a full OpenAPI 3.1 document — info, paths, components,
the lot. It updates automatically every time you register a model, change a
tag, or alter maniflex.Config. There is no separate codegen step.
What is generated
For every registered model, the spec includes:
-
Five paths —
/<table>(GET,POST) and/<table>/{id}(GET,PATCH,DELETE). -
One attachment path per
mfx:"file"field when storage is configured:GET /<table>/{id}/<file_field>withapplication/octet-stream(plus any MIME types from the field’saccept:list). See Per-model attachment routes. -
Every opt-in route the model actually mounts, and only those — each appears when the
ModelConfigflag that mounts it is set, and is absent otherwise:Path Mounted when GET /<table>/exportExportEnabledGET /<table>/aggregateAggregateEnabledPOST /<table>/{id}/restoreRestoreEnabledand the model soft-deletesGET /<table>/{id}/historyVersionedPOST /<table>/<file_field>/upload-urlthe field is mfx:"file,upload:presigned"and storage is configuredThe condition is the same one the router checks, and
TestOpenAPIRouteParitywalks the mounted routes to prove it: a route the spec omits and a path the spec invents both fail the build. That guard exists because four of these routes shipped mounted-but-undocumented, so a generated client could not reach an endpoint that had worked for releases. -
Three schemas — a full response shape, a create body shape, and an update (patch) body shape. The three differ by which fields are visible: the create shape drops
readonlyfields; the update shape additionally dropsimmutablefields. -
Standard query parameters for list endpoints —
page,limit,filter,sort,include. -
Field metadata taken from
mfx:tags —enum,min,max,required,readOnly,writeOnly. -
JSON / map fields — a field whose type is a
mapwith string keys is documented as a free-form{"type": "object"}. This coversmap[string]any,map[string]string, and named types over them such astype JSONObject map[string]any— the value type is not inspected. To emit a more precise shape instead, give the type its own schema (see Schemas for custom types). -
Relation fields — for each relation to a registered model, the full response schema embeds the related schema by reference (
$ref), shown when?include=requests it. A relation whose target model is not registered — for example a bareRelatedIDfield with noRelatedmodel — is omitted rather than emitting a dangling reference that would break spec validators and client generators. -
Response statuses beyond the obvious ones, derived from what this deployment can actually answer with — see Statuses the pipeline produces.
hidden fields are excluded entirely from every schema. writeonly fields
appear in the create and update schemas with writeOnly: true, but not in the
response shape.
Statuses the pipeline produces
An operation’s own shape gives it the obvious statuses: 201 and 422 on a
create, 404 on a read. Everything else a request can meet comes from the
pipeline and the configuration, so it is derived per operation rather than
assumed:
| Status | Documented when |
|---|---|
401, 403 | any Pipeline.Auth middleware applies to that model and operation |
409 | a write on a model with a unique field or a relation; a delete on a model with a restrict relation |
412 | ModelConfig.OptimisticLock, on PATCH and DELETE |
500 | always |
503 | Config.MaxConcurrentRequests is set — carries a Retry-After header |
504 | Config.QueryTimeout is set |
Auth is not special-cased: a middleware on the Auth step implies 401/403
because refusing is what that step is for, so auth.JWTAuth needs no OpenAPI
awareness of its own. A server with no auth middleware documents neither — which
is the point. A fixed list would tell clients to handle a 401 that cannot
happen, and a reviewer reading the spec as a security inventory would be misled
in one direction or the other.
With OptimisticLock set, both halves of the conditional-write handshake are
documented: the item GET carries an ETag response header, and PATCH /
DELETE take an optional If-Match request header. Optional because a request
without it writes unconditionally rather than failing.
Statuses your own middleware produces
The framework cannot know that your guard answers 402. Declare it on the
registration, where it inherits the same ForModel / ForOperation filters that
decide where the middleware runs:
server.Pipeline.Validate.Register(stockGuard,
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.DocumentsResponse(409, "Out of stock", nil),
)
That 409 appears on POST /orders and nowhere else. A declaration replaces the
derived entry for the same status, so this is also how to say what your 409
means rather than accepting the generic wording.
For a status no registered middleware produces — one a handler returns itself, or
one a proxy in front of the server can return — use openapi.AddResponse:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddResponse(
openapi.OperationTarget{Path: "/orders", Method: "post"},
402, "Payment required", nil),
maniflex.After,
)
Prefer DocumentsResponse when a middleware you register is what produces the
status. The Path above is a literal, so it stops matching silently if the
model’s table name or the route ever changes.
Schemas for custom types
Field types are mapped to OpenAPI schemas by their Go kind: strings, booleans,
the integer and float families, time.Time (as date-time), any string-keyed
map (as a free-form object), and slices and arrays (as array, with the
element type inferred — except a byte slice, which encoding/json base64s and
which is therefore documented as {"type": "string", "format": "byte"}).
A field whose type falls outside these — a custom struct, or any type with a
non-obvious JSON representation — is published with no type constraint, and
the server says so at startup: a warning naming the model and the field, and a
startup error under Config.Strict.
Such a field is not omitted. It used to be, and a spec that omits a field says
two false things about it. A generated client has no type for it — and because
the omission happened before the required list was built, a field the server
demands was absent from required as well, so the spec described a request
that always fails. An unconstrained schema says less than a real one, but
everything it says is true.
To document such a type, make it implement the ObjectWithSchema interface:
type ObjectWithSchema interface {
Schema() *maniflex.OASSchema
}
Whenever the generator encounters a field of that type it calls Schema() and
uses the returned value verbatim — taking precedence over the built-in kind
mapping, so this also lets you override the default object shape a JSON/map
column would otherwise get. Either a value or a pointer receiver works.
For example, to document a Geo JSON column as a structured object instead of a
free-form one:
type Geo struct {
Lat float64 `json:"lat"`
Lng float64 `json:"lng"`
}
func (Geo) Schema() *maniflex.OASSchema {
return &maniflex.OASSchema{
Type: "object",
Properties: map[string]*maniflex.OASSchema{
"lat": {Type: "number", Format: "double"},
"lng": {Type: "number", Format: "double"},
},
}
}
A Geo field now renders with that exact shape in the response, create, and
update schemas. A pointer field (*Geo) is additionally made nullable.
Custom actions
Actions — custom endpoints registered with
server.Action — are included in the spec alongside the generated model
routes. Each contributes its method, path, and any {...} path parameters
automatically. Fill in ActionConfig.OpenAPI to document request and response
bodies (inferred directly from Go structs), extra query parameters, security
requirements, and a long-form description. See
Documenting an action in OpenAPI.
The OpenAPI pipeline
The spec endpoint has its own three-step pipeline, parallel to the model-route pipeline:
OpenAPI.Auth → OpenAPI.Generate → OpenAPI.Response
| Step | Purpose |
|---|---|
| Auth | OpenAPI-specific middleware (OpenAPIMiddlewareFunc) for format-specific gates. |
| Generate | Builds the spec from the registry. After-position middleware mutates it. |
| Response | Serialises the spec to JSON. |
This is reached via server.Pipeline.OpenAPI.*. See
OpenAPI Middleware for the catalogue of
spec-shaping helpers — SetTitle, AddServer, AddSecurityScheme,
AddExtension.
Securing generated documentation
Use the shared router-level documentation policy to protect both OpenAPI and
AsyncAPI. AdaptAuth safely bridges request-level authentication middleware;
the middleware shares one context, so JWTAuth can populate the identity used
by RequireRole:
const jwtSecret = "replace-with-a-strong-32-byte-secret"
server := maniflex.New(maniflex.Config{
Documentation: maniflex.DocumentationConfig{
Middleware: []maniflex.HTTPMiddleware{
maniflex.AdaptAuth(
auth.JWTAuth(jwtSecret),
auth.RequireRole("internal"),
),
},
},
})
This mounts generated documentation behind the policy; it does not affect model
routes. Existing Pipeline.OpenAPI.Auth middleware remains supported for
OpenAPI-only custom gates, but it takes OpenAPIMiddlewareFunc, not the
model-route MiddlewareFunc returned by auth.JWTAuth and auth.RequireRole.
The framework does not force wildcard CORS headers on specifications. Configure
cross-origin access explicitly through Documentation.Middleware or global
HTTPMiddlewares.
Viewing the spec
The framework ships a Scalar API Reference viewer at
static/openapi.html. Point StaticDir
at the directory holding it (maniflex.Config{StaticDir: "static"}) and it is
served at http://localhost:8080/static/openapi.html, loading /api/openapi.json
directly. Public documentation works without additional viewer configuration.
For protected documentation, configure the viewer to send credentials and protect
the static viewer itself through your edge proxy or global HTTP middleware.
For tooling integration, the JSON document at /openapi.json is consumable by
any OpenAPI 3.1-compatible client generator, mock server, or contract testing
framework.
Customising the spec
Most customisation is one-line, through the OpenAPI Middleware helpers. For deeper edits, write your own middleware:
server.Pipeline.OpenAPI.Generate.Register(func(ctx *maniflex.OpenAPIContext, next func() error) error {
if err := next(); err != nil {
return err
}
// ctx.Spec is the just-generated *OpenAPISpec — mutate freely.
ctx.Spec.Info.Description = "Contact the API team at [email protected]."
return nil
}, maniflex.After)
The full set of types (OpenAPISpec, OpenAPIInfo, OASSecurityScheme, …) is in
the maniflex package.
Database Backends
maniflex ships two database adapters, both built on database/sql and sharing a
single SQL core (db/sqlcore). They expose the same interface; switching
between them is one import line.
| Adapter | Module | Driver |
|---|---|---|
| SQLite | maniflex/db/sqlite | modernc.org/sqlite — pure Go, no CGo |
| PostgreSQL | maniflex/db/postgres | github.com/lib/pq |
Each adapter lives in its own Go module so a project only pulls in the driver it actually uses.
SQLite
The default choice for development, tests, and small deployments. The pure-Go
driver means no CGo and no external service — go run . is enough to start a
local server with a working database.
import "github.com/xaleel/maniflex/db/sqlite"
db, err := sqlite.Open("./app.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
Common DSNs:
| DSN | Effect |
|---|---|
./app.db | persistent file in the working directory |
:memory: | per-process in-memory database; vanishes on shutdown |
SQLite is single-writer by design. The framework serialises writes through one connection internally; reads run on a pool. This is plenty for most internal tools and many production APIs.
Write connections open their transactions with BEGIN IMMEDIATE (the
_txlock=immediate DSN option, applied for you). That is what makes a
read-then-write transaction — LockForUpdate, an If-Match check,
mfx:"lock_scope" — behave the way it does on Postgres: a second transaction
waits at its BEGIN rather than reading the same stale row and then failing, or
overwriting, on the way out. Spell out your own _txlock= in the DSN and yours
is kept.
PostgreSQL
The recommended adapter for any multi-process deployment. It supports
genuine concurrent writers, real FOR UPDATE locks, and read replicas.
import "github.com/xaleel/maniflex/db/postgres"
// Open(writeDSN, readDSN, registry) — positional arguments.
db, err := postgres.Open(
"postgres://user:pass@host/db?sslmode=require", // write DSN
"postgres://user:[email protected]/db?sslmode=require", // read DSN (optional)
server.Registry(),
)
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
Pass an empty read DSN ("") to route reads to the primary. The adapter selects
the appropriate pool per request based on the operation — OpList and OpRead
go to the read pool, everything else to the write pool.
For connection-pool and session tuning use postgres.OpenWithConfig(writeDSN, readDSN, registry, writePool, readPool, session) — see
PostgreSQL in Production for replica lag
handling and SSL.
Switching between them
The adapter is the only thing that changes; nothing else in the application needs to know which database is in use:
- import "github.com/xaleel/maniflex/db/sqlite"
+ import "github.com/xaleel/maniflex/db/postgres"
- db, err := sqlite.Open("./app.db", server.Registry())
+ db, err := postgres.Open(os.Getenv("DB_URL"), "", server.Registry())
Models, middleware, and queries are portable across both backends because they
go through database/sql + the shared db/sqlcore adapter. Migrations
emitted by AutoMigrate use a portable subset of SQL.
AutoMigrate
Unless Config.DisableAutoMigrate is set (migration runs by default), the adapter:
- Creates any table that does not yet exist for a registered model.
- Adds any column that exists on the struct but not in the table.
- Logs a warning for columns that exist in the table but not on the struct (the framework never drops columns automatically).
- Logs a warning for a column whose type no longer matches its model field (the framework never rewrites columns automatically).
- Creates indexes declared in
ModelConfig.Indicesor auto-generated formfx:"scheduled"fields.
AutoMigrate adds; it does not rewrite. Change a field’s Go type — int to
string, say — and the existing column keeps the type it was created with. You
get a warning naming the table, the column, and both types on every startup, but
the schema is left alone: rewriting a column can lose data and locks the table
while it runs, so the conversion (and whatever backfill it needs) belongs in an
explicit, versioned migration you run against the database, not in a startup
routine that has to guess. Until that migration runs, reads and writes of the
field can fail against the old column.
AutoMigrate is suitable for development and many small deployments. For
larger systems, set DisableAutoMigrate: true and either manage the schema with
a dedicated migration tool or migrate from a single process — see below.
Migrations in production
Server.ValidateProduction requires Config.DisableAutoMigrate, so a
production server does not migrate as it boots. That flag is about the
automatic migration — the one Start runs as a side effect. MigrateOnly
is the explicit call and runs regardless, which is what lets one binary and one
config serve both roles:
cfg.DisableAutoMigrate = true // every replica; ValidateProduction requires it
server := maniflex.New(cfg)
// … register models, actions, middleware …
if os.Getenv("MIGRATE_ONLY") == "1" {
log.Fatal(server.MigrateOnly(ctx)) // the init container / pre-deploy job
}
log.Fatal(server.Start()) // the replicas, which migrate nothing
MigrateOnly validates and seals the whole configuration before it touches
schema, so a misconfigured application fails the job rather than half-migrating.
To validate and seal without any schema work, call Handler or
StartServices instead.
What is safe under a rolling deploy
AutoMigrate only ever adds. That is the property that makes it rolling-safe:
while old and new replicas run side by side, a new column is invisible to
the old ones, and a new table is unreferenced by them.
Everything else is yours, and is not rolling-safe:
| Change | Who does it | Safe mid-rollout |
|---|---|---|
| Add a field, model, or index | AutoMigrate | yes |
| Change a field’s type | you, explicitly | no — old and new replicas disagree about the column |
| Remove a field | you, explicitly | not until every old replica is gone |
| Rename a field | you, explicitly | no — it is a drop plus an add to the database |
For a removal, the two-phase shape is the usual one: ship the code that stops writing the column, complete the rollout, then drop it. For a type change, add a new column, backfill, switch reads, and drop the old one — each phase its own deploy.
Concurrent replicas
Several processes may run migration at once without corrupting each other. Each
table’s create-introspect-alter sequence is one transaction, columns are added
with IF NOT EXISTS (or duplicate-tolerantly on SQLite), and a foreign key
another process added first is accepted rather than treated as an error.
Prefer migrating from one process anyway. Concurrency-safe means the losing replica does not break; it does not mean the work is coordinated, and there is no migration lock or leader election here — the framework does not attempt to sequence a schema change against the replicas that are mid-rollout.
The DBAdapter interface
Both shipped adapters implement maniflex.DBAdapter. Custom backends — an HTTP
data service, a remote API, a different SQL database — implement the same
interface and inject the result through server.SetDB(myAdapter). The
interface is in db.go.
Adapters are compared by identity
The framework uses == on DBAdapter values to decide whether two models share
a database — that is what stops a transaction from silently spanning two of them.
So a custom adapter must:
- use a pointer receiver — return
*MyAdapter, notMyAdapter. Two separately constructed value-type adapters with equal fields compare equal, so the framework would treat two databases as one and let a transaction span them. That fails silently, not loudly. - stay comparable — comparing interface values whose dynamic type is not comparable (a struct holding a map, slice, or func, used as a value) panics at run time. A pointer receiver gives you this for free.
Nothing in the type system enforces either, which is why it is written down here
and on the DBAdapter godoc.
Rendering filters
A SQL-backed adapter does not have to write its own filter renderer, and should
not. maniflex.BuildFilterSQL turns a []*FilterExpr into the body of a
WHERE clause — grouping, operator rendering, LIKE escaping, IN expansion,
field resolution and value coercion — and maniflex.Quote quotes identifiers
the way the framework does:
p := sqlcore.NewPlaceholderBuilder(maniflex.SQLite) // or your own PlaceholderBinder
where := maniflex.BuildFilterSQL(model, qp.Filters, maniflex.SQLite, p)
sql := "SELECT * FROM " + maniflex.Quote(model.TableName)
if where != "" {
sql += " WHERE " + where
}
rows, err := db.Query(sql, p.Args()...)
PlaceholderBinder is one method, Add(any) string, returning the placeholder
text and recording the value. Implementations must append in call order:
SQLite binds ? by where it appears in the statement text, so a binder that
registered a placeholder out of order would misalign every argument after it
without raising a syntax error.
This is shared rather than copied for a reason. The framework used to carry two filter renderers, and every capability one grew that the other did not shipped as a silently wrong answer — three separate P0 bugs before they were merged. A third copy inside an adapter is the same trap.
BuildFilterSQL renders a nested-relation filter as "relation"."column",
which assumes your FROM clause joined that relation. An adapter or query path
that does not join must reject nested filters rather than pass them through.
Per-model adapter routing
Config.DB sets the default adapter. Individual models can override it by
passing ModelConfig.Adapter:
ordersDB, _ := postgres.Open(ordersDSN, "", server.Registry())
inventoryDB, _ := postgres.Open(inventoryDSN, "", server.Registry())
server.MustRegister(
Order{}, maniflex.ModelConfig{Adapter: ordersDB},
InventoryItem{}, maniflex.ModelConfig{Adapter: inventoryDB},
User{}, // unrouted — falls back to Config.DB
)
“Distinct” here means distinct under ==. Two models that share a database must
be given the same adapter value, not two opened against the same DSN — those
are separate connection pools and separate transactions, and nothing tells the
framework they point at one database.
The framework treats each distinct adapter as its own database:
- AutoMigrate runs once per adapter, with a filtered registry view so each
adapter only sees the models routed to it. Tables for
Orderare never created on the inventory DB and vice-versa. - CRUD requests (
GET /orders,POST /orders) route throughOrder.Adapter. The DB step picks the per-model adapter automatically. ctx.BeginTx/ctx.RawQuery/ctx.RawExecuse the request’s model adapter, so middleware and custom actions stay on the right DB.ctx.GetModel("OtherModel")uses the target model’s adapter — handy for cross-DB reads — but it cannot share a transaction across adapters: ifctx.Txwas opened ondbAand you callGetModel("X")whereXlives ondbB, the accessor falls back to a non-transactional read againstdbB.
Config.DB is optional when every registered model has its own Adapter.
The server starts cleanly with DB: nil and routes everything through the
per-model overrides. If any model is unrouted and Config.DB is also nil,
startup fails with a clear error naming the unrouted models.
Constraint: transactions are adapter-scoped
A single database transaction cannot span two adapters. Two consequences:
maniflex.Batchrejects ab.Create("X", ...)call whereXroutes to a different adapter than the batch transaction was opened on. The error message points topkg/sagaas the cross-adapter pattern.- Manually-opened
ctx.Txonly protects writes against the request’s own model adapter. Cross-adapter writes throughctx.GetModel(...)happen outside that transaction.
For coordinated writes across databases, use pkg/saga —
compensating transactions are the supported pattern.
Choosing
| Need | Pick |
|---|---|
| Quick start, tests, small single-process services | SQLite |
| Multi-process deployment, real concurrency, replicas | PostgreSQL |
| Both (the codebase will outgrow SQLite) | SQLite locally, Postgres in production — same code |
Configuration
maniflex.Config is the single struct passed to maniflex.New. Every field has a
sensible default; populate only the ones that differ from those defaults.
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
Reading the boolean flags
Config mixes positive and negative flag names — Strict and Public alongside
DisableAutoMigrate, StaticDisabled, ProbeConfig.Disabled. That is one rule
rather than two conventions:
Every boolean is named so its zero value is the behaviour you want if you say nothing.
A feature that is off until asked for gets a positive name, so leaving it unset
leaves it off — Strict, TrustProxyHeaders, Documentation.Public. A feature
that is on by default gets a negative one, so leaving it unset leaves it
working — DisableAutoMigrate, ProbeConfig.Disabled. Naming those positively
would mean AutoMigrate: false silently turning migration off for anyone who
never set it, which is the failure the negative spelling exists to prevent.
StaticDisabled spells the same way for a third shape. Static serving is opt-in
— it needs StaticDir — so the flag turns off something you already asked for,
letting an app that sets StaticDir unconditionally still disable serving from
an env var without clearing the field.
So an empty Config{} is always the intended default, and every field you set is
a deliberate departure from it. Read a negative name as evidence that leaving the
field unset never turns anything off — not that the feature is on by default.
Server
| Field | Default | Purpose |
|---|---|---|
Port | 8080 | TCP port the HTTP server binds to; outside 1–65535 is refused at startup |
PathPrefix | /api | URL prefix prepended to generated model and documentation routes; normalised |
Documentation | zero value (unmounted) | explicitly publish generated OpenAPI/AsyncAPI documents or protect both with shared middleware |
ServiceName | "" | service identifier added to logs, audit records, and the X-Service-Name response header |
StaticDir | "" | filesystem directory served as static files; empty serves nothing (opt-in). Relative paths resolve against cwd |
StaticPrefix | /static | URL prefix the static directory is mounted under, at the router root |
StaticDisabled | false | turn static file serving off even when StaticDir is set |
StaticDirectoryListing | false | serve a listing for a static directory with no index.html; 404 otherwise |
HTTPAccessControlled | false | assert that non-empty HTTPMiddlewares protects every route for ValidateProduction; does not install auth |
Both prefixes are normalised to one leading slash and no trailing one, so
"api", "/api/" and "//api" all mean /api, and "/" mounts the API at the
router root — chi panics on a pattern with no leading slash, and //api would
otherwise mount every route a doubled slash deep, where no client would find it.
Read the value back from the Config for the canonical form. A StaticPrefix
containing {, } or * cannot be guessed at and is refused at startup
instead: the static mount serves a literal path and takes no URL parameters.
PathPrefix does not affect /static or /files; those are mounted at the
router root. The probe endpoints — /live, /ready, and /health — do sit
under it, so the default prefix serves them at /api/live, /api/ready, and
/api/health. See Static Files for the
static serving options.
HTTP timeouts
The framework owns the http.Server, and net/http gives that struct no
deadlines by default. These are the ones it sets on your behalf:
| Field | Default | Purpose |
|---|---|---|
ReadHeaderTimeout | 10s | how long a connection may take to send its request headers |
IdleTimeout | 120s | how long a keep-alive connection may sit idle between requests |
BodyReadTimeout | 30s | time to wait for the next chunk of a request body, refreshed on every read |
ReadTimeout | 0 (unbounded) | time to read an entire request, headers and body |
WriteTimeout | 0 (unbounded) | time to write a response |
ReadHeaderTimeout is the slowloris defence. Without it a client can hold a
connection open forever by dribbling one header byte at a time, and enough such
connections exhaust the server’s file descriptors without a single request ever
reaching the pipeline. Set a negative value to disable a timeout (which is
what net/http reads as “no deadline”) — only sensible behind a proxy that
already bounds header reads.
ReadTimeout and WriteTimeout are deliberately left unset, because both are
whole-request deadlines rather than idle deadlines:
- A
ReadTimeoutcaps how long a client may take to upload, so a large file over a slow link is severed mid-transfer. - A
WriteTimeoutcovers the entire response, so any value at all would cut a long-lived stream —realtime.SSEHandler, a large download — off at that mark.
Set them when you know your request sizes and have no streaming endpoints. The
header phase stays bounded by ReadHeaderTimeout either way.
That left the body phase unbounded, which neither of the other defences
covers: ReadHeaderTimeout is satisfied the moment the headers are complete, and
the body size cap counts bytes rather than seconds. A client could announce a
Content-Length well inside the limit, send none of it, and hold a connection, a
goroutine and a file descriptor for as long as it liked.
BodyReadTimeout closes that. It is an idle bound — the wait for the next
chunk, refreshed on every read — which is what makes it safe on by default where
ReadTimeout is not: an upload that keeps making progress never trips it,
however slow the link or however large the file. These are the semantics of
nginx’s client_body_timeout. No deadline outlives the body, so a handler that
consumes it and then streams for minutes is unaffected.
The bound stands from the moment the request arrives, not from the first read, so
it covers a body no handler ever touches. It has to: Go’s HTTP server drains what
a client announced and never sent so the connection can be reused, and that drain
is one more read from the same silent client. Without it, any route that answers
without reading — a probe, a 401, a 404, an ordinary GET — could be held
open indefinitely.
It does not bound total upload time. A client that sends one byte just inside
the timeout, forever, is still holding a connection; bounding that means
ReadTimeout, with the trade-off above. Setting ReadTimeout disengages
BodyReadTimeout entirely, since the whole-request deadline is the stricter of
the two. A negative value disables it, as with the other timeouts.
Client address behind a proxy
| Field | Default | Purpose |
|---|---|---|
TrustedProxies | none | CIDRs or bare IPs whose forwarding headers may be believed; a non-empty list enables resolution on its own |
TrustProxyHeaders | false | legacy allowlist-free mode — believes the leftmost X-Forwarded-For from any peer |
Off by default, the client address is the direct TCP peer, which a caller cannot
forge. db.RateLimit, idempotency scoping, and read-audit records all key on it.
Config{TrustedProxies: []string{"10.0.0.0/8"}}
Headers are believed only from those peers, and the X-Forwarded-For chain is
walked right-to-left past them: a proxy appends the address it saw, so the
rightmost entries come from infrastructure and the leftmost is whatever the
client chose to send. An entry in Config.TrustedProxies that does not parse is
a startup error.
The walk stops at the first address no trusted proxy vouched for, and reads only that far. Entries further left are the client’s to write and are never consulted, so junk among them changes nothing. An entry that is the stopping point and cannot be read fails the chain closed — the request keeps its TCP peer — rather than being skipped past, which would walk the search onto the client’s own value.
An entry may carry a source port (203.0.113.7:54321, or [2001:db8::1]:443);
several gateways append one. When a chain from a trusted proxy yields nothing
usable, a throttled warning says so — the symptom otherwise is silent, since
every client behind that proxy collapses into one address for rate limiting and
audit.
The chain is every X-Forwarded-For line joined in the order received, not just
the first. Proxies differ here: nginx and AWS ALB extend the client’s line, while
HAProxy’s option forwardfor adds one of its own, leaving the client’s value on
the line above. Both are one chain per RFC 9110 §5.3, and both are walked whole.
X-Real-IP names a single address, so more than one line of it is believed from
nobody — the request keeps its TCP peer.
TrustProxyHeaders: true with no list keeps the old behaviour for compatibility.
It warns at startup and fails under Strict; prefer the allowlist.
Limits
| Field | Default | Purpose |
|---|---|---|
MaxConcurrentRequests | 0 (unlimited) | how many requests may be in flight at once, server-wide; over the limit is refused, not queued |
MaxConcurrentExports | 4 | how many GET /:model/export requests may run at once, server-wide; negative disables the limit |
QueryLimits | see below | bounds client-controlled URL query and aggregate complexity; ModelConfig.QueryLimits can override individual fields for one model |
An export holds its entire result set in memory until the last byte reaches the
client, so concurrency multiplies the largest allocation the server makes. The
per-model MaxExportRows bounds one export’s rows but not the row width nor the
number in flight; this bounds the product. Requests over the limit are refused
immediately with 503 EXPORT_BUSY and a Retry-After, not queued. See
CSV / XLSX Export.
MaxConcurrentRequests does the same for every request, answering
503 SERVER_BUSY with a Retry-After rather than queueing. Without it the
database pool is the concurrency limit by accident, and it is one that fails by
queueing: a burst is not refused, it waits for a connection while holding a
goroutine, a parsed body and a context, until QueryTimeout fires. Latency then
climbs for every request instead of being paid by the ones that are shed.
The probe endpoints are exempt. A shed /live tells Kubernetes to restart the
container and a shed /ready tells it to take the pod out of the Service, so a
spike the limit was absorbing becomes a restart or an empty endpoint list — and
because replicas saturate at the same moment, it happens to all of them at once.
The limit exists to shed individual requests, not the process. /live answers a
constant and the other two collapse onto one dependency check, so answering them
under load is cheap; use Probes.Middleware if you want a probe bounded anyway.
Size it against the pool rather than against the traffic you hope to serve — a
value far above the pool’s MaxOpenConns only moves the queue. It is off by
default because the right number depends on your pool and hardware, and a wrong
one refuses traffic the server could have served;
ValidateProduction requires it to be
set.
QueryLimits uses these safe defaults:
| Field | Default |
|---|---|
MaxURLBytes | 8 KiB |
MaxFilterClauses / MaxFilterGroups / MaxFiltersPerGroup | 32 / 8 / 8 |
MaxSortFields / MaxSelectFields / MaxIncludes | 8 / 64 / 8 |
MaxAggregateSelectFields / MaxAggregateGroupFields | 16 / 8 |
MaxAggregateFilters / MaxAggregateHaving / MaxAggregateSortFields | 32 / 16 / 8 |
DefaultAggregateRows / MaxAggregateRows | 100 / 200 |
A zero field inherits the default (or the global value in a per-model
override); a negative field disables that individual limit. The global
MaxURLBytes remains a router-level hard ceiling for every route. Oversized
URIs return 414 URI_TOO_LONG; invalid list-query shapes return
400 INVALID_QUERY.
Database
| Field | Default | Purpose |
|---|---|---|
DB | nil | the default DBAdapter. Usually set via server.SetDB(db) after MustRegister. Optional when every model has its own ModelConfig.Adapter — see Per-model adapter routing |
DisableAutoMigrate | false | skip the schema migration Start runs on boot (migration runs by default). Does not affect MigrateOnly — see Migrations in production |
DBWriteURL | "" | DSN for the primary database (informational; populated by ConfigFromEnv) |
DBReadURL | "" | DSN for the read replica (informational) |
QueryTimeout | 0 (unlimited) | per-request deadline applied to all DB calls; exceeding it produces 504 TIMEOUT |
See Database Backends for adapter construction.
SetDB, SetStorage, and SetKeyProvider support two-step initialization, but
they are configuration methods rather than runtime rotation APIs. Call them
before Handler, Start, StartServices, or MigrateOnly; those entry points
seal the server, and a later setter call panics without changing either the
configured or active backend. Construct a new server when these dependencies
must change.
File storage and encryption
| Field | Purpose |
|---|---|
FilesConfig.Storage | maniflex.FileStorage implementation for mfx:"file" fields and the /files endpoints. Required if any model uses file uploads. See File Fields & Uploads. |
FilesConfig.BeforeMiddlewares | []maniflex.MiddlewareFunc wrapping the standalone /files endpoints. Empty = no auth (backward-compatible default); production deployments should populate this with at least an auth middleware. See File Fields & Uploads. |
FilesConfig.AllowPublic | explicit declaration that mounted standalone /files routes are intentionally public; used by ValidateProduction and strict startup validation |
KeyProvider | maniflex.KeyProvider for mfx:"encrypted" fields. Without one, encrypted fields refuse writes with 500 ENCRYPTION_NOT_CONFIGURED. |
Logging
| Field | Default | Purpose |
|---|---|---|
Logger | slog.Default() | logger used for lifecycle, per-request, and adapter messages |
PanicLogger | falls back to Logger | sink for the panic recoverer’s structured panic records |
OnBackgroundPanic | nil | called after a recovered background-goroutine panic — see Graceful Shutdown |
Trace | zero (off) | pipeline tracing — see below |
Logger is used by ctx.Logger(), which adds request_id, trace_id, and
service attributes per request. Route it to a JSON handler in production.
PanicLogger receives panics from both the request path and the background
goroutines the framework runs on your behalf, so one sink sees all of them.
Pipeline tracing
Config.Trace enables verbose debug output of the request pipeline. All trace
output is at DEBUG level through Logger, so the handler must accept
DEBUG records to see anything.
| Sub-flag | Effect |
|---|---|
Enabled | shorthand for Steps + Timings + Aborts |
Steps | enter/exit record per middleware |
Timings | per-middleware elapsed time on exit records |
Aborts | the source file:line of every ctx.Abort call |
Bodies | log field names present in ctx.ParsedBody (opt-in; may expose sensitive field names) |
Skips | log middleware skipped by ForModel/ForOperation filters |
cfg.Trace = maniflex.PipelineTrace{Enabled: true, Skips: true}
Enabled expands into the three standard flags unless one of those three is
already set, in which case it stays out of the way and you get exactly what you
named. Bodies and Skips are additive: setting one does not suppress the
expansion, so the example above gives all four.
cfg.Trace = maniflex.PipelineTrace{Enabled: true, Steps: true} // Steps only
cfg.Trace = maniflex.PipelineTrace{Bodies: true} // Bodies only
Leave Bodies off in production.
Lifecycle
| Field | Default | Purpose |
|---|---|---|
ShutdownTimeout | 30s | maximum time Start() waits for in-flight requests to finish on SIGINT / SIGTERM before forcing the listener closed |
See Graceful Shutdown.
Probes
Three endpoints are mounted under PathPrefix. The first two have fixed
meanings and need no configuration:
| Endpoint | Answers | Behaviour |
|---|---|---|
GET {prefix}/live | is this process alive? | always 200 {"status":"ok"}. No I/O, no dependency, no lifecycle coupling — including throughout the drain |
GET {prefix}/ready | should this process receive traffic? | 503 while starting or stopping, otherwise the result of every dependency check |
GET {prefix}/health | (legacy) | meaning follows HealthCheckDB; kept as a compatibility alias |
Point livenessProbe at /live and readinessProbe at /ready. Pointing both
at /health is what the split exists to end: with HealthCheckDB on, an
unreachable database answers 503 to both, so Kubernetes restarts a process
whose only problem is a dependency it cannot fix by dying.
/ready reports the lifecycle first, without touching a dependency:
503 {"status":"starting"}
503 {"status":"stopping"}
stopping appears the moment shutdown begins, which is what deregisters the pod
from its load balancer while in-flight requests drain. Otherwise every
dependency is checked concurrently:
200 {"status":"ok"}
503 {"status":"not_ready"}
The per-dependency results are withheld by default, because the map names every
ReadinessChecks entry and says which of them are failing — telling anyone who
can reach the probe what your application is built on, and when it is degraded.
Set Probes.PublishReadinessChecks where the probe is reachable only from
inside the cluster, or alongside a Probes.Ready.Middleware that says who may
read it:
200 {"status":"ok", "checks":{"db":"ok","broker":"ok"}}
503 {"status":"not_ready", "checks":{"db":"ok","broker":"error"}}
Withholding them costs the orchestrator nothing — the status code is the whole
of its contract — and a failing check is logged through Config.Logger either
way, so a 503 is never undiagnosable. GET {prefix}/health is unaffected: its
db key is a name the framework owns rather than one you chose, so it describes
no topology.
In full, that endpoint answers:
HealthCheckDB | Database | Response |
|---|---|---|
| off | not checked | 200 {"status":"ok"} |
| on | reachable | 200 {"status":"ok","db":"ok"} |
| on | unreachable | 503 {"status":"degraded","db":"error"} |
db also reads unknown — not a failure — when the adapter implements no
Ping, or none is configured yet.
What that discloses to an unauthenticated caller is one bit: whether this
service’s database is reachable. The 503 carries the same bit on its own,
since with HealthCheckDB on the database is the only thing /health checks,
so the key adds no disclosure over the status line. The raw driver error is
logged rather than written, so no DSN fragment reaches the wire. Where even that
bit should not be public, Probes.Health.Middleware puts the endpoint behind
your own check and Probes.Health.Disabled takes it off the router.
A check reads unknown — which is not a failure — when the framework has no way
to test it: an adapter that does not implement Ping, or no adapter configured
at all.
Concurrent probe requests share one run of the checks, so a flood of probes cannot be amplified into a flood against the dependencies they report on. It coalesces rather than caches: a request arriving after the run finished starts a new one, since readiness that is even slightly stale keeps a pod in the load balancer after its database has gone away.
| Field | Default | Purpose |
|---|---|---|
ReadinessChecks | none | the application’s own dependency probes, reported by /ready beside the built-in db check |
HealthTimeout | 3s | budget shared by all dependency checks on /ready, and by /health when HealthCheckDB is on |
HealthCheckDB | false | when true, GET /health pings every distinct registered adapter (Config.DB plus any per-model overrides) and returns 503 on failure. Governs /health alone — /ready always checks the database |
Set HealthTimeout shorter than your probe’s timeoutSeconds so the handler
can return 503 cleanly before the probe times out. It must be positive —
unlike the connection timeouts there is no negative “disable” spelling, and a
negative value is refused at startup.
Add a dependency of your own with ReadinessChecks:
cfg := maniflex.Config{
ReadinessChecks: []maniflex.ReadinessCheck{{
Name: "broker",
Check: func(ctx context.Context) error { return broker.Ping(ctx) },
}},
}
Checks run on every readiness request, so keep them to a pool or connection
probe rather than a full round-trip, and honour the ctx. A check that returns
an error — or panics, which is recovered — makes /ready answer 503. The
error text is logged through Config.Logger and never written to the response:
a probe body is the one place an unauthenticated client reads straight from a
dependency, and connection strings live in those messages.
Names must be non-empty, unique, and not db, which the framework reserves;
anything else panics when the router is built rather than on a probe request.
A mounted probe owns its path. An action or a model whose routes would answer
GET {prefix}/live, /ready or /health is refused when the router is built,
naming the probe it would have replaced — chi overwrites rather than collides, so
without that check the endpoint an orchestrator polls quietly starts running
application code.
The reservation follows what is actually mounted, so Disabled is how you serve
one of these paths yourself:
Config{Probes: maniflex.ProbesConfig{Health: maniflex.ProbeConfig{Disabled: true}}}
// GET /api/health is now free for your own action.
Only the methods the framework serves are reserved. The probes mount GET, so a
POST {prefix}/live action is fine. The same rule covers the other built-in
routes under PathPrefix — /openapi.json, /asyncapi.json, the global search
path, and the /files endpoints — each reserved only while the feature that
mounts it is switched on.
Gating and unmounting the probes
The probes are mounted straight onto the router and never enter the model
pipeline, so no Pipeline.Auth middleware runs for them and
authx.AllowPublic has nothing to exempt them from. That is deliberate: an
orchestrator’s probe is the canonical unauthenticated request.
Config.Probes is the override — the only lever scoped to the probes alone.
(Config.HTTPMiddlewares reaches them too, but it reaches every other route at
the same time.) Its zero value mounts all three publicly.
cfg.Probes = maniflex.ProbesConfig{
// Readiness names your dependencies and reports which are down.
Ready: maniflex.ProbeConfig{
Middleware: []maniflex.HTTPMiddleware{probeToken},
},
// The legacy endpoint, retired in favour of /live and /ready.
Health: maniflex.ProbeConfig{Disabled: true},
}
| Field | Purpose |
|---|---|
Probes.Middleware | wraps every mounted probe, in order |
Probes.{Live,Ready,Health}.Middleware | wraps that one probe, after the shared chain — appended, not instead of it |
Probes.{Live,Ready,Health}.Disabled | leaves that probe off the router entirely |
Probes.PublishReadinessChecks | writes the per-dependency results into the /ready body; off by default |
A disabled probe is not mounted, so the request gets the router’s plain 404
and neither middleware chain runs. That is a more honest answer than a handler
that refuses: a 401 says the endpoint is there.
AdaptAuth reuses pipeline auth middleware here, the same way it does for
Documentation.Middleware:
cfg.Probes.Ready.Middleware = []maniflex.HTTPMiddleware{
maniflex.AdaptAuth(auth.JWTAuth(opts), auth.RequireRole("ops")),
}
Think before gating
/live. A liveness probe that receives a401is a liveness probe that fails, and Kubernetes answers a failing liveness probe by killing the container — during the graceful drain, taking its in-flight requests with it. Gating/readyalone is usually what you want.If you do gate
/live, the probe has to carry the credential, and a kubelethttpGetvaries a path far more easily than it rotates a header:livenessProbe: httpGet: path: /api/live?token=... port: 8080Which means the middleware must accept the query parameter, not only a header.
A public /ready is a defensible default, and the two things that used to make
it uncomfortable are handled without gating: the dependency names are withheld
unless you ask for them, and concurrent requests share one run of the checks.
Gate it when the endpoint is reachable from outside the cluster, or when even
{"status":"not_ready"} is more than you want to publish.
Reading from environment
maniflex.ConfigFromEnv(prefix) populates a Config from a conventional set of
environment variables. Use it for twelve-factor deployments, then override
individual fields in code where needed.
cfg, err := maniflex.ConfigFromEnv("") // or "ORDERS" → ORDERS_PORT, ORDERS_DB_WRITE_URL, …
if err != nil {
log.Fatal(err)
}
cfg.DisableAutoMigrate = true // disable for production
server := maniflex.New(cfg)
These are the variables it reads, and the only ones — anything else on Config
is set in code:
| Variable | Field | Value |
|---|---|---|
PORT | Port | integer, 1–65535 |
DB_WRITE_URL | DBWriteURL | string |
DB_READ_URL | DBReadURL | string |
QUERY_TIMEOUT_MS | QueryTimeout | positive integer, milliseconds |
SHUTDOWN_TIMEOUT_S | ShutdownTimeout | positive integer, seconds |
SERVICE_NAME | ServiceName | string |
HEALTH_CHECK_DB | HealthCheckDB | true/false, 1/0, yes/no, on/off |
A variable that is unset leaves its field at the zero value, for
ApplyDefaults to fill in. A variable that is set but unreadable is an
error — PORT=808O, QUERY_TIMEOUT_MS=abc, HEALTH_CHECK_DB=ture — naming the
variable and the value it could not read. Every bad variable is reported at once,
so two typos take one deploy to find rather than two. Don’t discard this error: a
mistyped PORT that is quietly ignored gives you a healthy-looking server
listening on 8080, and nothing anywhere says why.
Graceful Shutdown
server.Start() blocks on the HTTP listener and additionally listens for
SIGINT and SIGTERM. When either signal arrives, the server stops accepting
new connections and gives in-flight requests up to
Config.ShutdownTimeout to finish before forcing the listener closed.
How it works
- A signal arrives.
http.Server.Shutdown(ctx)is called with a deadline ofConfig.ShutdownTimeout(default: 30 seconds).- The listener stops accepting new connections immediately.
- In-flight requests are allowed to complete — including their pipeline middleware, transaction commits, and Response writes.
- Services are stopped in reverse registration order and
Config.OnShutdownruns, on what is left of the same budget. Server.Goloops and background work spawned by requests — audit writes, cache invalidations, event publishes — are waited for, still on that budget.Start()returns. The database adapter is not closed for you — see Database adapters.
If the deadline passes with requests still running, the underlying TCP connections are closed — those requests fail mid-flight but the process exits cleanly.
Long-lived handlers
Step 2 waits for every in-flight request and does not cancel their contexts. A handler that only returns when its client goes away, e.g. Server-Sent Events, a WebSocket pump, or a long poll, therefore holds the whole drain, and steps 5 to 7 get whatever is left of the budget, which for one such connection is nothing.
Server.ShuttingDown() returns a channel closed at the very
start of shutdown, before any waiting begins:
for {
select {
case <-r.Context().Done(): // this client went away
return
case <-server.ShuttingDown(): // the server is going away
return
case ev := <-events:
writeEvent(w, ev)
}
}
Inside an action, ctx.ShuttingDown() is the same channel.
Nothing is cancelled by it: a handler that ignores it keeps the whole
ShutdownTimeout to finish in, exactly as before. Returning promptly is what
leaves the rest of the budget for services, hooks and background writes.
The realtime hub is the case this matters most for, and it has a dedicated wiring — see Let the hub hear the shutdown coming.
Embedding Handler() in your own server
An embedding owns the HTTP listener, while Maniflex still owns its registered
services, Server.Go loops, and request background work. Start and stop those
two halves explicitly. Finish model, action, and middleware registration before
calling MigrateOnly, which validates and seals that configuration:
if err := server.MigrateOnly(ctx); err != nil {
log.Fatal(err)
}
if err := server.StartServices(); err != nil {
log.Fatal(err)
}
httpServer := &http.Server{
Addr: ":8080",
Handler: server.Handler(),
}
go httpServer.ListenAndServe()
// On termination, stop requests first so none can add new background work.
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_ = httpServer.Shutdown(shutdownCtx)
_ = server.Shutdown(shutdownCtx)
StartServices validates and seals the router, runs OnStart, and starts
services in registration order, but does not migrate or open a listener.
Server.Shutdown then stops those services in reverse order, runs
OnShutdown, cancels Server.Go, and drains both application and request
background work.
An embedding with no services or lifecycle hooks may omit StartServices.
It must still call Server.Shutdown after its own http.Server.Shutdown;
Maniflex will cancel and drain Server.Go and ctx.GoBackground work without
running lifecycle phases that never started.
Database adapters
SetDB takes an opened handle, and its lifetime isn’t the server’s — which is
what lets a jobs queue or the admin panel share the same pool. Closing it should be as follows:
db, err := sqlite.Open("./app.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
if err := server.Start(); err != nil {
log.Fatal(err)
}
Closing the database before Server.Shutdown completes terminates active database connections while Service.Stop and in-flight audit writes are still executing.
Since Server.Start() blocks until those operations finish, placing db.Close() in a defer guarantees it runs only after they complete. If the server components are orchestrated manually instead of through Server.Start(), the rule remains the same: db.Close() must be called after Server.Shutdown returns, never before.
Event buses
A bus is not owned by the server, so nothing closes it for you. inproc.Bus.Close
stops accepting events and waits for in-flight handlers, bounded by
Options.DrainTimeout; a non-nil error means the drain did not finish and those
events were not processed. Broker adapters close their connections.
Tuning ShutdownTimeout
Pick the value based on the longest legitimate request your service serves:
| Environment | Suggested ShutdownTimeout |
|---|---|
| Tests | 0–1s — exit instantly |
| Lambdas / fast-cycling containers | 5–10s |
| General OLTP API | 30s (default) |
| Bulk import or large file uploads | 60s+ |
Setting ShutdownTimeout shorter than your slowest request will sever it on
shutdown. Setting it longer makes deploys slower with no benefit beyond the
slowest real request.
Why graceful shutdown matters
Cutting a request mid-write produces inconsistent state at the boundary — a write that may or may not have committed, a webhook that may have fired but not been recorded, a client that may or may not have seen the response. The graceful path:
- ensures transactions commit or roll back cleanly,
- lets the Response step write its envelope before the connection drops,
- gives
maniflex.WithTransaction’s deferred rollback a chance to run.
For Kubernetes deployments, set terminationGracePeriodSeconds on the pod to
a value larger than ShutdownTimeout, otherwise the orchestrator will send
SIGKILL before the graceful handler completes.
Manual shutdown
For tests or custom lifecycle code, the same graceful path is available without waiting for a signal:
go server.Start()
// ... run tests ...
server.Shutdown(ctx)
Shutdown uses the supplied context as the deadline. Pass context.Background
for “wait as long as it takes”; pass a context.WithTimeout for an explicit
budget.
It is safe to call from any goroutine at any point in the server’s life — the
snippet above races Start by construction, and the outcome does not depend on
who wins. A Shutdown that lands while the server is still booting (migrating,
or waiting on a service that dials its backend) countermands the boot: the
listener is never opened, Start unwinds whatever it had already brought up and
returns nil, and Shutdown waits for that to finish before returning. On a
server that never started services or a listener, it still cancels and drains
any Server.Go and request background work before returning.
Shutdown is terminal rather than a pause: a server that has been shut down will
not open a listener afterwards. Start following Shutdown returns
maniflex.ErrStopped. A Server is not restartable — build a new one.
Only one caller may own startup. A second Start, StartWithContext, or
StartServices while a startup mode is active returns
maniflex.ErrAlreadyStarted; it does not repeat validation, migration, service
startup, or listener publication. After shutdown or a startup failure, later
startup calls return maniflex.ErrStopped. Both errors support errors.Is.
Background writes
Audit-log writes, cache invalidations (db.Invalidate), and async file
cleanups (Config.FilesConfig.Storage with mfx:"auto_delete" fields) run on
goroutines tracked by the server. Shutdown waits for those to drain
within the same deadline as the HTTP listener. If the deadline elapses
with goroutines still in flight, the server logs a warning with the
in-flight count and proceeds — the goroutines see their context cancelled
and exit on the next checkpoint.
Custom middleware can opt into the same lifecycle via
ctx.GoBackground(fn func(context.Context)); the supplied context is
independent of the request (which has already returned) but IS cancelled
when shutdown’s deadline hits.
Panics on background goroutines
A panic in a ctx.GoBackground task or a server.Go loop is recovered, logged
through Config.PanicLogger with its stack, and contained — it does not take
the process down. Without that, a panic in one audit write killed a server that
was otherwise healthy, which is worse than the request panic the framework
already recovers.
Containment has a cost worth naming: a server.Go loop that panics is gone
until the next restart while the process keeps serving HTTP. The ERROR record
is the only signal, so treat it as one. Config.OnBackgroundPanic is the hook
for acting on it programmatically:
cfg.OnBackgroundPanic = func(recovered any, stack []byte) {
metrics.Inc("background_panic")
os.Exit(1) // let the orchestrator restart a half-dead process
}
It runs on the panicking goroutine after the log is written, so it must not
block, and a panic inside the hook is not recovered again. Leave it nil and
the panic is only logged.
Supervised services & lifecycle hooks
Applications often own long-lived background components — a poller, cache
warmer, queue consumer, or an in-memory pool manager — that must start after
the database is ready and stop cleanly before the process exits. Register
them as services and the framework folds them into the boot and shutdown
lifecycle instead of you hand-supervising them around Start.
type Service interface {
Start(ctx context.Context) error // ctx is cancelled at shutdown
Stop(ctx context.Context) error // carries what's left of the shutdown budget
}
server.AddService(pool) // a custom Service
server.AddService(maniflex.ServiceFunc(startFn)) // adapter for a bare start func
AddService must be called before Start or StartServices — it panics once
either startup path has been entered, including while earlier services are
still starting. Startup fixes the service list when it begins, so a service
registered after that point would never be started, and a panic is a better
answer than a component that silently never runs. The same applies to Action,
RealtimeDoc and EnableGlobalSearch, which fix the routing table.
For app-scoped fire-and-forget work (e.g. a periodic reconciler) that doesn’t
need an ordered Stop, use server.Go. Its context is cancelled when shutdown
begins, and the goroutine is drained before Start returns:
server.Go(func(ctx context.Context) {
t := time.NewTicker(time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
reconcile(ctx)
}
}
})
Callers that want a hook without defining a Service type can set the
lightweight Config.OnStart / Config.OnShutdown functions.
Boot order: migrate → OnStart → Service.Start (registration order) → listen. The embedded sequence performs migration explicitly, then
StartServices runs OnStart → Service.Start; the caller opens its listener.
A StartServices, Start, or OnStart error aborts startup; services that
already started are stopped in reverse first.
A failed boot still tears down. Whatever boot managed to bring up is put back
down before Start returns the error — including the server.Go loops, which
run from the moment you call server.Go, not from Start. If the listener fails
to bind (the port is taken), the services stop and the goroutines drain exactly as
they do on the graceful path, in a ShutdownTimeout window of its own; if the
migration or a service refused to start, the loops are cancelled and awaited. Your
goroutines are never abandoned mid-write, however boot ends.
Shutdown order: http.Shutdown → Service.Stop (reverse order) → OnShutdown → drain server.Go + ctx.GoBackground goroutines. The Start context is
cancelled when shutdown begins so loops wind down on their own.
One budget, shared. Every phase runs on the same deadline context —
ShutdownTimeout, or whatever deadline you pass to server.Shutdown(ctx). The
phases are sequential, so they draw down a single window rather than each getting
a fresh one: a drain that eats 25 of 30 seconds leaves Stop, OnShutdown and
the goroutine drain 5 seconds between them. Honour the ctx you are handed — that
is what keeps total shutdown inside the window your orchestrator allows before it
escalates to SIGKILL.
AddService, OnStart, and server.Go are inert for apps that register
nothing — there is no behavioural change unless you opt in.
Probes during shutdown
The two probes deliberately diverge the moment shutdown begins:
| Endpoint | During the drain |
|---|---|
GET {prefix}/ready | 503 {"status":"stopping"} immediately, without waiting on any dependency check — this is what deregisters the pod from its load balancer |
GET {prefix}/live | still 200, for as long as the process can answer |
Liveness must stay green here. A liveness probe that fails during the drain
earns the process a SIGKILL in the middle of the requests the drain exists to
finish — the exact outcome graceful shutdown is meant to prevent.
Requests already in flight are honoured throughout, so the endpoints keep answering until the listener closes.
Satellite Modules
maniflex is a multi-module monorepo. The core module — also named maniflex —
carries only chi and uuid. Every heavy dependency (a database driver, a
message broker client, a crypto library) lives in its own satellite module
under the same repository, so a consumer pulls only the dependencies it
imports.
Layout
maniflex/ # core module — chi + uuid
├── maniflex/ # framework
├── storage/ # local disk file storage
└── ...
storage/
└── s3/ # aws-sdk-go-v2 — S3, MinIO, R2, Spaces, etc.
db/
├── sqlite/ # modernc.org/sqlite — pure-Go SQLite
├── postgres/ # lib/pq — PostgreSQL
└── sqlcore/ # shared SQL adapter used by both
events/
├── kafka/ # segmentio/kafka-go
├── nats/ # nats.go
├── rabbitmq/ # rabbitmq/amqp091-go
└── redis/ # go-redis
jobs/
├── inproc/ # goroutine pool — tests and single-binary apps
├── sql/ # *sql.DB-backed (Postgres + SQLite) with transactional outbox
├── redis/ # Redis Streams / BRPOP — high-throughput worker fleets
├── cron/ # scheduled EnqueueAt ticker
└── maniflex/ # StatusModel + Mount helper — REST polling for job status
middleware/
├── auth/, body/, db/, … # catalogue middleware (see Middleware Catalogue)
├── service/bcrypt/ # golang.org/x/crypto for password hashing
└── db/redis/ # Redis cache invalidation
examples/ # runnable example apps — its own module
tests/ # e2e suite — its own module
maniflextest/ # supported consumer integration-test harness
Why split modules
The split keeps the core dependency graph minimal:
- A project that uses SQLite imports
maniflex/db/sqliteand gets the pure-Go driver. PostgreSQL’slib/pqis not in its build. - A project that publishes events to Kafka imports
maniflex/events/kafkaand pulls insegmentio/kafka-go. NATS and RabbitMQ stay out of the build. - A project that does not authenticate doesn’t import
middleware/authand pays nothing for the JWT library.
This matters most for binary size, attack surface, and CI build time. It also keeps the core stable: changing a database driver does not require a release of the framework itself.
Importing satellites
Each satellite is a normal Go module — add it with go get:
go get github.com/xaleel/maniflex # core
go get github.com/xaleel/maniflex/db/sqlite # SQLite adapter
go get github.com/xaleel/maniflex/middleware/auth # auth helpers
go get github.com/xaleel/maniflex/events/kafka # Kafka publisher
go get github.com/xaleel/maniflex/maniflextest # application test harness
In code:
import (
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
"github.com/xaleel/maniflex/middleware/auth"
"github.com/xaleel/maniflex/events/kafka"
)
There are no required satellites for the framework itself to function —
maniflex alone gives you the registry, pipeline, and HTTP layer. You need
at least a database adapter (sqlite or postgres) before server.Start() can
serve a request.
Workspace mode
The repository ships a go.work file that includes every satellite module.
For consumers, this is invisible — Go modules resolve normally through
go.mod. For contributors working across modules, go.work makes
cross-module changes possible without replace directives.
go build ./... and go test ./... operate per-module. To build or test
every module at once, use the helper scripts in scripts/:
bash scripts/test-all.sh
# or
powershell scripts/test-all.ps1
Before producing release artifacts, scan every workspace module against the current Go vulnerability database:
bash scripts/vulncheck-all.sh
# or
powershell scripts/vulncheck-all.ps1
Versioning
Each satellite carries its own v0.x tags. The core module is the only one a
typical app depends on by name; satellites are usually pulled in transitively
or by direct import as needed. Pin satellite versions in go.mod when
reproducibility across machines is required.
Admin Panel
maniflex/admin is an opt-in satellite module that mounts a server-rendered
administration panel on top of any maniflex server. It introspects the model
registry to build its navigation and views, and reads/writes data by issuing
in-process HTTP requests against the server’s own REST API — so every
operation travels the full auth/validate/pipeline stack. The admin never
touches the database directly.
Adding the module
go get github.com/xaleel/maniflex/admin
Because it is a satellite, importing it is the only thing needed to bring the
admin into your binary. The core maniflex module has no dependency on it.
Quick start
package main
import (
"net/http"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/admin"
"github.com/xaleel/maniflex/db/sqlite"
"github.com/go-chi/chi/v5"
)
func main() {
server := maniflex.New(maniflex.Config{PathPrefix: "/api"})
server.MustRegister(User{}, Post{}, Comment{})
db, _ := sqlite.Open(":memory:", server.Registry())
server.SetDB(db)
adminHandler := admin.Mount(server, admin.Config{
Title: "My App Admin",
AllowUnauthenticated: true, // local dev only
})
r := chi.NewRouter()
maniflex.Mount(r, server)
r.Mount("/admin", http.StripPrefix("/admin", adminHandler))
http.ListenAndServe(":8080", r)
}
Mount must be called after all models are registered and the DB adapter
is set, and before the server starts handling requests. It panics early if
neither Config.Auth nor Config.AllowUnauthenticated is set, so an
unprotected panel can never be shipped by accident.
maniflex.Mount forwards PathPrefix only. If this server also sets
Config.StaticDir, its files are served outside that prefix and will 404 here
— Mount warns, and
Behind maniflex.Mount
has the two-line fix.
Config reference
| Field | Type | Default | Description |
|---|---|---|---|
PathPrefix | string | "/admin" | Mount path; the returned handler serves routes under this prefix |
Title | string | "maniflex admin" | Displayed in the panel header |
Auth | func(http.Handler) http.Handler | — | Wraps the whole panel with an auth gate; required unless AllowUnauthenticated is set |
AllowUnauthenticated | bool | false | Skips the auth requirement; local dev only |
Models | []string | (all) | Struct names to show; empty means every registered model |
ReadOnly | bool | false | Hides create/edit/delete UI and unmounts those routes |
Templates | fs.FS | — | Override FS for custom templates (see Templates) |
StaticFS | fs.FS | — | Replaces the embedded CSS/asset bundle |
Authentication
Set Config.Auth to any func(http.Handler) http.Handler middleware — for
example the JWT middleware from maniflex/middleware/auth:
import "github.com/xaleel/maniflex/middleware/auth"
adminHandler := admin.Mount(server, admin.Config{
Title: "My Admin",
Auth: auth.JWTAuth(secret, auth.JWTOptions{}),
})
The Auth wrapper runs before every panel request. Because data reads and
writes go through the API pipeline, the upstream Auth middleware registered
on your model endpoints also enforces field-level and operation-level rules —
the admin doesn’t bypass them.
Never set
AllowUnauthenticated: truein a production deployment. Mount panics at startup if neither option is provided, so there is no way to accidentally omit auth and only discover it at runtime.
Views
Dashboard
GET /admin/ — one summary card per visible model showing its total row
count, fetched in-process from the API.
List
GET /admin/{model} — a paginated table of records.
- Pagination — 20 rows per page;
?page=Nnavigates. - Sorting — a dropdown built from fields tagged
mfx:"sortable". The current direction is preserved across filter changes. - Filtering — one input per field tagged
mfx:"filterable". Enum fields render a<select>; other fields render a text input. Active filters persist in the URL as?f_<field>=<value>.
Detail
GET /admin/{model}/{id} — all readable fields for one record.
- FK fields rendered as links to the related record (
/admin/{related}/{fk_id}). HasManyrelations shown as “View {related}” links (list pre-filtered to this record’s ID).- Edit and Delete actions, each protected by a CSRF token.
Create form
GET /admin/{model}/new — an empty form; POST /admin/{model} submits it.
Edit form
GET /admin/{model}/{id}/edit — the form pre-filled from the existing record;
POST /admin/{model}/{id} submits it.
Both forms share the same template and widget logic:
| Widget | When used |
|---|---|
text | default string fields |
textarea | long-text / text DB type |
number | integer and float fields |
checkbox | boolean fields |
select | fields with mfx:"enum:…" |
relation | BelongsTo FK fields — a <select> populated from the target model |
file | fields tagged mfx:"file" — includes a preview/download link when a file is already stored |
datetime | time.Time fields — rendered as an <input type="datetime-local"> |
Fields tagged mfx:"hidden" or mfx:"writeonly" are excluded from the list
and detail views. Fields tagged mfx:"readonly" appear on the edit form as
disabled inputs (they are server-managed). mfx:"immutable" fields are
editable on create but disabled on edit.
Delete
POST /admin/{model}/{id}/delete — deletes the record via the API and
redirects to the list. Requires a valid CSRF token (present on the detail
page’s Delete button).
CSRF protection
The panel uses double-submit cookies. On first form load a random 32-byte
hex token is written to the CSRF cookie and mirrored in a hidden _csrf form
field. Every mutating POST verifies that both match before forwarding to the
API. The check itself has nothing to configure — it is always on.
The cookie is HttpOnly, SameSite=Lax, and Secure by default. It used to
infer that per request from r.TLS and X-Forwarded-Proto, which failed quietly
in the wrong direction: a panel served over plaintext in production issued a
non-Secure cookie and nothing said so. A fixed default fails in the open
instead — the browser refuses to return the cookie and the panel visibly stops
working, which is worth knowing, because an admin panel on plaintext is exposing
a great deal more than a CSRF token.
http://localhost is a secure context in current Chrome and Firefox, so local
development is unaffected. For a panel deliberately served over plaintext on a
host that is not — a LAN hostname, say — opt out explicitly:
insecure := false
admin.Mount(server, admin.Config{
Secure: &insecure, // admin session and CSRF token now travel in the clear
})
This is the panel’s own check, over the panel’s own forms. It is separate from
the auth.CSRF middleware, which guards
your API’s routes and is what you reach for when browsers authenticate to the
API with cookies. Neither one configures the other.
Model whitelist
To show only a subset of registered models:
admin.Mount(server, admin.Config{
Models: []string{"User", "Post"},
// "Comment" will not appear in the panel
})
Model names are Go struct names, not table names. Models omitted from the whitelist are hidden from navigation, list, and detail views — they are still served by the API.
Read-only mode
admin.Mount(server, admin.Config{
ReadOnly: true,
})
In read-only mode the create/edit/delete routes are not mounted at all, and the corresponding controls are hidden in the UI. Useful for support teams that need visibility without write access.
Templates
Drop in a replacement for any individual template by providing a fs.FS on
Config.Templates. Any file not present in the override FS falls back to
the embedded default. The template file names are:
| File | View |
|---|---|
layout.html | outer chrome (header, sidebar, <head>) |
dashboard.html | model summary cards |
list.html | paginated table |
detail.html | single-record field list |
form.html | shared create/edit form |
error.html | error page |
Example — override only the layout to inject custom branding:
//go:embed templates
var myTemplates embed.FS
admin.Mount(server, admin.Config{
Templates: myTemplates,
})
The templates receive the viewData struct. Consult the admin package source
(view.go) for the full shape of each page’s data.
Static assets
The embedded asset bundle is served under {PathPrefix}/static/. To replace
it entirely with a custom CSS file:
//go:embed assets
var myAssets embed.FS
admin.Mount(server, admin.Config{
StaticFS: myAssets,
})
StaticFS replaces the whole bundle — include any assets the templates
reference (or adjust the templates to match).
How it works
The panel is self-contained: it holds a reference to server.Handler() and
issues normal http.Request objects against it in-process. There is no
separate HTTP round-trip.
browser → GET /admin/users
→ admin handler
→ apiClient.list(r, "users", "limit=20&sort=…")
→ server.Handler().ServeHTTP(rw, r') // in-process
→ full pipeline (Auth → Validate → DB → Response)
← []map[string]any
← rendered list.html
This means:
- Pipeline middleware on the model (tenant isolation, field redaction, soft- delete visibility) is enforced on every admin read and write.
- Auth cookies or tokens present on the browser request are forwarded unchanged to the API, so per-user permission checks work automatically.
- The admin has no SQL access of its own and cannot bypass business rules.
→ Satellite Modules
→ Field Tags Reference
→ File Fields & Uploads
→ Pipeline Overview
Custom Endpoints (Actions)
The five generated REST routes per model cover the standard CRUD shape, but
some endpoints don’t fit that shape — POST /orders/{id}/cancel,
POST /invoices/{id}/send, GET /reports/revenue. Actions are maniflex’s
mechanism for adding these.
Registering an action
An action is a method, a path, and a handler:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Handler: cancelOrder,
})
For ValidateProduction, matching global Pipeline.Auth middleware is the
normal access decision. If authorization lives inside the action’s middleware
or handler, set AccessControlled: true; use AllowPublic: true only for an
intentionally public action. These flags document the decision and do not
install authorization.
The handler receives the standard *maniflex.ServerContext:
func cancelOrder(ctx *maniflex.ServerContext) error {
orderID := ctx.URLParam("id")
if _, err := ctx.GetModel("Order").Update(orderID, map[string]any{
"status": "cancelled",
}); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusOK,
Data: map[string]any{"ok": true},
}
return nil
}
The trimmed pipeline
Action requests run a shorter pipeline than CRUD requests:
Auth → [per-action middleware...] → handler → Response
Deserialize, Validate, Service, and DB are skipped. The action handler
is responsible for parsing its own body (ctx.BindJSON) and performing its
own database work (via ctx.GetModel, ctx.RawExec, or directly).
ctx.Operation is OpAction inside the handler. Middleware registered on the
trimmed-out steps with ForOperation(maniflex.OpAction) does not run; only Auth
and Response middleware do.
DB-step middlewares don’t cover actions. Anything registered on
Pipeline.DB— includingdb.RateLimit,db.AuditLog,db.ForceFilter, anddb.Tenancy— is silently skipped for action routes. For an all-action service this means zero rate limiting, zero audit records, and no tenancy unless you wire them per action. Use the action-flavoured variants in the action’s ownMiddlewarelist:db.RateLimitAction(cfg)(keys on the caller + method/path),db.AuditLogAction(sink)(records actor/resource/result from the action context), anddb.TenancyAction(field, fn)/db.ForceFilterAction(field, fn)(row-level scoping — see Scoping Actions).
Under db.TenancyAction / db.ForceFilterAction the handler’s database work is
scoped through ctx.GetModel, the typed generics, ctx.Aggregate,
ctx.LockForUpdate, and the Tx from ctx.BeginTx (so WithTransaction and
Batch work too) — and ctx.RawQuery, ctx.RawExec, ctx.Search and
ctx.RecursiveQuery refuse, because a scope cannot be applied to raw SQL and
running it anyway would return every tenant’s rows. ctx.Unscoped() bypasses
that deliberately where a path genuinely must. An action with no scope registered
is unaffected: every path behaves as it always has.
Per-action middleware
Actions can carry their own middleware list, which runs between Auth and the
handler:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Handler: cancelOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.RequireRole("admin"),
idempotency.Key("Idempotency-Key"),
},
})
This is the equivalent of the Service step for an action — anything that should run before the handler but after authentication.
Reading input
The action handler does its own request parsing:
type RefundReq struct {
Amount float64 `json:"amount"`
Reason string `json:"reason"`
}
func refundOrder(ctx *maniflex.ServerContext) error {
var req RefundReq
if err := ctx.BindJSON(&req); err != nil {
return nil // ctx.Abort already called
}
// ... work ...
ctx.Response = &maniflex.APIResponse{StatusCode: http.StatusOK}
return nil
}
ctx.BindJSON enforces the same 4 MB body limit as the default Deserialize
step. ctx.URLParam and ctx.QueryParam read URL and query parameters.
Unknown fields are ignored, not rejected. BindJSON is encoding/json, so a
key your struct has no field for is silently dropped — the stdlib default, and
the same thing every Go HTTP handler does. It is worth knowing because the model
write path is not like this: a create or update against a registered model
knows the full column set, so Config.Strict can reject a body naming a field
that does not exist, and body.StripUnknownFields() can drop them deliberately.
An action’s request struct is ordinary Go with no registry behind it, so neither
applies and a client typo — {"amonut": 500} — arrives as a zero value rather
than an error.
Where that matters, decode strictly yourself:
raw, err := ctx.EnsureRawBody()
if err != nil {
return nil // EnsureRawBody already called ctx.Abort
}
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&req); err != nil {
ctx.Abort(http.StatusBadRequest, "INVALID_JSON", err.Error())
return nil
}
EnsureRawBody reads under the same size limit, caches on ctx.RawBody, and
restores the request body — so it replaces BindJSON here rather than following
it.
Multipart uploads:
ctx.Filesis populated by the Deserialize step, which actions skip — so it is always empty inside an action. To accept a file upload in an action, parse the request yourself:if err := ctx.Request.ParseMultipartForm(32 << 20); err != nil { ctx.Abort(http.StatusBadRequest, "BAD_REQUEST", "invalid multipart form") return nil } for _, headers := range ctx.Request.MultipartForm.File { // headers[0].Open() → the uploaded file }
Transactional actions
ctx.BeginTx works inside an action just as it does in middleware. For most
actions, wrap the handler body in a BeginTx / Commit block:
func cancelOrder(ctx *maniflex.ServerContext) error {
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
// ... transactional work via ctx.GetModel / ctx.RawExec ...
return tx.Commit()
}
Because the action does not pass through the Service step, maniflex.WithTransaction
registered there does not apply — actions manage their own transactions.
SQLite deadlock — fetch the model accessor after setting
ctx.Tx. The SQLite adapter uses a single write connection (MaxOpenConns(1)). Actx.GetModel(...)accessor binds to whateverctx.Txis at the time you callGetModel. If you grab the accessor beforeBeginTxand then write with it afterctx.Tx = tx, the write opens a second writer connection and blocks forever behind the transaction holding the single writer — the request hangs with no error. Always callctx.GetModel(...)afterctx.Tx = tx:tx, _ := ctx.BeginTx(ctx.Ctx, nil) defer tx.Rollback() ctx.Tx = tx orders := ctx.GetModel("Order") // bound to ctx.Tx now — safe orders.Update(id, patch) if err := tx.Commit(); err != nil { return err } ctx.Tx = nil // reset: reads built for the response after commit must not // route through the finished tx
Streaming a raw (non-JSON) response
An action normally returns JSON by setting ctx.Response. For a binary or
streaming endpoint — serving an image, a generated PDF, a CSV export — write
directly to ctx.Writer (the raw http.ResponseWriter) and leave
ctx.Response nil. After the handler returns, the framework writes nothing
further when ctx.Response is nil, so the bytes you wrote are the whole
response:
func downloadEvidence(ctx *maniflex.ServerContext) error {
f, err := openEvidence(ctx.URLParam("id"))
if err != nil {
ctx.Abort(http.StatusNotFound, "NOT_FOUND", "evidence not found")
return nil
}
defer f.Close()
ctx.Writer.Header().Set("Content-Type", "image/png")
ctx.Writer.WriteHeader(http.StatusOK)
_, err = io.Copy(ctx.Writer, f) // leave ctx.Response nil
return err
}
When the bytes are an mfx:"file" model field, prefer the built-in per-model
attachment route (GET /{model}/{id}/{field}) instead — it runs the read
pipeline (auth, soft-delete, tenancy) and streams for you. If you scope a
db.ForceFilter to OpList/OpRead, remember to add OpReadAttachment too, or
downloads bypass the filter.
Documenting an action in OpenAPI
Actions appear in the generated OpenAPI spec automatically. By
default each one contributes its method, path (with path parameters extracted
from {...} segments), Summary, Tags, and Deprecated flag:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Summary: "Cancel an order",
Tags: []string{"Orders"},
Handler: cancelOrder,
})
For request/response bodies, query parameters, and security, fill in the
optional OpenAPI block. Its most useful feature is schema inference: point
RequestSchema / ResponseSchema at a Go struct tagged with the same json
and mfx tags you already use on models, and maniflex reflects it into a JSON
schema — no hand-written OpenAPI types:
type RescheduleReq struct {
NewTime string `json:"new_time" mfx:"required"`
Reason string `json:"reason"`
}
type RescheduleResp struct {
ID string `json:"id"`
Status string `json:"status" mfx:"enum:scheduled|cancelled"`
}
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/appointments/{id}/reschedule",
Summary: "Reschedule an appointment",
Handler: reschedule,
OpenAPI: maniflex.ActionOpenAPI{
Description: "Moves an appointment to a new time.",
RequestSchema: RescheduleReq{},
ResponseSchema: RescheduleResp{},
ResponseStatus: http.StatusOK, // status the response schema documents; defaults to 200
QueryParams: []maniflex.OASParameter{{
Name: "notify",
Schema: &maniflex.OASSchema{Type: "boolean"},
}},
Security: []map[string][]string{{"bearerAuth": {}}},
},
})
The reflected schemas honour the field tags you already use on models —
required, enum, min, max, readonly, writeonly — and skip hidden
fields. RequestSchema and ResponseSchema each accept a struct value, a
pointer, or a reflect.Type.
Security names a scheme you register separately with
openapi.AddSecurityScheme.
If you’d rather build the OpenAPI types by hand, set RequestBody and
Responses directly on the ActionConfig — those take precedence over the
inferred schemas when both are present.
Query parameters
An action’s path parameters are extracted from the route automatically, so
/reports/{id}/export documents id without being told. Query parameters have
no such source — an action reads them with ctx.QueryParam, which the generator
cannot see — so declare each one you want in the contract:
server.Action(maniflex.ActionConfig{
Method: "GET",
Path: "/reports/{id}/export",
Summary: "Export a report",
Handler: exportReport,
OpenAPI: maniflex.ActionOpenAPI{
QueryParams: []maniflex.OASParameter{
{
Name: "format",
Required: true,
Description: "Output format.",
Schema: &maniflex.OASSchema{
Type: "string",
Enum: []any{"csv", "xlsx"},
},
},
{
Name: "since",
Schema: &maniflex.OASSchema{Type: "string", Format: "date-time"},
},
},
},
})
The operation then carries id first, followed by the declared parameters in
the order you wrote them:
"parameters": [
{"name": "id", "in": "path", "required": true, "schema": {"type": "string"}},
{"name": "format", "in": "query", "required": true, "description": "Output format.",
"schema": {"type": "string", "enum": ["csv", "xlsx"]}},
{"name": "since", "in": "query", "schema": {"type": "string", "format": "date-time"}}
]
In defaults to "query", so the common case is just a name and a schema. Set
it explicitly to "header" or "cookie" to document one of those on the same
action; an explicit value is never overwritten. Leave path parameters to the
route — declaring one here duplicates the extracted entry rather than replacing
it.
Every other field is emitted as written, Name included, so a typo reaches the
document unchallenged.
These declarations document; they do not enforce. Nothing parses, validates,
coerces, or rejects a request based on them. Required: true tells consumers the
parameter is mandatory; your handler still has to check that ctx.QueryParam
returned something and ctx.Abort if it did not.
Serving a model’s own path from an action
An action and a model cannot both own the same method + path. Registering an
action at a path a model already owns (e.g. GET /threads when a model’s table
is threads) is rejected at startup with a clear panic, rather than letting
the router silently mount two handlers and serve whichever one it happens to
match first:
panic: maniflex: action GET /threads would shadow the auto-generated collection route for model "Thread"
The check covers every route a model mounts, not just the five CRUD ones: the
export and aggregate endpoints, per-field attachment paths
(GET /{model}/{id}/{field}), and a singleton’s GET / PATCH on its bare path.
A conflict is reported when the action resolves the same path and method as a
model route — a path parameter’s name is irrelevant (GET /threads/{threadId}
collides with the read route just as GET /threads/{id} does), while a method the
model does not serve at that path is free to take (POST /threads/{id} is fine —
the item route has no POST).
The framework’s own routes under PathPrefix are covered by the same rule, and
reported when the router is built rather than at the Action call — global
search can be enabled after the action is registered, so the full picture is only
known at the end:
maniflex: [route] action GET /live would replace the framework's probe route at
/api/live — chi overwrites rather than collides, so the built-in endpoint would
stop answering with nothing reported (set Config.Probes.Live.Disabled to serve
this path yourself)
Each is reserved only while the feature that mounts it is on: the probes unless
Disabled, /openapi.json and /asyncapi.json while the documentation is
published, the search path after EnableGlobalSearch, and /files when
FilesConfig.MountEndpoints is set. Turning the feature off frees the path.
When you want to serve a model’s collection path yourself — returning a custom shape, or composing several models — mark the model headless so it mounts no REST routes at all, freeing its path for the action:
server.MustRegister(Thread{}, maniflex.ModelConfig{Headless: true})
server.Action(maniflex.ActionConfig{
Method: "GET",
Path: "/threads", // no collision: Thread mounts no routes
Handler: listThreads,
})
A headless model is still registered in full — it migrates, participates in
relations, and is reachable through ctx.GetModel("Thread") and typed CRUD — it
simply has no auto-generated HTTP surface (and no auto-generated OpenAPI paths;
its schema is still emitted for $refs). Use it whenever the generated CRUD
shape doesn’t match the contract you want to expose for that resource.
When to use an action
| Need | Use |
|---|---|
| Standard CRUD | The generated routes |
One-off state transitions (/cancel, /publish) | Action |
| Aggregations and reports | Action, or Raw Queries & Query Models |
| Bulk operations | Batch Operations & Sagas |
| Background processing | Events & Background Jobs |
| Reaching the pipeline from Go, rather than exposing Go to HTTP | Execute |
An action and Execute are mirror images: an action lets HTTP reach your logic,
and Execute lets your logic reach the pipeline — with a typed
principal instead of a header, and your transaction instead of N of them. If you
have ever been tempted to have your server make an HTTP request to itself, that is
the tool.
Reserve actions for endpoints that genuinely don’t fit CRUD. Resist the temptation to use them as a general-purpose handler API — the framework’s strength is in the generated routes; every action is one more thing to test and document by hand.
In-Process Invocation (Execute)
server.Action lets HTTP reach your logic. server.Execute is the other half: it
lets your logic reach the pipeline.
res, err := srv.Execute(ctx.Ctx, maniflex.Invocation{
Model: "Item",
Operation: maniflex.OpUpdate,
ID: itemID,
Body: map[string]any{"status": "approved"},
Auth: &approver, // a typed principal, not a header
Tx: ctx.Tx, // joins the caller's transaction
})
Every step runs — Auth, Deserialize, Validate, Service, DB, Response — in the same order, through the same middleware, as the equivalent request from a client. The result is the same envelope that request would have produced.
Why it exists
Without it, HTTP is the only way in, and code that needs to run a model operation from Go has to make a request to itself:
// Don't do this.
req, _ := http.NewRequest("PATCH", path, body)
req.Header.Set("X-Replay", "1")
req.Header.Set("X-Requester-ID", requesterID)
router.ServeHTTP(httptest.NewRecorder(), req)
Both defects that follow are structural, not careless:
- A principal passed as a header is a principal any client can send. The gates then grow bypasses keyed on that header, and those bypasses are reachable from the internet by construction. No amount of care fixes it at that layer — you cannot pass an identity to a router except through the request.
- N requests cannot be one transaction. A loop that replays N items over HTTP commits each separately, so a failure at item 3 leaves 1 and 2 written while the batch is marked unfinished — and re-running it re-applies the prefix.
Invocation.Auth is a typed *AuthInfo, so there is no header to forge and no
bypass to grow. Invocation.Tx is your transaction, so N invocations commit or
roll back together.
Maker–checker, atomically
The use case this was built for: a staged write, captured earlier, executed on approval as one unit.
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() // no-op once committed
for _, item := range request.Items {
if _, err := srv.Execute(ctx.Ctx, maniflex.Invocation{
Model: item.Model,
Operation: maniflex.OpUpdate,
ID: item.ResourceID,
Body: item.Payload, // the bytes captured at request time
Auth: &requester, // whoever the write is attributed to
Tx: tx,
}); err != nil {
return err // the whole batch rolls back — no committed prefix
}
}
return tx.Commit()
Events from an Execute batch
The transaction above is yours, and the framework cannot tell when your Commit
succeeds. So a side effect any invocation registers with ctx.AfterCommit —
which is how events.Emit publishes to a direct broker bus — runs inline,
inside the still-open transaction. If item 3 then fails and the whole thing rolls
back, subscribers have already been told about items 1 and 2. AfterCommit
returns false and logs a warning when this happens.
Let maniflex.Batch own the transaction instead, and the queue survives to its
commit. It works inside a custom action, where the Service step — and so
WithTransaction — never runs:
err := maniflex.Batch(ctx, func(b *maniflex.Batcher) error {
for _, item := range request.Items {
if _, err := srv.Execute(ctx.Ctx, maniflex.Invocation{
Model: item.Model,
Operation: maniflex.OpUpdate,
ID: item.ResourceID,
Body: item.Payload,
Auth: &requester,
Tx: ctx.Tx, // the batch's transaction
}); err != nil {
return err // rolls back, and the queued events go with it
}
}
return nil
})
Pass ctx.Ctx and ctx.Tx together: the queue is published on the context
beside the transaction it belongs to, and a hook is only queued when the two
agree — otherwise it would fire on a commit that says nothing about the
transaction the write went into. An outbox.Bus needs none of this, since the
event row is INSERTed in the transaction itself.
A non-2xx answer comes back as an error, not a value, which is what makes that
loop correct. if err != nil { return err } is the natural Go loop, and it has to
be the one that rolls back — handing a 422 back as (res, nil) would make the
naive loop commit items 1 and 2. Inspect the status with errors.As:
var execErr *maniflex.ExecuteError
if errors.As(err, &execErr) && execErr.StatusCode == http.StatusNotFound {
// the record is gone
}
The *APIResponse is returned either way, so you can read Data and Meta on a
success and the error envelope on a failure.
Authentication: ctx.InProcess()
An in-process call has no Authorization header, because there is no client. The
shipped auth.JWTAuth and auth.JWKSAuth stand aside when — and only when — the
request came from Execute and already carries a principal:
if ctx.InProcess() && ctx.Auth != nil {
return next() // identity already established, by trusted code
}
Everything else on the Auth step still runs against that principal: RequireRole,
RequireOwner, ABAC policies, your own middleware. An Execute with no Auth is
anonymous and is refused exactly as an anonymous request is — being internal is
not being authenticated.
ctx.InProcess() reads an unexported field that only Execute sets, so no client
and no middleware can claim it. Over HTTP it is always false, and the JWT header
path behaves bit-for-bit as it always has. Use it in your own middleware where a
check is about the transport rather than the caller:
server.Pipeline.Auth.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if ctx.InProcess() { return next() } // no browser to protect from CSRF
return csrfCheck(ctx, next)
}, maniflex.WithName("csrf"))
That is also the supported way to exempt a middleware from in-process calls. There
is deliberately no SkipMiddlewares list on Invocation: a named list of security
checks to switch off is the shape of the very bypass Execute exists to delete,
and middleware names are labels for trace logs, not identifiers (six shipped
middlewares share the name otel.span). Writing the exemption at the registration
site keeps it type-checked, greppable, and rename-proof.
No step can be skipped
There is no Steps bitmask. Execute runs the whole pipeline, so it cannot become
a quieter way into the database than the HTTP route it mirrors:
- Validate runs, so
mfxrules —required,enum,readonly,immutable— bind anExecuteexactly as they bind a request. AnExecuteis not a mass-assignment hole. - Auth runs, so authorisation registered there cannot be walked around by
naming yourself in
Invocation.Auth. - Service runs, so your business hooks fire.
Skipping Auth in particular would remove the isolation the injected principal is meant to be constrained by — which is the vulnerability, not the fix.
Invocation
| Field | Meaning |
|---|---|
Model | Registered model’s struct name, e.g. "Order". Required. |
Operation | OpCreate, OpRead, OpUpdate, OpDelete, OpList. Required. |
ID | Primary key. Required for read, update, delete. |
Body | Request body for create/update. []byte/json.RawMessage/string verbatim; anything else is marshalled. |
Query | url.Values carrying filters, sort, include, pagination. |
Auth | The principal. Typed, unforgeable, not a header. |
Tx | Transaction to join. The caller owns it — Execute neither commits nor rolls back. |
Header | Request headers for middleware that reads them (If-Match, Accept-Language). Not Authorization — use Auth. |
An update is a PATCH, not a replace. A field absent from Body is left alone,
exactly as over HTTP — so send a map when you mean to patch some columns:
Body: map[string]any{"status": "approved"}, // every other column untouched
This is not a re-implementation of presence semantics; Execute synthesises the
request and the real Deserialize step binds the body, so the rule is the one the
HTTP path has, not a second copy of it that agrees today and drifts later.
Refused operations. OpExport and OpReadAttachment stream bytes to a
response writer rather than producing a value, so there is nothing to return.
OpAction and OpSearch run trimmed pipelines of their own — call an action’s
handler directly, or ctx.Search.
Registration window. Execute builds the router on first use, exactly as
Handler() does, so Pipeline.Register and Server.Action must come first.
When to use it
| Need | Use |
|---|---|
| Serve a client | The generated routes |
| Run a model operation from Go, with auth and validation | Execute |
| Read or write from Go, without the pipeline | Typed CRUD / Model Accessor |
| Several operations, one transaction | Execute with a shared Tx, or Batch |
Reach for typed CRUD or ctx.GetModel when you want the database. Reach for
Execute when you want the rules — the same auth, validation, hooks, scoping
and response an HTTP caller would get, minus the socket.
Raw Queries & Query Models
The generated CRUD routes cover one-table reads. Anything that needs joins, aggregates, or custom SQL goes through one of the framework’s escape hatches: raw queries from middleware, or query models — read-only models backed by a hand-written SELECT.
Raw queries from middleware
ctx.RawQuery and ctx.RawExec run parameterised SQL through the active
transaction or the bare adapter:
rows, err := ctx.RawQuery(
`SELECT status, COUNT(*) AS n
FROM orders
WHERE organization_id = ?
GROUP BY status`,
ctx.Auth.TenantID,
)
rows is a []map[string]any with column-name keys. Placeholders are rebound to
the adapter’s dialect, so ? works on both SQLite and Postgres ($1, $2).
Never interpolate values into the query string — that’s a SQL injection.
ctx.RawExec is the same shape for non-SELECT statements and returns the
number of rows affected. ctx.RawQuery also returns the rows from a
data-modifying statement with a RETURNING clause (e.g. UPDATE … RETURNING id).
When ctx.Tx is non-nil, both methods participate in the active transaction
automatically. The built-in SQLite and Postgres adapters support this; if a
custom adapter’s transaction cannot run raw SQL, the call fails with
maniflex.ErrRawNotSupportedInTx rather than quietly running on a different
connection outside the transaction — where the write would commit on its own and
survive the rollback.
? is also a Postgres operator
Rebinding rewrites ? wherever it appears in executable code. A ? inside a
string literal, a quoted identifier, a dollar-quoted body or a comment is left
alone — but Postgres also spells three jsonb operators with it, and a bare
? is indistinguishable from a placeholder:
| Operator | Meaning | Rebinding |
|---|---|---|
?| | any of these keys exists | preserved |
?& | all of these keys exist | preserved |
? | this key exists | rewritten as a placeholder |
?| and ?& are safe because no placeholder is spelt that way. The bare
key-exists operator is not expressible in a raw query: write it as
data::jsonb @> '{"key": null}' or jsonb_exists(data, 'key'), both of which
mean the same thing without the ambiguous character.
Getting this wrong is loud rather than silent — the statement ends up asking for more parameters than were supplied, or naming a column that does not exist — so it fails on the first run rather than returning a wrong answer.
Portability pitfalls
Hand-written SQL runs on both SQLite and Postgres, which differ in ways the ORM normally hides:
- Parameterise booleans.
WHERE active = 1works on SQLite but errors on Postgres (operator does not exist: boolean = integer). Bind a Goboolinstead:WHERE active = ?,true. - Know your column names. A column’s name comes from the field’s
dbtag, else itsjsontag, else the snake-cased field name. A camelCasejsontag (json:"orderId") produces a camelCase column (orderId) — and Postgres folds unquoted identifiers in hand-written SQL to lowercase, soorderIdsilently won’t match. Keep raw SQL to snake_case columns, or set an explicitdb:"snake_case". SQLite is case-insensitive, so this only bites on Postgres. - Pin table names. Physical table names are pluralised implicitly
(
VisitorDay→visitor_days). When you reference tables in raw SQL, setModelConfig.TableNameso the name can’t drift from what your SQL expects.
Structured aggregation: ctx.Aggregate
For typed, validated aggregations there’s a structured builder that doesn’t require hand-written SQL:
rows, err := ctx.Aggregate("Order", maniflex.AggregateQuery{
Select: []maniflex.AggregateField{
{Op: maniflex.AggCount, As: "n"},
{Op: maniflex.AggSum, Field: "total", As: "revenue"},
},
GroupBy: []string{"status"},
Where: []*maniflex.FilterExpr{
{Field: "created_at", Operator: maniflex.OpGte, Value: "2026-01-01"},
},
Having: []maniflex.HavingClause{
{Alias: "revenue", Operator: maniflex.OpGt, Value: 1000},
},
OrderBy: []maniflex.SortExpr{{DBName: "revenue", Direction: maniflex.SortDesc}},
Limit: 100,
})
Each AggregateField.Op is one of AggCount, AggCountDistinct, AggSum,
AggAvg, AggMin, AggMax. Leave Field empty on AggCount to mean
COUNT(*). As overrides the alias used in the result row and in Having
or OrderBy; if omitted the default is <op>_<field> (or count for
COUNT(*)).
All DB column names — in Select.Field, GroupBy, and Where.Field — are
validated against the registered model. A typo fails fast with a clear
error rather than emitting bad SQL. OrderBy.DBName may reference either
an aggregate alias or a GroupBy column. Nested-relation filters are not
yet supported in Aggregate — use the raw-query escape hatch when you need
them.
When ctx.Tx is active the aggregate participates in the transaction,
matching RawQuery/GetModel(...).List.
To keep an aggregate on a parent column rather than compute it per request —
Order.PaidAmount maintained as SUM(OrderPayment.amount) — see
Maintained Rollups, the write-side counterpart of ctx.Aggregate.
Auto-generated aggregate endpoint
Opt a model into a built-in HTTP aggregation route with
ModelConfig.AggregateEnabled:
server.MustRegister(Order{}, maniflex.ModelConfig{AggregateEnabled: true})
This mounts GET /:model/aggregate. The aggregation is described by a JSON
document passed URL-encoded in the ?aggregate= query parameter, and the
group rows come back under the usual {"data": [...]} envelope:
GET /api/orders/aggregate?aggregate=<url-encoded JSON>
# where the JSON is:
{
"select": [{"op": "count", "as": "n"}, {"op": "sum", "field": "amount", "as": "total"}],
"group_by": ["status"],
"where": [{"field": "created_at", "operator": "gte", "value": "2026-01-01"}],
"having": [{"alias": "total", "operator": "gt", "value": 1000}],
"order_by": [{"field": "total", "direction": "desc"}],
"limit": 100
}
const spec = {
select: [{ op: "sum", field: "amount", as: "total" }],
group_by: ["status"],
};
const res = await fetch(
`/api/orders/aggregate?aggregate=${encodeURIComponent(JSON.stringify(spec))}`,
);
The spec travels in the query string, not in a request body, because this is a
GET: a body on a GET is dropped by many proxies and CDNs and cannot be sent
by fetch() at all, so an endpoint that needed one worked in development and
failed in production. A request body is not read; sending one gets a
400 INVALID_AGGREGATE pointing you at ?aggregate=.
op is one of count, count_distinct, sum, avg, min, max (omit
field on count for COUNT(*)). Field names use the same convention as
?filter=/?sort= — the JSON name (DB column name also accepted) — and every
referenced field must be mfx:"filterable" or mfx:"sortable", so the public
endpoint can never aggregate a hidden or sensitive column. The WHERE clause
takes every operator ?filter= takes, and renders them the same way, so a
filter counts what it lists.
The endpoint runs as the list operation: any auth or tenancy middleware you
registered for OpList applies unchanged (no separate registration needed), and
request ?filter= conditions — including middleware-injected tenancy
force-filters — are folded into the aggregate WHERE alongside the spec’s own
where. Filters sharing a group OR together and groups AND with everything
else, exactly as they do on the list endpoint:
GET /orders/aggregate?aggregate=…&filter[0]=status:eq:open&filter[0]=status:eq:draft
counts the orders that are open or draft — the same rows
GET /orders?filter[0]=… returns.
Aggregates report totals, so a WHERE clause that quietly means something other than what it says is worse here than on a list: there are no rows to eyeball. A filter the builder cannot render is therefore refused, or failed closed to match nothing — never degraded into a predicate that happens to parse.
Result types
A numeric aggregate comes back as a JSON number on every driver:
{"data": [{"region": "us", "total": 150}]}
That needs saying because it did not used to be true. Postgres computes SUM
over a BIGINT, and AVG over anything, as NUMERIC, which lib/pq hands
back as text — so the identical query answered "total": "150" there and
"total": 150 on SQLite, and every consumer had to accept both. The aggregate
path now normalises it, keeping the exact digits the database produced rather
than routing them through a float, so a total wider than 53 bits is not rounded
and a large round number is not re-rendered as 1e+06.
Only results that are numeric by definition are normalised — count,
count_distinct, sum, avg, and min/max over a numeric column. A min
over a text column returns text and is left alone, as is any group_by column:
coercing "00123" into 123 would be a new bug, not a fix.
ctx.RawQuery is deliberately untouched. It is the escape hatch and returns
what the driver returned, so a raw SUM still arrives as a string on Postgres —
see pkg/ledger for the small coercion helper that implies.
Expression aggregates
An aggregate normally totals one column. To total something the schema does not
store — revenue as price × count, margin as (price − cost) × count — register
a named expression on the model:
srv.MustRegisterAggregateExpr(maniflex.AggregateExpr{
Model: "OrderLine",
Name: "revenue",
Expr: maniflex.Mul(maniflex.Col("price"), maniflex.Col("count")),
Exposed: true,
})
Name is then usable wherever an aggregate takes a field:
ctx.Aggregate("OrderLine", maniflex.AggregateQuery{
Select: []maniflex.AggregateField{
{Op: maniflex.AggSum, Field: "revenue", As: "total"},
},
})
{"select": [{"op": "sum", "field": "revenue", "as": "total"}]}
Without this the figure is computed in Go over paged rows — a loop that has to read every row it sums, and pages while it does.
Expressions are built from Col, Lit, Add, Sub, Mul and Div, and
nest. The Expr interface is sealed, so those constructors are the only way to
make one: there is no path from a string to SQL. A Col name is resolved
against the model at registration (either spelling), and a Lit is bound as a
parameter, never interpolated.
Everything is checked when you register, not when a query runs — an unknown column, a non-numeric one, a name that collides with a field, or an expression nested past the depth limit is a startup error naming the problem.
Exposed is what publishes an expression to the generated HTTP endpoint, and it
is false by default: server-side ctx.Aggregate can always use one, while a
public client gets only what the application opted in, the same decision
mfx:"filterable" makes for a column. Expressions may be selected, not grouped
or sorted by; order_by can still name the alias.
Divdivides byNULLIF(divisor, 0). SQLite answersNULLfor a division by zero and Postgres raises, so an unwrapped division would return a row in a SQLite dev run and fail the identical request in Postgres production. The wrap costs you the error: dividing by zero yieldsNULLon both. Note also that integer columns divide with truncation on both drivers —Div(Col("total"), Col("count"))over twointcolumns is integer division. Use a float column, or multiply byLit(1.0)first.
The HTTP endpoint applies a default limit of 100 and clamps larger requested
limits to 200; a negative limit is invalid. It also caps select, group, where,
having, and order terms before SQL or placeholder lists are built. Configure
the defaults through Config.QueryLimits, or override a model through
ModelConfig.QueryLimits. Validated query mistakes return 400; database
failures return a redacted 500, and cancellation/deadline failures return 504.
These HTTP safeguards do not change programmatic ctx.Aggregate calls, whose
AggregateQuery.Limit remains explicit.
The route appears in the OpenAPI spec once the
model opts in, and the ?aggregate= parameter’s description carries the shape of
the JSON document along with this model’s aggregatable fields and exposed
expressions — the spec is the only place a client can learn them, since the
aggregation is a JSON value inside a query string rather than a typed body.
Tree traversal: ctx.RecursiveQuery
For self-referential models — categories, org charts, threaded comments, bill of
materials — ctx.RecursiveQuery issues a WITH RECURSIVE CTE without hand-writing
SQL:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: "some-uuid",
ParentField: "parent_id",
MaxDepth: 5,
})
// rows[0]["_depth"] == int64(0) is the root; rows[1..n] are descendants.
Every returned row is a map[string]any with all the model’s columns plus a
synthesised _depth integer (0 = the root node). Rows are ordered by _depth
ascending.
Fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
RootID | string | yes | — | Primary key of the starting node |
ParentField | string | yes | — | DB column that holds the parent’s ID, e.g. "parent_id" |
Direction | RecursiveDirection | no | RecursiveDescendants | Walk downward (RecursiveDescendants) or upward (RecursiveAncestors) |
MaxDepth | int | no | 0 → DefaultRecursiveMaxDepth (100) | Stop after this many levels; negative means unlimited |
Where | []*FilterExpr | no | nil | Additional filters applied in both the anchor and recursive members |
Descendant vs. ancestor traversal
Descendants (default) — walks down the tree. Given a root category it returns all children, grandchildren, etc.:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: rootID,
ParentField: "parent_id",
// Direction defaults to RecursiveDescendants
})
Ancestors — walks up the tree. Starting from a leaf, it returns the node itself, its parent, grandparent, and so on up to the root:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: leafID,
ParentField: "parent_id",
Direction: maniflex.RecursiveAncestors,
})
Limiting depth
MaxDepth: 1 returns the root plus its immediate children only — no further
descendants:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: rootID,
ParentField: "parent_id",
MaxDepth: 1, // depth 0 (root) + depth 1 (children)
})
Left at its zero value, MaxDepth applies maniflex.DefaultRecursiveMaxDepth
(100) rather than running unbounded. Pass a negative value for a genuinely
unlimited traversal:
MaxDepth: -1, // the whole hierarchy, however deep
Before v0.2.5,
0meant unlimited. If you relied on that for a hierarchy deeper than 100 levels, setMaxDepth: -1explicitly.
Cyclic data
A parent chain that loops — a row that is its own ancestor, whether directly
(parent_id pointing at itself) or through a chain — is not an error. The
traversal tracks the ids it has visited and stops at the repeat, so each node is
returned once:
A.parent_id = B, B.parent_id = A, root = A
→ A (_depth 0), B (_depth 1)
This holds in both directions. It matters because such data is not exotic: a category tree with no application-level guard against re-parenting a node under its own descendant will produce one eventually, and before v0.2.5 a single such request looped until it exhausted the process.
Note that a cycle is the only pathology here. Because the traversal follows one
scalar ParentField, every node has exactly one parent and so exactly one path
from the root — the same node cannot be reached twice by different routes.
Filtering nodes
Where filters are applied in both the anchor and recursive members, so a node
that fails the filter is excluded regardless of depth, and the traversal does
not continue through it:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: rootID,
ParentField: "parent_id",
Where: []*maniflex.FilterExpr{
{Field: "status", Operator: maniflex.OpEq, Value: "active"},
},
})
Nested-relation filters are not supported in RecursiveQuery — use
ctx.RawQuery for those cases.
Soft-delete awareness
When a model uses WithDeletedAt or a boolean soft-delete field, the recursive
query automatically excludes deleted records from both the anchor and recursive
members. No extra filter is needed.
Transaction participation
RecursiveQuery participates in ctx.Tx exactly like RawQuery:
tx, _ := ctx.BeginTx(ctx.Ctx, nil)
ctx.Tx = tx
defer tx.Rollback()
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
// traversal options
})
tx.Commit()
Database support
Both Postgres ($N placeholders) and SQLite (since 3.8.3, ? placeholders)
are handled transparently.
Stable read-only endpoints for aggregates
maniflex has no SQL-backed “query model” — a struct cannot be registered with a SQL body. For a stable, repeatable read endpoint over computed data you have two real building blocks:
- The auto-generated aggregate endpoint. Opt a model in with
ModelConfig.AggregateEnabledand it exposesGET /{model}/aggregate(see Auto-generated aggregate endpoint), driven byctx.Aggregate. Grouping, counts, sums/averages, and the standard?filter=all apply, and the route is in the OpenAPI spec — reach for it first for counts/sums/averages over a registered model. - A custom action running raw SQL. For a shape the aggregate endpoint cannot
express (a multi-table join, a window function), mount a
custom action whose handler runs
ctx.RawQueryand returns the rows; you own filtering and pagination inside the handler. On Postgres, back an expensive aggregate with a materialised view maintained in your migrations andSELECTfrom it in the handler.
When to use which
| Need | Tool |
|---|---|
| One-off aggregate inside an action or middleware | ctx.RawQuery |
| Counts / sums / averages as a stable, filterable endpoint | GET /{model}/aggregate (ctx.Aggregate) |
| A bespoke read endpoint (joins, window functions) | custom action + ctx.RawQuery |
| Tree traversal (descendants, ancestors, depth limit) | ctx.RecursiveQuery |
| Bulk mutation inside a single request | ctx.RawExec (inside a transaction) |
| Per-row business logic across many rows | Batch Operations & Sagas |
Performance notes
- Raw queries do not cache; each request executes the SQL. For a frequently-hit
aggregate exposed through a custom action, wrap it with
response.Cache(see Response Middleware) or maintain a summary table (the write-side Maintained Rollups are built for this). - Avoid unbounded scans — add
WHEREandLIMITclauses to any hand-written SQL when the underlying table is large. - For Postgres, a materialised view often beats recomputing an expensive aggregate per request; refresh it on a schedule and read it from the handler.
Maintained Rollups
A rollup is a denormalised aggregate column on a parent that the framework keeps
in step with its children. Order.PaidAmount as SUM(OrderPayment.amount),
StoreSite.ReviewsCount as COUNT(Review) — the columns an app would otherwise
recompute by hand in every write path, and which drift the moment one path
forgets.
Where ctx.Aggregate
computes an aggregate on read, a Rollup maintains one on write.
srv := maniflex.New(cfg)
srv.MustRegister(Order{}, OrderPayment{})
// Every OrderPayment write recomputes its Order's paid_amount.
srv.MustRegisterRollup(maniflex.Rollup{
Parent: "Order", ParentField: "paid_amount", Op: maniflex.AggSum,
Child: "OrderPayment", ChildField: "amount", On: "order_id",
})
// A rollup needs the child write to be transactional.
srv.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForModel("OrderPayment"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Configuration
| Field | Meaning |
|---|---|
Parent | model carrying the denormalised column |
ParentField | JSON name of that column |
Op | AggSum, AggCount, AggAvg, AggMin, or AggMax |
Child | model whose rows are aggregated |
ChildField | JSON name of the aggregated column (omit only for AggCount) |
On | JSON name of the foreign key on Child pointing to Parent’s id |
Where | optional []*FilterExpr narrowing which children the aggregate covers |
Field names are resolved and validated at registration — a typo is a startup
error naming the field, not a silently drifted total. This is the whole reason
the config is a typed struct and not a mfx:"rollup:sum(...)" tag: a tag would
be a mini query-language inside a string, invisible to go vet and failing in
the worst possible way.
RegisterRollup returns an error; MustRegisterRollup panics. Both must be
called before Start()/Handler().
Filtering the children
Where narrows the rows the aggregate covers — the captured payments rather than
every payment row:
srv.MustRegisterRollup(maniflex.Rollup{
Parent: "Order", ParentField: "captured_amount", Op: maniflex.AggSum,
Child: "OrderPayment", ChildField: "amount", On: "order_id",
Where: []*maniflex.FilterExpr{
{Field: "status", Operator: maniflex.OpEq, Value: "captured"},
},
})
The filters AND onto the foreign-key match and the soft-delete guard a rollup
always applies; filters sharing a Group >= 1 OR among themselves, as
everywhere else. Fields take the DB column name or the json name and are
validated at registration alongside the rest of the config.
Narrowing costs nothing in correctness. Because every child write recomputes its
parent from scratch, a child that leaves the filtered set — a payment going
captured → refunded — moves the total exactly as a delete would, and one
that enters it is added. There is no delta to keep in step.
Only flat filters on the child’s own columns are accepted. A nested-relation
filter (author.status) reads a column on a joined table and a locale filter
(name.ar) a key inside a JSON document; the recompute aggregates the child
table alone, so neither has anything to resolve against. Both are refused at
RegisterRollup, not at the first child write.
BackfillRollups deliberately does not apply Where when it discovers which
parents to visit. The filter says which children count, not which parents have
one: an order whose every payment fails the filter still has to be driven back to
0, and narrowing the discovery scan would skip it and leave the stale total in
place.
How it stays correct
On every create, update or delete of a child, the affected parent is
recomputed from scratch — Op(ChildField) over that parent’s live children —
and written to the parent column, inside the child write’s transaction.
Recomputing rather than applying a +delta/-delta is what makes it correct by
construction:
- Delete — the parent recomputes without the deleted row.
- Re-parenting — a PATCH that changes the child’s foreign key recomputes both the old and the new parent.
- Soft delete — a soft-deleted child is excluded, matching what a fresh aggregate returns.
- No drift — the column is always exactly the aggregate of the rows it summarises, never an accumulator that can diverge.
Empty sets follow SQL: a sum or count of no children is 0; a min/max/avg of no
children is null.
Concurrent writes
Recomputing from scratch is only correct if the recompute sees every committed sibling, so the parent’s row is locked before the aggregate runs, not after. Two concurrent child writes for the same parent therefore queue: the second waits at the lock, and by the time it aggregates the first has committed and its row is counted.
Locking after the aggregate — or not at all — would leave a window that loses
updates on Postgres under its default READ COMMITTED. Both transactions would
aggregate without seeing the other’s uncommitted row, the second UPDATE would
block on the parent row and then overwrite the first with its own stale total,
and nothing would error. That drift is permanent, in exactly the counter a rollup
exists to keep correct.
A write that touches two parents — a re-parenting update recomputes both — takes their locks in a fixed order, so two such writes moving children in opposite directions wait for each other rather than deadlocking.
SQLite is unaffected either way and pays nothing for the lock: its write lock is
the transaction’s own, taken at BEGIN, because db/sqlite opens write
connections with _txlock=immediate.
Transactions are required
A rollup refuses a child write that is not in a transaction, with
500 ROLLUP_NO_TX. Without one, a child insert could commit while the parent
update fails — the exact drift the rollup exists to prevent. Register
maniflex.WithTransaction on the Service step for the child’s writes, as above.
This follows the same fail-loud rule as mfx:"lock_scope".
Backfilling
Adding a rollup to a table that already has children, or reconciling a column edited out of band, needs a one-time recompute:
if err := srv.BackfillRollups(context.Background()); err != nil {
log.Fatal(err)
}
BackfillRollups recomputes every registered rollup for every parent from the
current child rows. It reconciles rather than locks — each parent is recomputed
independently, and a concurrent live write is simply picked up by its own rollup
— so prefer a quiet window for very large tables.
Cost and limits
- Each child write costs one parent row lock, one aggregate query and one parent update, inside the transaction. Concurrent children of the same parent serialise on that parent’s row from the lock until the transaction commits, which is what keeps the total consistent — so a very hot parent bounds how fast its children can be written.
- The rollup fires on the generated CRUD routes and any write that runs the DB
step. A write that bypasses the pipeline (a raw
INSERT, a direct adapter call) does not trigger it — runBackfillRollupsafter such a bulk load. - A cascade does trigger it, though it does not run the DB step either. A
child removed by its parent’s
onDelete:cascade, or re-pointed byonDelete:setNull, is taken out of its parent’s total: the sweep notes every parent it disturbs and recomputes each one once, in a fixed order, when the walk ends. Registering a rollup is also what keeps that edge out of a databaseON DELETEclause, which would remove the rows without telling anyone — see Relations. AggCountDistinctis not supported as a rollup op; write it by hand with an After-DB middleware if you need it.
Rollups and row-level scopes
A recompute writes the parent, and the parent is whatever row the child’s
foreign key names. Under a row-level scope (db.Tenancy, db.ForceFilter) that
matters: a child naming another tenant’s parent would move a total in a row its
author can neither read nor reach.
The scope closes this at the write. Every BelongsTo key a create or update sets
is read through the request’s own scope first, and a key pointing at a parent the
caller cannot see is refused with that parent’s 404 — so the child never
exists, and there is nothing left for a later recompute to count. See
ForceFilter for the exact rule and
the shapes it deliberately leaves alone.
Refusing the write rather than skipping the recompute is the part that matters.
BackfillRollups aggregates by foreign key with no scope at all — it has no
request to take one from — so a row merely left uncounted by the live path would
be folded into the victim’s total at the next reconcile.
A write that bypasses the pipeline still bypasses both, so a raw INSERT or a
direct adapter call can place a child under any parent. That is the same
exemption raw writes already have from the rollup itself.
Model Accessor
The generated REST routes expose a model over HTTP. Application code (action handlers, middlewares, event listeners, etc.) frequently needs to read or write a different model than the one the request targets. The model accessor is the in-process door for that work.
ctx.GetModel(name) returns an object exposing the five standard CRUD
operations bound to one registered model. Every call routes through the active
transaction, respects per-model database overrides, and returns
map[string]any rows without the HTTP round-trip. These are the same underlying
rows the REST layer reads, but keyed by DB column name and not run through
response marshalling — so hidden / writeonly columns are present and the
result is not the REST envelope shape. The main difference is the interaction
with middlewares, more on that below.
row, err := ctx.GetModel("User").Read(userID)
if err != nil {
return err
}
name := row["name"].(string)
The ModelAccessor interface
ctx.GetModel(name) yields a *ModelAccessor. All methods use
map[string]any, keyed by DB column name (each field’s mfx DB name), not the
JSON name — see Relationship to pipeline middleware
below.
| Method | Signature | Returns |
|---|---|---|
List | List(q *QueryParams) ([]map[string]any, error) | A page of rows |
Read | Read(id string) (map[string]any, error) | One row, or ErrNotFound |
Create | Create(data map[string]any) (map[string]any, error) | The stored row |
Update | Update(id string, data map[string]any) (map[string]any, error) | The updated row |
Increment | Increment(id string, deltas map[string]any) (map[string]any, error) | The updated row |
Delete | Delete(id string) error | — |
// List - q may be nil (page 1, limit 20, no filters or sorts).
admins, err := ctx.GetModel("User").List(&maniflex.QueryParams{
Filters: []*maniflex.FilterExpr{
{Field: "role", Operator: maniflex.OpEq, Value: "admin"},
},
Page: 1,
Limit: 100,
})
A FilterExpr you build in Go is not parsed the way a ?filter= is, but it is
resolved and normalised the same way once it reaches the query builder:
Fieldmay be either the column’s DB name or itsjsonname. A field the model does not have is a programming error, not client input, and aborts the request with500 INVALID_FILTERnaming the field. (The adapters also degrade such a filter to a false predicate, so a misspelt forced filter narrows to nothing rather than silently widening to everything.)Valueis coerced against the column like a URL value is: a realtime.Timeor*time.Timeis written in the canonical form the write path stores, and a boolean column accepts thetrue/falseand1/0spellings. You do not need to pre-format either.Valueforin,not_inandbetweenmay be a[]stringor[]anyas well as the comma-separated string a URL carries. A slice is taken as written, so a value containing a comma stays one element.Operatoris a bare string type, so a typo compiles. Use theOp*constants; an operator no adapter implements is refused rather than dropped.
filters := []*maniflex.FilterExpr{
// Both of these are correct, and both find the same rows.
{Field: "owner_id", Operator: maniflex.OpEq, Value: ownerID},
{Field: "ownerId", Operator: maniflex.OpEq, Value: ownerID},
// A time value needs no formatting.
{Field: "created_at", Operator: maniflex.OpLte, Value: time.Now()},
// A set may be a slice or the CSV spelling.
{Field: "status", Operator: maniflex.OpIn, Value: []string{"open", "held"}},
{Field: "status", Operator: maniflex.OpIn, Value: "open,held"},
}
// Create - returns the stored representation, id and defaults populated.
created, err := ctx.GetModel("User").Create(map[string]any{
"name": "Carol",
"email": "[email protected]",
"role": "viewer",
"password": "secret",
})
// Update - a partial patch; only the supplied keys are written.
updated, err := ctx.GetModel("User").Update(userID, map[string]any{
"role": "admin",
})
// Delete - soft-deletes when the model opts into soft delete, else hard-deletes.
err := ctx.GetModel("User").Delete(userID)
List accepts the same QueryParams the query
parser builds from ?filter= / ?sort= / ?page=; filters, sorts, and
pagination all apply. Read, Update, and Delete return
maniflex.ErrNotFound when the id is absent. Create and Update return a
*maniflex.ErrConstraint on unique or check violations, allowing DB errors to
be mapped to HTTP responses the same way the pipeline does (see
Error Handling).
Increment — arithmetic the database does
Use Increment wherever the new value is a function of the old one. Reading a
counter, adding in Go, and writing the sum back is a lost update waiting to
happen: two requests that read before either writes both store the same value,
one of the increments disappears, and both writes report success.
row, err := ctx.GetModel("Item").Increment(id, map[string]any{
"stock": -3, "reserved": 3,
})
Keys are DB column names, values are amounts to add — negative subtracts. All the columns move in one statement, so a transfer between two of them is never observable half-done and never left inconsistent by a partial failure.
Bounds are enforced in the same statement
“Decrement, but never below zero” is the shape most counters have, and checking
it in Go before calling Increment puts the race straight back. So the column’s
existing mfx:"min:" / mfx:"max:" bounds become conditions on the UPDATE
itself:
type Item struct {
maniflex.BaseModel
Stock int `json:"stock" db:"stock" mfx:"min:0"`
}
An increment that would cross a bound writes nothing and returns
maniflex.ErrIncrementOutOfBounds. That is deliberately distinct from
ErrNotFound, which the same zero-rows-matched result would otherwise be
indistinguishable from — one means “no such row” and the other means “not right
now”, and only the second is worth retrying:
switch {
case errors.Is(err, maniflex.ErrIncrementOutOfBounds):
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "not enough stock")
case errors.Is(err, maniflex.ErrNotFound):
ctx.Abort(http.StatusNotFound, "NOT_FOUND", "no such item")
}
The guard applies to the value after the increment, so a positive delta is never judged against a minimum it is moving away from.
What it will not do
- Non-numeric or unknown columns are refused, rather than dropped. A silently
ignored column would bump
updated_at, report success, and leave the counter where it was. mfx:"encrypted"columns are refused. The database holds ciphertext; adding to it would produce a number unrelated to the plaintext.- It never falls back to read-then-write. An adapter without atomic increment
returns
maniflex.ErrIncrementNotSupported. A fallback would be correct under test and lossy under load, with no way for the caller to tell which they got.
Inside a transaction it routes through ctx.Tx like every other accessor call,
so it rolls back with the rest of the request. When you need to read other
fields under the same lock and compute from them, ctx.LockForUpdate is still
the tool — Increment replaces the read-then-write, not the row lock.
Errors surface on first use
GetModel never returns an error directly. When the name is not registered — or
the registry is not wired onto the context — it returns an error accessor that
surfaces the failure on the first method call rather than at construction:
_, err := ctx.GetModel("NoSuchModel").List(nil)
// err: maniflex: model "NoSuchModel" is not registered
This keeps call sites terse (ctx.GetModel("User").Read(id)) without a nil check
on the accessor; the error is handled at the method call as usual.
Transactions
Accessor operations route through ctx.Tx whenever a transaction is active, so
work performed through the accessor commits or rolls back atomically with the
rest of the request:
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
ctx.Tx = tx
defer tx.Rollback() // no-op after a successful Commit
if _, err := ctx.GetModel("Order").Create(order); err != nil {
return err // the deferred Rollback undoes everything
}
if _, err := ctx.GetModel("Inventory").Update(sku, dec); err != nil {
return err
}
return tx.Commit()
The accessor must be obtained after
ctx.Txis set. An accessor captures whateverctx.Txholds at the momentGetModelis called. Obtaining it beforeBeginTxleaves its writes outside the transaction — and under SQLite they can deadlock against the open tx. Always callctx.GetModel(...)afterctx.Tx = tx:ctx.Tx = tx orders := ctx.GetModel("Order") // bound to ctx.Tx — safe
Per-model database routing
When a model is pinned to a non-default adapter (see Database Backends), the accessor resolves that model’s own adapter, so a cross-model read reaches the correct database automatically. The request transaction, however, belongs to the request’s adapter. When the target model lives on a different adapter, the accessor cannot enlist that transaction and instead runs the operation outside it against the correct adapter. Within a single adapter, transactional routing behaves exactly as above.
Relationship to pipeline middleware
The accessor talks directly to the database adapter (through the active
transaction when one is set). It does not run the request pipeline. Response
transforms, dynamic redaction, field-visibility rules, and any other registered
middleware never observe an accessor read or write — they operate on
ctx.Response/ctx.Body, which the accessor does not populate.
The one exception is the model’s own value constraints. enum, min and
max are checked on accessor and typed writes just as they are on an HTTP
request, and a violation returns an error before anything reaches the database:
_, err := ctx.GetModel("Account").Create(map[string]any{"role": "superuser"})
// err: maniflex: Account: field "role" must be one of: [user admin]
Pass maniflex.SkipValidation() where the violation is deliberate — backfilling
rows that predate an enum, say:
_, err := ctx.GetModel("Account").Create(legacyRow, maniflex.SkipValidation())
readonly and immutable are not applied here, deliberately. Those tags
say a value may not come from a client, and the accessor is not a client —
stamping such a column from a background job is much of why these APIs exist.
The split is between data integrity, which is a bug whoever writes it, and
access control, which depends on who is asking.
Before v0.2.5 no constraint was checked on these paths, so the same model accepted a value in Go that it answered
422for over HTTP.
This is not a substitute for validating untrusted input. Value constraints
are the model’s own rules, not an authorization boundary: decoding a request
body into a struct and handing it to maniflex.Create still writes whatever
fields it contains, including readonly ones. Untrusted bodies belong on the
HTTP path, or must be checked before they reach here.
The practical consequence: an accessor returns the record as stored, before any response-layer shaping. Given a redacting transform on the Response stage —
s.Pipeline.Response.Register(
response.TransformField("secret", func(any) any { return "[redacted]" }),
maniflex.ForModel("Account"),
maniflex.AtPosition(maniflex.After),
)
— a REST read of Account returns "secret": "[redacted]", because the
transform rewrites ctx.Response.Data after the DB step. An accessor read does
not:
row, _ := ctx.GetModel("Account").Read(id)
row["secret"] // the raw stored value — NOT "[redacted]"
Two further differences follow from bypassing the response marshaller:
- Keys are DB column names. Accessor maps are keyed by each field’s
mfxDB name, not the JSON name that response middleware and the API envelope use. Where the two names differ, the DB name is the one present in the map. hiddenandwriteonlyfields are included. The response marshaller stripsmfx:"hidden"andmfx:"writeonly"columns from every payload; the accessor does not. Their raw values are present in accessor maps.
This is by design — the accessor is trusted, in-process access for application logic, not a client-facing surface. It does mean that sanitisation implemented in response middleware (redaction, field hiding, CDN rewriting) is not inherited by data read through the accessor. When accessor output is forwarded outside the process, that shaping is the caller’s responsibility.
Typed accessor: map[string]any → *T
The string-named accessor is dynamic — suited to cases where the model name is data, or where maps are the natural representation. When the type is known at compile time, the typed CRUD free functions provide the same five operations against concrete structs. They resolve the model from the type parameter, so no name string must be kept in sync:
users, err := maniflex.List[User](ctx, nil) // []*User
u, err := maniflex.Read[User](ctx, id) // *User
created, err := maniflex.Create(ctx, &User{Name: "Jane"}) // *User
updated, err := maniflex.Update(ctx, id, &User{Name: "J."}) // *User
err := maniflex.Delete[User](ctx, id)
These route through ctx.Tx, honour per-model adapters, and return the same
ErrNotFound / ErrConstraint errors as the string-named accessor — they are
its typed counterpart, not a separate code path.
One difference is significant: maniflex.Update[T] performs a full-record
update. Every writable column except id is written from the supplied struct,
so any zero-valued field overwrites the stored value. For a partial patch — only
the specified keys — use the map-based ctx.GetModel(name).Update(id, data)
instead.
Columns the model marks mfx:"readonly" or mfx:"immutable" are not among
them, and neither are id, created_at and updated_at. Those tags used to be
enforced only by the Validate step, which the typed helpers do not run, so a
caller building a fresh struct to change one field stamped the zero time over the
row’s real created_at and blanked every readonly column along with it. Note
this is a guard against accidental blanking on a full-struct update, not an
access rule: maniflex.Create writes readonly columns from the struct, and
ctx.GetModel(name).Update writes whatever key you name. To write such a column
deliberately, name it:
ctx.GetModel("User").Update(id, map[string]any{"origin": "import"})
An explicit key is a different statement of intent from a struct that happens to carry a zero value — which is the whole distinction the two accessors draw.
Choosing between the accessors
| Requirement | Accessor |
|---|---|
| Dynamic access where the model name is a variable | ctx.GetModel(name) |
| Concrete structs and compile-time field names | maniflex.List[T] / Read[T] / … |
| A partial patch (only some fields) | ctx.GetModel(name).Update(id, data) |
| A full-record replace from a struct | maniflex.Update[T](ctx, id, &rec) |
| Joins, aggregates, custom SQL | Raw Queries & Aggregates |
Both accessors serve in-process cross-model work inside a request. Exposing a model over HTTP is a matter of registering it and letting the generated routes handle it; the accessor is the tool a handler reaches for once execution is already inside the pipeline.
Batch Operations & Sagas
The generated REST routes work on one row at a time. When the work fans out — inserting hundreds of rows from a CSV, fulfilling an order across inventory, payment, and shipping — two patterns appear: batch for atomic same-table work, and saga for multi-step workflows that span services.
Batch inside a single transaction
The simplest “bulk write” is a single transaction that issues many inserts or updates. Use an action endpoint so the request does not pass through the per-row Validate/Service hooks:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/users/import",
Handler: importUsers,
})
func importUsers(ctx *maniflex.ServerContext) error {
var req struct {
Users []map[string]any `json:"users"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
users := ctx.GetModel("User")
inserted := 0
for _, row := range req.Users {
if _, err := users.Create(row); err != nil {
ctx.Abort(http.StatusConflict, "IMPORT_FAILED",
fmt.Sprintf("row %d: %s", inserted, err.Error()))
return nil
}
inserted++
}
if err := tx.Commit(); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusCreated,
Data: map[string]any{"inserted": inserted},
}
return nil
}
Either every row commits or none does. Validation still runs because
ctx.GetModel(...).Create goes through the adapter — but per-row middleware
on the Service or Validate steps does not, since this is an action.
The maniflex.Batch helper
The hand-rolled BeginTx / ctx.Tx / defer Rollback dance above is
canonicalised by maniflex.Batch(ctx, func(*maniflex.Batcher) error)
(batch.go). It opens a transaction (or joins ctx.Tx if one is already set),
points ctx.Tx / ctx.Ctx at it for the duration of the callback, commits on
success, and rolls back on any returned error or ctx.Abort. The *Batcher
exposes the same five CRUD operations, all enlisted in the shared transaction:
err := maniflex.Batch(ctx, func(b *maniflex.Batcher) error {
inv, err := b.Create("Invoice", invoiceData)
if err != nil {
return err
}
for _, line := range lines {
line["invoice_id"] = inv["id"]
if _, err := b.Create("InvoiceLine", line); err != nil {
return err // rolls the whole batch back
}
}
return nil
})
Prefer maniflex.Batch over the manual transaction plumbing shown above — it
gets the rollback, abort, and ctx.Tx restoration semantics right. A single
batch transaction cannot span adapters; use a saga for cross-database work.
When Batch opens the transaction it also owns the ctx.AfterCommit queue:
callbacks registered inside the batch — a webhook, a cache invalidation, an
events.Emit to a direct broker bus — run after the batch commits, and are
dropped if it rolls back. Hand-rolled transaction plumbing has no such owner, so
those callbacks fire inline, inside the open transaction, where a later item’s
failure can no longer take them back. A Batch that joins an outer ctx.Tx
claims nothing: that transaction’s owner still decides.
For larger imports, batch the inserts (INSERT … VALUES (…), (…), … via
ctx.RawExec) and commit every N rows.
Cross-service workflows: sagas
When a workflow touches more than one downstream — charge a payment provider, reserve inventory, notify a partner — a single database transaction is no longer enough. The standard pattern is a saga: a sequence of forward steps, each with a compensating undo step.
maniflex ships a lightweight saga coordinator in pkg/saga:
saga.New(name).Step(name, do, undo).Execute(ctx, state) runs the forward
steps in order and, on any failure, runs the compensating undo functions in
reverse order.
err := saga.New("dispense_charge").
Step("create_invoice", createInvoice, voidInvoice).
Step("debit_ar_ledger", debitAR, reverseAR).
Step("deduct_stock", deductStock, restock).
Execute(ctx, saga.State{"patient_id": pid, "items": items})
It is an in-process, non-crash-safe coordinator: if the process dies
mid-saga, no compensation runs. For durable, crash-safe workflows the
transactional outbox pattern below remains the option to reach for (or wrap the
saga’s Execute inside a pkg/jobs job so a retry re-drives it). The outbox
mechanics fit naturally on the pipeline:
- Start a request transaction with
maniflex.WithTransaction. The local database changes commit or roll back atomically. - Make external calls from the Service step, recording an outbox row in the same transaction for each call that needs a follow-up.
- Process the outbox asynchronously with a background runner (from
jobs/redisor a similar) that performs the external call, marks the outbox row done, and triggers compensation on failure.
server.Pipeline.Service.Register(maniflex.WithTransaction(nil),
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil {
return err
}
// Both writes are in the same transaction as the Order insert.
_, err := ctx.GetModel("OutboxEvent").Create(map[string]any{
"kind": "charge-payment",
"payload": ctx.DBResult,
"status": "pending",
})
return err
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate), maniflex.AtPosition(maniflex.After))
A separate worker reads pending OutboxEvent rows and processes them. If the
payment provider fails, the worker records the failure and enqueues a
compensating action (e.g. “cancel the order”).
This pattern — transactional outbox + asynchronous worker — is the practical alternative to two-phase commit. It costs you one table and one background job but gains durability and isolation that distributed transactions cannot offer.
When to use which
| Workload | Pattern |
|---|---|
| Bulk same-table writes | One transaction, one action endpoint |
| Multi-table writes touching only your database | maniflex.WithTransaction on the request |
| Writes that depend on external systems | Transactional outbox + saga |
| Long-running background work | Background job (see Events & Background Jobs) |
See also
- Events & Background Jobs — running the outbox worker and emitting domain events.
- Transactions —
maniflex.WithTransaction, manualBeginTx, andLockForUpdate. - Custom Endpoints (Actions) — the right place to host a bulk endpoint.
Events & Background Jobs
maniflex offers two complementary mechanisms for work that happens outside the request pipeline: an event bus for lightweight domain-event fan-out, and a job queue for durable, retriable background work.
| Mechanism | When to use |
|---|---|
Event bus (events/*) | Notify other services or modules that something happened. Fire-and-forget. |
Job queue (jobs/*) | Do something reliably after a request — report generation, email, reconciliation. Needs retry and status tracking. |
Event bus
The event bus lets pipeline middleware publish domain events that any number of
subscribers consume independently. An events.Emit call on the DB-After step
publishes user.created, order.placed, etc. to whichever bus is wired up:
import (
"github.com/xaleel/maniflex/events"
"github.com/xaleel/maniflex/events/redis"
)
bus := redis.New(redisClient, "myapp") // prefix namespaces the Redis stream keys
server.Pipeline.DB.Register(
events.Emit(bus),
maniflex.ForModel("Order"),
maniflex.AtPosition(maniflex.After),
)
Publishing under a transaction
Emit never publishes before the write is durable. Which mechanism it uses
depends on the bus:
| Bus | Under WithTransaction |
|---|---|
outbox.Bus (a TxPublisher) | the event row is INSERTed inside the transaction, so event and write commit or roll back together |
| a direct broker bus (redis, kafka, nats, rabbitmq) | the publish is deferred to after the commit, and dropped if the transaction rolls back |
The second row is the weaker guarantee of the two: the commit can succeed and the
broker still be unreachable, and there is no record left to retry from. Use an
outbox.Bus when losing an event is worse than storing one — see
Example 3 for the pattern end to end.
If you register your own side effect from a middleware — a webhook, a cache
invalidation — reach for ctx.AfterCommit rather than firing it inline:
ctx.AfterCommit(func() { go notify(orderID) })
It runs the callback immediately when no transaction is active, so it is safe to use unconditionally. It runs synchronously after the commit, so start a goroutine for anything slow.
Deferral needs an owner. AfterCommit can only queue a callback for someone
who has promised to drain the queue, and that means a transaction the framework
opened: WithTransaction on the pipeline, or maniflex.Batch anywhere —
including inside a custom action, where the Service step never runs. Both drain
on commit and drop on rollback, and both publish the queue on ctx.Ctx, so an
Execute handed that same transaction queues onto it rather than firing on its
own.
A transaction you open yourself with ctx.BeginTx cannot be drained: you call
Commit, so only you know when it succeeded. AfterCommit then returns false
and runs the callback inline, inside the open transaction — where a rollback
can no longer take it back. It logs a warning saying so. Either let one of the
two owners hold the transaction, or do the side effect yourself after your
Commit returns.
Subscribers register a Subscription:
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: func(ctx context.Context, e events.Event) error { /* ... */ return nil },
})
For WebSocket fan-out, connect a realtime.Hub to the bus — see
Realtime / WebSockets.
What the payload carries
Event.Data is the written row, keyed by database column name — not by
json name, and not the response shape. It is deliberately not the response
projection: locale resolution and ctx.RedactResponseField masking are
decisions made for one requesting caller, and an event is durable, replayable,
and read by subscribers who never made that request.
Four kinds of column are stripped before the event leaves:
| Excluded | Why |
|---|---|
mfx:"hidden" | never leaves the server |
mfx:"writeonly" | never read back — password hashes and the like |
mfx:"encrypted" | the row reaching Emit is already decrypted, so emitting it would publish the plaintext |
{field}_hmac | the searchable digest companion of an encrypted+unique column |
This matters because the payload is not a transient in-memory value: it is
persisted verbatim to event_outbox.payload, replayed by the outbox relayer,
pushed to every WebSocket and SSE client through the hub, and written to
whatever the broker retains. An encrypted column emitted in the clear defeats
the at-rest guarantee everywhere downstream at once.
maniflex.RedactRecord(model, row) applies the same exclusion set, if you build
an event by hand (see the custom-action note below) or serialize ctx.DBResult
in your own middleware.
Ordering
Event order is not guaranteed by default. Two events are delivered in the order they were produced only while nothing fails: a delivery that fails is retried after a backoff, and later events keep flowing past it in the meantime. For one record that means an update can be applied before the create it follows, or an older state can overwrite a newer one.
The outbox can preserve order per aggregate:
bus.Relay(outbox.RelayOptions{OrderedByKey: true})
A row is then held back while an older unshipped row shares its ordering key,
which is the event’s Subject ("invoice/abc123" by default — the same value
the Kafka adapter uses as its partition key). Ordering is per key, so an
aggregate that is stuck holds up only its own events. Events with no Subject
are never held: they name no aggregate, so there is nothing to order them
against.
It is opt-in because it costs head-of-line blocking — while one row for an
aggregate is failing, every later row for that aggregate waits with it, up to
MaxAttempts and its backoff. Enabling it adds an ordering_key column;
Migrate adds it to an existing table.
This covers the outbox only. No broker adapter serialises per key on the consumer side, so with
Subscription.Concurrencyabove 1 two events for one record can be handled concurrently whatever order they arrived in. Kafka’s partition key gives per-partition ordering on the wire, not in the handler. Make handlers idempotent and safe to apply out of order, or setConcurrency: 1.
Idempotent delivery
Every broker adapter here is at-least-once, so a handler can see the same event
twice. That is by design and not a rare edge: a consumer that crashes with work
in flight replays it on restart, because the alternative — treating in-flight
work as consumed — loses it. events.Dedupe wraps a handler to suppress the
repeat:
store := events.NewSQLDedupeStore(db, "sqlite") // or NewInMemoryDedupeStore
store.Migrate(ctx)
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: events.Dedupe(store)(myHandler),
})
The ID is claimed before the handler runs, so two workers handed the same event concurrently do not both process it. If the handler then returns an error, the claim is released, so the retry is not mistaken for a duplicate — a transient failure retries normally and only a genuine redelivery is dropped.
Releasing is an optional capability: a custom DedupeStore may also implement
events.DedupeReleaser. Both bundled stores do. A store that does not cannot
undo its claim, so a handler that fails transiently loses the event rather than
retrying it — Dedupe logs a warning naming the store when you wrap one.
A claim outlives a process crash mid-handler: the ID stays recorded and the event is not reprocessed.
InMemoryDedupeStorebounds this with its TTL; forSQLDedupeStore, pruneevent_dedupeon whatever window you can tolerate replaying.
Handler panics
A panic in a Subscription.Handler is recovered and turned into a failed
attempt, so it flows into the same retry and dead-letter path as a returned
error: a handler that panics once and then succeeds has delivered its event, and
one that panics every time exhausts its attempts and dead-letters. Without that
the panic unwound into the broker’s delivery goroutine, where nothing recovered
it, and the Go runtime killed the process — one bad event type ending every
other subscription and the HTTP server with it.
Each panicking attempt is logged at ERROR with the event type, its id, and the stack. That is deliberately louder than a returned error, which only WARNs until its attempts run out: an error says a delivery failed, a panic says the handler is broken.
Subscription.OnPanic is the programmatic signal, for counting or alerting
without parsing logs:
events.Subscription{
Patterns: []string{"invoice.*"},
Handler: handleInvoice,
OnPanic: func(e events.Event, recovered any, stack []byte) {
metrics.Inc("event_handler_panic", "type", e.Type)
},
}
It fires once per panicking attempt — three times for a handler that panics
through MaxRetry: 2 — and runs on the delivery goroutine, so it must not
block. A panic inside the hook is not recovered again.
Dead-lettering
Set Subscription.DLQ (or RelayOptions.DLQType on the outbox relayer) to
re-publish an event that exhausted its attempts under a separate type, through
the same broker. Both paths produce the same payload:
ID | a fresh one — the original was already published under its ID, so reusing it gets the dead-letter dropped by any downstream deduper |
Type | the configured DLQ type |
Headers | every original header, plus original_type and original_id |
Everything else is copied unchanged, so the dead-letter carries the same Data,
Model, RecordID and TenantID as the event it came from.
A DLQ publish that itself fails is logged, and what happens to the event then
depends on the adapter. The core withholds the acknowledgement, so a broker
that acknowledges per message brings the event back and retries the
dead-letter with it — events/redis and events/nats do this. events/kafka
and events/rabbitmq acknowledge anyway and the event is gone; both have
reasons, and both are in the delivery matrix with
the rest of the row.
The outbox relayer keeps its row instead. The DLQ rides the same broker that
just failed every delivery attempt, so “the dead-letter failed too” is the
ordinary shape of an outage rather than an edge case. The row is retained and
stays claimable, and each later poll retries delivery and then the dead-letter,
until one is accepted. During a long outage the table therefore grows and drains
again on recovery; that is the trade an outbox makes, and losing the event is the
alternative. last_error records what happened, and retries back off to
relayBackoff(MaxAttempts) so a dead broker is not hammered.
Setting no DLQType is still an opt-out: dead-lettering is disabled and the row
is dropped once its attempts are spent, as documented on RelayOptions.DLQType.
An outbox row whose payload will not decode is dead-lettered immediately,
without consuming its retry budget: decoding is deterministic, so a retry parses
the same bytes and fails identically. That dead-letter is synthesised from the
row itself — original_id is the outbox row id and original_type its type
column, since there is no event to read them from — and carries the raw bytes as
Data with DataType: application/octet-stream, because they are the only
remaining evidence of what was written. The row is then marked shipped so the
sweep can reclaim it; last_error records that it was resolved rather than
delivered.
Custom actions emit manually.
events.Emitruns on the DB step, which custom actions skip — so aserver.Actionhandler never fires the middleware and must publish to the bus itself:data, _ := json.Marshal(maniflex.RedactRecord(ctx.Model, ctx.DBResult)) err := bus.Publish(ctx.Ctx, events.Event{ Type: "order.cancelled", Model: "Order", RecordID: orderID, Data: data, })Nothing redacts a hand-built payload for you — marshal
ctx.DBResultdirectly and you publish the plaintext of every encrypted column.For the transactional outbox, publish inside the action’s own transaction so the event commits atomically with the write.
Adapter delivery matrix
Available adapters: events/redis, events/kafka, events/nats,
events/rabbitmq. The in-process adapter (inproc.New() from
github.com/xaleel/maniflex/events/inproc) ships in the core module for tests.
Every one is at-least-once, and the retry, backoff, panic, and dead-letter
behaviour above is shared: it lives in events.DeliverWithRetry, not in the
adapters. What follows is where they genuinely differ. The notes after the
tables give the reasoning for each; these are the answers.
| Durability and replay | Redelivery after a consumer dies | Backpressure | Shutdown | |
|---|---|---|---|---|
events/redis | Redis Streams. Entries are retained after reading and trimmed at MaxLen (default 100,000), which drops unacked entries | pending list, taken over by a periodic XAUTOCLAIM sweep (ClaimMinIdle, default 5m) | Concurrency slots; the publisher is never blocked | in-flight deliveries cancelled; whatever was unacked is reclaimed by another consumer |
events/kafka | topic retention, consumer-group offsets | replays from the last committed offset | Concurrency slots | uncommitted offsets replay on restart |
events/nats | JetStream — you create the stream; durable consumers per Group | unacked messages redelivered on AckWait | Concurrency slots; the JetStream callback is refused once they are taken | unacked messages redelivered |
events/rabbitmq | queue durability is yours to declare. Reconnects only when built with NewWithDialer; a bus given a connection by New cannot redial it, and a drop ends every subscription on it permanently | unacked messages requeued when the channel closes, then redelivered to the rebuilt subscription | Concurrency slots, plus a broker-side prefetch bound (Options.Prefetch, default Concurrency) | unacked messages requeued |
inproc | none. Nothing published before a subscription exists, or while the process is down, survives | none | bounded queue; Publish returns inproc.ErrQueueFull — the only publisher-visible backpressure here | Close drains in-flight handlers within DrainTimeout |
Ordering is the same everywhere: no adapter serialises per key on the
consumer side, so Concurrency above 1 means two events for one record can be
handled at once. See Ordering.
What happens to an event that is not delivered
The four outcomes of a delivery, and what each adapter does with them. The third column is the one that differs.
| retries exhausted, no DLQ | DLQ publish succeeds | DLQ publish fails | abandoned mid-retry by a shutdown | |
|---|---|---|---|---|
events/redis | acked — a deliberate drop | acked | not acked → reclaimed, and the dead-letter is retried with it | not acked → reclaimed |
events/nats | acked | acked | not acked → redelivered on AckWait | not acked → redelivered |
events/kafka | committed | committed | committed — the event is gone | withheld → replays on restart |
events/rabbitmq | acked | acked | acked — the event is gone | withheld → requeued |
inproc | dropped | dropped | dropped | dropped |
Redis and NATS follow the core rule: the safety net you asked for did not catch the event, so it is not acknowledged as though it had.
Kafka cannot. Its commits are cumulative, so a gap left by a consumer that keeps running stalls every later commit on that partition and grows its pending map without bound. That is an unbounded failure traded against one event whose dead-lettering had already failed too. During a shutdown there is no later commit to stall, which is why that column differs.
RabbitMQ’s reason used to be unbounded prefetch. Prefetch is bounded now, and the answer did not change — it got worse. Withheld messages come back only when the channel closes, so with prefetch N a run of N unsettled deliveries fills the window and the consumer receives nothing further for the life of the process. A handful of poison events would end consumption entirely, which is a heavier failure than losing those events. Per-message acknowledgement is what lets Redis and NATS withhold one message without blocking the next; AMQP’s prefetch window does not.
These are behaviours, not implementation details: a change to any cell above is marked (behaviour change) in the CHANGELOG. The Kafka and RabbitMQ settle rules are named functions with tests over each column, so they cannot move quietly. The rest of the rows are pinned by review — acknowledgement on those adapters goes through the broker client’s own message type rather than a seam a test can drive.
A consumer that cannot reach its broker now says so. The
events/kafkaandevents/redisread loops retry forever — stopping would silently end consumption — but the retry is paced by an exponential, jittered backoff (100ms up to 30s) rather than the fixed one-second interval they used before. The jitter matters on recovery: without it every consumer in a fleet retries on the same tick and stampedes the broker the moment it comes back. Each failed read logs at WARN, escalating to ERROR once when the backoff first reaches its ceiling, so a sustained outage is findable without a long one burying the logs. The wait honours the context, so shutdown no longer blocks behind it. An idle stream is not a failure and does not advance the backoff. The policy isevents.ReadBackoffif you need the same behaviour elsewhere.
events/kafkaconnects in plaintext unless told otherwise. SetConfig.TLSandConfig.SASL— build the mechanism with kafka-go’s ownsasl/plainorsasl/scram— and both apply to publishing, consuming and topic creation, which each open their own connection. Managed clusters require at least one of the two. SASL/PLAIN sends credentials in the clear, so pair it with TLS.
inprocapplies backpressure. Each subscription has a bounded queue (Options.QueueSize, default 1024) drained byConcurrencyworkers.Publishnever blocks; it returnsinproc.ErrQueueFullwhen a subscription is full, so a handler slower than the publish rate shows up as an error rather than as memory growth.events.Emitcannot return it to you — it publishes after the response — so it logs an ERROR naming the event instead.
events/redisreclaims abandoned messages. A consumer that dies mid-delivery leaves its messages pending; each consumer runs a periodicXAUTOCLAIMsweep to take them over (Redis 6.2+; see minimum versions). TuneOptions.ClaimMinIdle(default 5m) above your slowest handler including retries — a message becomes claimable while its original consumer may still be working on it, so claiming early means delivering twice.Options.ConsumerName(default hostname+pid) must be stable across restarts: Redis never removes consumers from a group.
events/redistrims its streams, and trimming loses events. Each stream is capped atOptions.MaxLen(default 100,000). Streams are not queues — an entry stays after it is read — so an uncapped stream grows until Redis runs out of memory. But the cap is paid in events: trimming deletes the oldest entries without consulting consumer groups, so an entry a consumer has read but not yet acknowledged goes with them. The publisher gets no error and the consumer never learns it existed. SizeMaxLenagainst how far behind you are willing to let a consumer fall, not against throughput;MaxLenUnlimiteddisables trimming if you would rather bound growth with a Redismaxmemorypolicy. Each event’s two writes — its own stream and the hub — go out as oneMULTI/EXEC, so the two can never disagree about whether it happened.
events/rabbitmqreconnects only if it owns its connection. amqp091-go connections do not self-heal, so recovering from a drop means dialing a new one — which the bus can do only when it dialed the first:bus, err := rabbitmq.NewWithDialer(func() (*amqp.Connection, error) { return amqp.Dial(os.Getenv("AMQP_URL")) })A dropped subscription then re-declares its exchange, queue, bindings and prefetch on a fresh channel and resumes, paced by
Options.ReconnectBackoff. Unacked messages were requeued when the channel closed, so it resumes from them. Attempts are logged, escalating once to ERROR when the backoff reaches its ceiling.
Newtakes an*amqp.Connectionyou own and cannot replace it: a drop ends every subscription on it for the life of the process while the app keeps serving. That death is made loud — an ERROR naming the queue andOptions.OnSubscriptionClosed— but nothing below it brings the subscription back. PreferNewWithDialerfor anything long-running.
Options.Prefetchbounds what the broker may push at one consumer, defaulting to the subscription’sConcurrency. ItsPublishwaits for a broker confirm, so a failed publish is reported rather than assumed delivered.
Broker adapters are nested modules. Adapters with heavy dependencies (e.g. NATS) ship as their own Go modules —
go get github.com/xaleel/maniflex/events/nats— so the core module stays dependency-light. Pin each one explicitly ingo.mod.
NATS: one bus binds one JetStream stream.
nats.New(nc, stream)ties a bus to a single stream for both publish and subscribe, and JetStream rejects two streams whose subjects overlap. A service that publishes its own subjects while consuming another service’s subjects needs a stream-ownership decision — either a shared stream, or consume from the owning service’s stream. Create the stream yourself (scoped to your business subjects, not">");Newdoes not create it.
NATS durable names changed, and
Groupnow works.Subscription.Groupbecomes a JetStream queue group, so replicas sharing aGroupshare the work and each event is handled once by the group — whatGroupalready meant on Kafka and Redis. Previously the adapter bound a durable with no queue group, which accepts exactly one subscription, so a second replica was refused outright. Durable names also gained a hash suffix ({group}-{subject}-{hash}) because the old form was not unique:.renders as_and>asall, soinvoice.*andinvoice.allproduced the same name and the second subscription was rejected withErrSubjectMismatch.Both are breaking for existing deployments. Consumers created by an earlier version are not reused: the old durables keep their position and go unconsumed, and the new ones start at the stream’s default delivery policy — which may replay retained events. Drain the old consumers before upgrading a busy deployment, then remove them (
nats consumer ls <stream>,nats consumer rm).
Job queue
The jobs/ packages provide a producer/consumer queue with retries, dead-letter
routing, and optional status persistence through the REST layer.
Adapters
| Package | Backing store | Transactional enqueue | Best for |
|---|---|---|---|
jobs/inproc | goroutine pool | no (best-effort) | tests, single-binary dev |
jobs/sql | Postgres or SQLite | yes — enqueue in the same ctx.Tx | production (recommended) |
jobs/redis | Redis Streams / BRPOP | no | high-throughput fleets |
All three share the same jobs.Queue and jobs.Source interfaces so swapping
adapters is a one-line change.
jobs/sqlon SQLite needs SQLite 3.35 or newer (March 2021 — both common Go drivers bundle something far newer). The claim is oneUPDATE … RETURNING, so a worker receives exactly the rows it stamped. It was previously anUPDATEfollowed by aSELECTthat re-found those rows by theirlease_untiltimestamp — which is not a unique identifier, so two claims taken in the same clock tick each matched the other’s rows. That handed one job to several workers, and stranded others asrunningwith an attempt already spent that no worker had ever received.
NotBeforeis honoured to the nanosecond on SQLite. SQLite compares the timestamp columns as text, so the stored format has to sort the same way the instants do. Timestamps are written with a fixed-width fractional part for that reason; an earlier variable-width format could fire a scheduled job up to a second early or late whenever itsNotBeforefell on a whole second.
jobs/sqlrecovers jobs from a crashed worker. A worker that dies after claiming a job leaves the rowrunningwith a lease nobody is renewing.Dequeuesweeps such rows before it claims, returning them toenqueuedonce their lease has lapsed, so the job is redelivered rather than stranded. The lease is a crash-detection window, not a job-duration limit — a live worker keeps renewing — so only a worker that has stopped renewing lets its jobs age out. A reclaim costs a retry attempt, which bounds a crash loop; a job whose attempts have already reachedMaxRetryis dead-lettered by the sweep instead of being handed out again, withlast_errorrecording the expired lease. That case is a poison pill: the worker running it died, so the next one likely dies the same way, and it never reaches the code that would otherwise dead-letter it. The sweep is rate-limited to roughly a tenth of the lease, so crash recovery lags the timeout slightly rather than costing a write on every poll.
jobs/redisrecovers jobs from a crashed worker. A worker that dies after claiming a job but before completing it leaves the job in the consumer group’s pending list.Dequeuereclaims such entries (viaXAUTOCLAIM) once they have been idle pastOptions.ReclaimMinIdle(default 5m), so the job is redelivered rather than lost.ReclaimMinIdleis a crash-detection window, not a job-duration limit: a live worker renews its hold on a long-running job through the worker’s lease-renewal loop, so only a worker that has stopped renewing — crashed or hung — lets its jobs age past the threshold. Give each worker a uniqueOptions.ConsumerID(the default ismaniflex-{hostname}-{pid}); two workers sharing one ID are a single consumer to Redis and share one pending list, which defeats per-worker recovery.
Delayed jobs are promoted once, even with many replicas. A job enqueued with
NotBefore/EnqueueAtwaits in a sorted set until due, then the promoter moves it to the stream. Every replica runs a promoter, but the move is a single atomic server-side script, so Redis serialises them: whichever replica runs first claims the due jobs and the rest find them already gone — a delayed job is delivered once, not once per replica, and a dropped connection cannot leave one half-moved.
Defining and enqueueing a job
import (
"database/sql"
"github.com/xaleel/maniflex/jobs"
jobssql "github.com/xaleel/maniflex/jobs/sql"
)
// jobs/sql takes a database/sql handle, not the maniflex DB adapter.
db, _ := sql.Open("sqlite", "./app.db")
queue := jobssql.New(db)
if err := jobssql.Migrate(ctx, db, "sqlite"); err != nil { /* ... */ } // "postgres" on PG
Driver dialect.
Newdetects whether the handle is Postgres or SQLite from the driver, recognisinglib/pqandjackc/pgx. The dialect fixes both the SQL and the placeholder style ($1vs?), so a wrong guess fails outright rather than running slow. If you use a Postgres driver it does not recognise, state it explicitly withjobssql.New(db, jobssql.WithDriver("postgres"))— the same value you pass toMigrate.
Lanes and secrets
-
Separate lanes: run an isolated queue on its own table so a type-restricted worker can’t interfere with other jobs. Pass
WithTableNameto bothNewandMigrate(indexes are renamed to match, so two queues share one DB):otp := jobssql.New(db, jobssql.WithTableName("otp_jobs")) jobssql.Migrate(ctx, db, "sqlite", jobssql.WithTableName("otp_jobs"))The name must be a plain SQL identifier (
[A-Za-z_][A-Za-z0-9_]*). It is interpolated directly into every statement and into the migration DDL — a table reference cannot be bound as a parameter — so anything else is rejected:Migratereturns an error andNewpanics. Do not build it from user input. -
Encrypt payloads at rest: payloads are stored as cleartext JSON by default. Pass
WithKeyProvider(kp, keyID)to encrypt the payload column with the same key machinery asmfx:"encrypted"struct fields —encryption.EnvKeyProviderorencryption.VaultKeyProvider:kp := &encryption.EnvKeyProvider{Prefix: "MYAPP_KEY"} // reads MYAPP_KEY_JOBS q := jobssql.New(db, jobssql.WithKeyProvider(kp, "jobs"))Stored values are
enc:<base64(envelope)>, and the envelope carries the key id, so rotation works: point the queue at a new key id and jobs already in the queue still decrypt, as long as the provider can still resolve the id they were written under. Retire an old key only once no job holds a payload encrypted with it.The older
WithPayloadCipher(cipher)(anyEncrypt([]byte)/Decrypt([]byte)implementation, storedencq:) still works and is still read, but records no key id — so rotating its key strands every job still holding one. Prefer a key provider. Both may be set while migrating: new rows are written through the provider, existingencq:rows keep decrypting with the cipher.A payload that cannot be decrypted — key retired, option removed — is never handed to a handler as-is; the row is quarantined as
deadwith the reason recorded. -
Unhandled types are requeued, not killed: a worker that lacks a handler for a job’s type now requeues it (so another worker can claim it) instead of dead-lettering it — safe for a type-restricted worker sharing a table.
-
A row that will not decode is quarantined, not fatal: if a job’s stored payload cannot be decoded or decrypted — a rotated cipher key, a corrupted value — that one row is marked
deadwith the reason in itslast_error, and the rest of the batch it was claimed with dispatches normally. Previously the wholeDequeuefailed, and because the claim had already committed, every good job claimed alongside it was stranded asrunning. Quarantined rows stay visible throughInspector.List/Get(with an empty payload) so you can see what happened and re-enqueue if the cause was recoverable. -
Visibility timeout: a claimed job is invisible to other workers until its lease expires, after which another
Dequeuemay reclaim it. The default is 5 minutes;WithLeaseDuration(d)changes it. It must exceed how long a handler runs, or a still-running job is reclaimed and executed a second time — but a long handler does not need a large value if the worker renews the lease, which it does automatically. Renewal only ever extends: a renewal horizon shorter than the current lease leaves it alone, so renewing can never make a job reclaimable sooner than the timeout promises.q := jobssql.New(db, jobssql.WithLeaseDuration(30*time.Minute))The timeout is therefore also the crash-recovery window: it is how long a dead worker’s jobs wait before another one picks them up. Trading it off against handler duration is the whole decision — too short re-runs live work, too long delays recovery. Renewal is what lets you keep it short.
// Inside a pipeline middleware or action handler:
id, err := queue.Enqueue(ctx, jobs.Job{
Type: "send_receipt",
ActorID: ctx.Auth.UserID,
TenantID: ctx.Auth.TenantID,
Payload: json.RawMessage(`{"order_id":"abc"}`),
})
Fields worth knowing:
| Field | Effect |
|---|---|
Type | Selects the handler on the Worker (required) |
MaxRetry | Max attempts before dead. Default 3. |
NotBefore | Delay execution until this time (use EnqueueAt as a shortcut) |
GroupKey | At most one job with this key runs at a time — useful for per-tenant serialisation |
TraceID | Propagated to the handler context for end-to-end trace correlation |
The Worker
import "github.com/xaleel/maniflex/jobs"
w, err := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Handlers: map[string]jobs.Handler{
"send_receipt": func(ctx context.Context, j jobs.Job) (jobs.Result, error) {
var p struct{ OrderID string `json:"order_id"` }
json.Unmarshal(j.Payload, &p)
return jobs.Result{}, mailer.SendReceipt(ctx, p.OrderID)
},
},
Concurrency: 8, // goroutines; default = GOMAXPROCS
Logger: slog.Default(),
})
ctx, cancel := context.WithCancel(context.Background())
go w.Run(ctx)
// On shutdown:
cancel()
w.Shutdown(shutdownCtx)
Cancelling the run context does not orphan a finished job. When you
cancel()to stop the worker, a handler already running is interrupted through its context — but once a handler returns, the worker records the outcome (ack, retry, or dead-letter) on a context detached from the cancellation, so a job that just succeeded is acknowledged and not redelivered on the next start. Those writes are still bounded, so a hung queue backend cannot holdShutdownopen indefinitely.
Migrate before you launch background goroutines.
server.Go(fn)(and a barego w.Run(ctx)) starts running immediately, butAutoMigrateonly runs insideStart(). A worker that touches a table beforeStart()migrates it races table creation. When you launch workers yourself, callserver.MigrateOnly(ctx)afterSetDBand before starting them so the tables exist.
Result carries an optional URL (pre-signed storage URL for file outputs)
and Output (small structured JSON). Both are surfaced through the status model
below.
StatusModel — REST polling
Mount the status model once, alongside other model registrations:
import jobsmaniflex "github.com/xaleel/maniflex/jobs/maniflex"
sink, queue, err := jobsmaniflex.Mount(server, rawQueue)
if err != nil { log.Fatal(err) }
// Pass sink to the worker:
w, _ := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Status: sink,
Handlers: handlers,
})
Mount registers a StatusModel (table job_statuses) and returns:
sink— ajobs.StatusSinkto pass toWorkerConfig.Status; the worker writes a row for every lifecycle transition.queue— a wrappedjobs.Queue; everyEnqueuecall creates an initialenqueuedstatus row so clients can poll immediately.
The REST layer exposes these endpoints automatically (no extra code):
GET /api/job_statuses list (filterable, paginated)
GET /api/job_statuses/:id single row
POST /api/job_statuses → 405 (worker-only)
A typical client flow after an action returns {"job_id": "abc"}:
GET /api/job_statuses/abc
→ {"data": {"status": "enqueued", ...}}
GET /api/job_statuses/abc (poll until done)
→ {"data": {"status": "succeeded", "result_url": "https://...", "completed_at": "..."}}
Status values: enqueued → running → succeeded | failed | dead | cancelled.
Scope
The built-in force-filter restricts the list to the caller’s own actor_id (and
tenant_id when set); callers with the admin role see everything. Override the
role name with MountOptions.AdminRole.
A request with no authenticated caller is refused with 401. These rows are
per-actor, so there is no scope to apply without an identity — and answering
without one would return every actor’s and every tenant’s job metadata. This
means the endpoints are only useful behind whatever authentication your app
installs; a nil ctx.Auth is treated as a misconfiguration, not as permission.
Atomic enqueue with jobs/sql
When jobs/sql is the adapter and a maniflex.WithTransaction middleware is
active, queue.Enqueue runs its INSERT through the same *sql.Tx:
// Service step:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { // DB write commits first
return err
}
_, err := queue.Enqueue(ctx.Ctx, jobs.Job{
Type: "reconcile_inventory",
Payload: json.RawMessage(`{"product_id":"` + productID + `"}`),
})
return err
}, maniflex.ForModel("Order"), maniflex.AtPosition(maniflex.After))
If the transaction rolls back, the job row never appears. If the process crashes after commit, the job row is durable and the worker will pick it up. This eliminates the “DB committed but job lost” race that an in-memory queue cannot prevent.
GroupKey — serialised execution
Set GroupKey to ensure at most one job for a given key runs at a time:
queue.Enqueue(ctx, jobs.Job{
Type: "generate_payroll",
GroupKey: "tenant:" + tenantID, // one payroll run per tenant at a time
})
jobs/inproc tracks running keys in memory. jobs/sql enforces the key on two
levels: the claim query ranks candidates with ROW_NUMBER() OVER (PARTITION BY group_key) and takes only the top row per key, so a single Dequeue — however
large its batch — never starts two jobs of one key; and a partial unique index
on (group_key) WHERE status = 'running' makes a second running job of a key
impossible even across two workers claiming at the same instant, which the query
alone cannot prevent on Postgres. An empty GroupKey opts out of serialisation
entirely, so unkeyed jobs run fully in parallel.
Upgrade note: the partial unique index is created by
Migrate. If a queue already contains two running jobs for one key — the very bug this closes — creating the index fails and the migration stops. Drain or clear the duplicate running rows, then migrate.
Retry and dead-letter
When a handler returns an error the worker re-queues the job after an
exponential backoff (base 1 s, cap 5 min). After Job.MaxRetry attempts the
job is marked dead and the status row records the final error. Set
WorkerConfig.DLQType to route dead jobs to a separate handler for inspection
or alerting.
Job.Backoff overrides the policy per job:
jobs.Job{
Type: "sync_ledger",
MaxRetry: 20,
Backoff: jobs.ExponentialBackoff{Base: time.Minute, Max: time.Hour},
}
Both fields take a default when left at zero: Base zero means 1 s, and Max
zero means uncapped — the delay keeps doubling and saturates at the largest
representable time.Duration rather than overflowing. A Base of zero used to
mean no delay at all, so a literal that set only Max silently retried in a
tight loop; use jobs.FixedBackoff{} if that is what you want. Delays are
clamped rather than allowed to wrap, so a long MaxRetry with a coarse Base
cannot produce a negative delay and an immediate re-run.
Jobs of an unhandled type. A worker that dequeues a job whose Type it has
no handler for does not fail or drop it — a type-restricted worker sharing a
queue with others must let a job pass to the worker that does handle it. It
requeues the job instead, without spending a retry attempt, so the job’s
budget is preserved for its real handler. To stop a type that no worker
handles from bouncing forever, the worker counts these requeues in a header and
dead-letters the job once it reaches WorkerConfig.MaxUnhandledRequeues
(default 20) — surfacing the misconfiguration rather than storming. This
requires the queue to implement jobs.Requeuer (all three built-in adapters
do); a custom adapter that does not falls back to the older unbounded Nack.
The status row follows the job either way, so it never reads as executing on a
worker that has already let it go. Through Requeuer it returns to enqueued,
since a requeue spends no attempt; through the Nack fallback it becomes
failed — or dead once the budget is spent — because that is what Nack
does. The row is written only after the queue write succeeds: if the requeue
itself fails the job is still held, and running is then the truthful status.
Cancellation
When the inner queue implements jobs.Cancellable (both jobs/inproc and
jobs/sql do), the wrapped queue returned by Mount also implements it:
c := queue.(jobs.Cancellable)
c.Cancel(ctx, jobID) // marks the job cancelled in the queue and updates the status row
Only jobs that have not yet started can be cancelled; a running job must finish or fail before the status row moves.
The two adapters retain a cancelled job differently: jobs/sql keeps the row with
status='cancelled', while jobs/inproc drops the entry — as it does for a
succeeded or dead-lettered job — so its own Inspector reports only live jobs.
Either way the job_statuses row created by Mount is the durable record of the
outcome, which is what clients poll.
Completion events (optional)
Set WorkerConfig.EventBus to publish job.{type}.completed and
job.{type}.failed events on every terminal transition. Pair with a
realtime.Hub to push completion notifications to connected clients without
polling:
w, _ := jobs.NewWorker(jobs.WorkerConfig{
// ...
EventBus: bus, // any value implementing Publish(ctx, type, payload) error
})
Scheduled jobs with jobs/cron
jobs/cron provides a minimal ticker that calls Queue.EnqueueAt on a fixed
interval. It does not offer durable cron (if a replica is down at fire time the
tick is missed); for durable scheduling, combine jobs/sql with a
next_fire_at column in your model. For field-based transitions (auto-publish,
auto-expire), see Scheduled Fields & Runner.
Schedules are fixed intervals (Every), not cron expressions:
import (
"time"
"github.com/xaleel/maniflex/jobs"
"github.com/xaleel/maniflex/jobs/cron"
)
cr := cron.New(queue, nil) // nil logger → slog.Default()
cr.Add(cron.Entry{
Every: 24 * time.Hour,
Job: jobs.Job{Type: "daily_report"},
})
cr.Start(ctx) // returns immediately; cr.Stop() halts the tickers
Start is idempotent — a duplicate call is a safe no-op, never a second set of
tickers. Add must come before it: entries registered after Start are not
scheduled. A Scheduler is not reusable, so Start after Stop stays stopped.
Running cron on more than one replica
A Scheduler ticks in its own process and knows nothing about its peers, so
three replicas each running one enqueue daily_report three times a day.
Run the Scheduler in exactly one process, or pass a cron.Locker and let the
replicas elect a single winner per interval:
cr := cron.New(queue, nil, cron.WithLocker(myLocker))
// Locker is the whole interface. Return true to exactly one caller per key.
type Locker interface {
Acquire(ctx context.Context, key string, ttl time.Duration) (bool, error)
}
The framework ships no implementation — use whatever the deployment already runs. A Redis one is three lines:
func (l redisLocker) Acquire(ctx context.Context, key string, ttl time.Duration) (bool, error) {
return l.client.SetNX(ctx, key, "1", ttl).Result()
}
Keys look like cron|daily_report|24h0m0s|1784678400 and carry the fire time
truncated to Every, so replicas that started at different moments still agree
on which interval a tick belongs to — one ticking at 00:03 and another at 00:47
contend for the same midnight key. Entry.Name overrides the daily_report
segment; set it when two entries share both a Type and an interval.
Two properties worth knowing:
- There is no release. The lock marks the interval as claimed, not the work
as in progress, so it must outlive the firing — let it expire with its TTL
(which is
Every). Deleting it early lets the next replica to tick within the same interval fire again. - A Locker error fails open and the job fires, matching
idempotency. A lock backend outage produces visible duplicates rather than a nightly job that quietly never ran.
Realtime / WebSockets
maniflex is a synchronous request/response framework, but the realtime package
ships a first-class event hub that pushes domain events to browsers over
WebSocket and Server-Sent Events. It is a pure consumer of the
event bus: producers publish through events.Emit exactly as
they would for any other subscriber, and the hub fans those events out to
connected clients.
Nothing about realtime leaks into a CRUD-only app — the hub is mounted by your
own code outside server.Handler(), so a blog that never imports realtime
pays no websocket dependency, goroutine, or shutdown phase.
The shape of it
import (
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/events"
"github.com/xaleel/maniflex/events/inproc"
"github.com/xaleel/maniflex/realtime"
)
bus := inproc.New() // or events/redis, events/nats, … for multi-replica
// Producer: every create/update/delete publishes a domain event.
server.Pipeline.DB.Register(
events.Emit(bus, events.EmitConfig{Source: "billing"}),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
// Consumer: the hub fans those events out to clients.
hub, err := realtime.NewHub(realtime.HubConfig{Bus: bus})
if err != nil {
log.Fatal(err)
}
r := chi.NewRouter()
r.Mount("/api", server.Handler())
r.Handle("/ws", hub.Handler()) // WebSocket upgrade
r.Handle("/sse", hub.SSEHandler()) // Server-Sent Events fallback
http.ListenAndServe(":8080", r)
Removing realtime is a one-line revert: drop the two r.Handle lines and the
events.Emit registration.
Topics
Events are addressed by their CloudEvents type — a dotted string like
invoice.created or queue.position_changed. Clients subscribe with glob
patterns (the same matcher the event bus uses):
| Pattern | Matches |
|---|---|
invoice.* | invoice.created, invoice.updated, … |
*.created | any ….created event |
* | every event |
HubConfig.AllowPatterns is an optional whitelist of subscribable patterns; an
empty list allows any. A client that asks for a forbidden pattern gets a
FORBIDDEN_PATTERN error (WS) or a 403 (SSE).
WebSocket protocol
The client speaks a tiny JSON protocol over the socket:
client → server server → client
{"op":"subscribe","patterns":["invoice.*"]} {"op":"ack","subId":"s_1"}
{"op":"unsubscribe","subId":"s_1"} {"op":"event","subId":"s_1","data":<event>}
{"op":"ping"} {"op":"pong"}
{"op":"error","code":"…","msg":"…"}
The data field is the full CloudEvents JSON document, so a
browser can parse it with any CE SDK.
Each message must arrive as a single, masked WebSocket frame — which is what
every browser WebSocket sends. The hub does not reassemble fragmented
messages: a fragmented or unmasked frame, a set RSV bit, a reserved opcode, or
an over-long control frame is a protocol error and the connection is closed with
1002. This is not a limitation in practice — the inbound vocabulary above is a
few dozen bytes of JSON that no client fragments.
SSE protocol
SSE is push-only and subscribes via query parameters — ideal for corporate networks that break WebSockets:
GET /sse?subscribe=invoice.*&subscribe=queue.position_changed
Each event arrives as a standard data: frame whose body is the same
CloudEvents JSON.
The SSE response always sets X-Accel-Buffering: no — a safe default that stops
NGINX from buffering the stream and delivering events in batches.
Authentication
Connections are authenticated once, on connect (never per message). Supply
an Authenticator; the default AnonymousOnly{} accepts everyone.
hub, _ := realtime.NewHub(realtime.HubConfig{
Bus: bus,
Authenticator: realtime.BearerToken(func(tok string) (*realtime.Principal, error) {
claims, err := verifyMyJWT(tok)
if err != nil {
return nil, err
}
return &realtime.Principal{UserID: claims.Sub, TenantID: claims.Tenant, Roles: claims.Roles}, nil
}),
})
BearerToken pulls the token from the Authorization: Bearer … header, the
?access_token= query parameter (browsers can’t set headers on WebSocket()),
or the Sec-WebSocket-Protocol: access_token.<token> subprotocol. Composite
tries several authenticators in order.
Origin checking
Set Origins to restrict which web origins may open a connection; leave it
empty (the default) to allow all. It gates both transports — the WebSocket
handshake and the SSE stream. This matters most when the hub authenticates from
an ambient credential such as a cookie, where the browser attaches it
automatically: without an Origin check a page on any origin could open a
connection with the victim’s credentials (the realtime equivalent of CSRF).
The two transports treat a missing Origin header differently, and the
difference is dictated by the browser:
- A WebSocket handshake always carries
Origin(RFC 6455), so a missing one means the caller is not a browser and is refused whenOriginsis set. If you connect from a non-browser WebSocket client — a mobile app, a backend relay — either send an allowedOriginyourself or leaveOriginsempty. - A same-origin
EventSourcesends noOriginheader at all (the Fetch standard adds it only to CORS and WebSocket requests), so a missing one is allowed — otherwise every ordinary same-origin SSE client would break the moment you configureOrigins. A present but unlisted origin is refused, which is what a cross-origin browser always sends.
Origin is one layer, not the whole story. Because SSE is an ordinary CORS
request, a cross-origin page also cannot read the stream unless your app’s own
CORS configuration permits it; the Origins check additionally stops the
connection being established at all.
Per-event authorisation
AllowPatterns controls which topics a client may subscribe to; Visibility
controls which individual events it actually receives. The hook runs once per
(event, client) pair and can also redact the payload:
HubConfig{
Visibility: func(p *realtime.Principal, e events.Event) (bool, *events.Event) {
if e.TenantID != p.TenantID {
return false, nil // suppress cross-tenant events
}
return true, nil
},
}
Return (true, ©) to deliver a transformed event — the hub clones before
mutation so each client sees its own view.
What the payload contains
The hub forwards an event’s Data verbatim — it does no field redaction of
its own. For events produced by the framework, that is safe by construction:
events.Emit runs each record through maniflex.RedactRecord before it reaches
the bus, stripping hidden, write-only and encrypted columns (and the _hmac
companion of an encrypted-unique column), so a subscriber never sees the
plaintext of a secret field. The same redacted bytes are what the ResumeStore
buffers, so a replay carries no more than the live delivery did.
Two consequences worth stating plainly:
- If you publish your own events (calling
bus.Publishdirectly, or building anevents.Eventby hand), nothing redacts that payload for you. Run the record throughmaniflex.RedactRecord(model, record)before attaching it, exactly asevents.Emitdoes. Visibilityis authorisation, not secret-scrubbing. Use it to decide who may see an event (tenant isolation, per-role suppression), not to strip fields that should never be on the wire — those are already gone by the time the hook runs, and relying on an opt-in hook to remove secrets means a hub configured without one would leak them.
Heartbeat
Idle connections are kept alive automatically so L7 proxies (ALB, NGINX, with their typical 30–60s idle timeouts) don’t drop them:
- WebSocket — the server sends a ping frame every
PingInterval(default 30s); compliant clients answer with a pong. - SSE — the server emits a
: keepalivecomment on the same interval.
Disconnects
A WebSocket connection is served by a read pump and a write pump, and whichever
one first notices the peer is gone closes the socket and stops the other. That
holds for a clean close frame, an abrupt drop (EOF/RST), and a half-close —
a client that shuts down its write side but keeps reading. The half-close is
worth naming because nothing about it fails on its own: every ping the server
writes still succeeds, so before v0.3.3 such a connection was never reaped and
its goroutine and CLOSE_WAIT socket were held for the life of the process.
Either way Hub.Stats().Connections drops as soon as the peer goes away.
A peer that stops answering without closing — the half-open connection a
network partition leaves behind — produces no error at all, so it is caught by
a deadline instead. Every WebSocket connection must send something within
ReadTimeout (default 2×PingInterval, so 60s) or it is closed with
1001 Going Away. Any inbound frame refreshes it, including the pong a
compliant client returns for the server’s ping, so a connection that is merely
idle is kept alive by the heartbeat alone and never needs application traffic.
This is the one rule that can disconnect a client which is genuinely there: a
hand-rolled client that ignores ping frames now has a connection lifetime of
ReadTimeout rather than forever. Answering pings is the fix — RFC 6455
requires it, and every mainstream client library does it for you. Where that
isn’t possible, set ReadTimeout: realtime.ReadTimeoutDisabled, understanding
that half-open connections then accumulate undetected.
Resumable streams (lastEventId)
By default delivery is ephemeral: a client that disconnects misses whatever was published while it was away. Enable resume to give clients a replay buffer.
hub, _ := realtime.NewHub(realtime.HubConfig{
Bus: bus,
ResumeBuffer: 1024, // retain the most recent 1024 events for replay
})
With resume enabled, every delivered event carries a cursor:
- SSE — the cursor is the standard
id:line. On reconnect the browser’sEventSourceautomatically sendsLast-Event-ID, and the hub replays everything after it before resuming the live stream. (You can also pass?lastEventId=<cursor>explicitly.) - WebSocket — events include a
"cursor"field; resume by adding it to your subscribe message:{"op":"subscribe","patterns":["invoice.*"],"after":"<cursor>"}.
If the cursor is older than the retained buffer (or the hub restarted), the
client receives a resync signal — event: resync on SSE, {"op":"resync"}
on WebSocket — telling it to refetch current state instead of silently missing
events. Across the reconnect seam delivery is at-least-once; because cursors are
monotonic, clients drop anything at or below their last applied cursor.
ResumeBuffer installs an in-process ring buffer, so resume works when the
client reconnects to the same replica (WebSocket affinity). For
cross-replica resume, supply your own ResumeStore (e.g. backed by a Redis
stream) via HubConfig.ResumeStore.
Schema-emitting events (AsyncAPI)
Just as /openapi.json lets clients codegen typed REST clients, the hub’s event
catalogue can be published as an AsyncAPI 2.6 document so clients codegen
typed event payloads. Declare it once:
server.RealtimeDoc(maniflex.AsyncAPIConfig{
Title: "Billing events",
Servers: []maniflex.AsyncAPIServerConfig{
{Name: "ws", URL: "ws://localhost:8080/ws", Protocol: "ws"},
},
// Derive invoice.created|updated|deleted channels from registered models:
AutoModelEvents: true,
// …and/or declare custom events with a Go struct payload:
Events: []maniflex.EventDoc{
{Type: "payment.received", Title: "Payment received", Payload: PaymentReceived{}},
},
})
This makes an AsyncAPI document available for mounting. Generated documentation
is private-by-default, so also set Config.Documentation.Public explicitly or
provide Config.Documentation.Middleware; the same policy protects OpenAPI and
AsyncAPI. The payload struct is reflected with the same json + mfx tags as
models and actions (Actions). Apps that never call RealtimeDoc
have no AsyncAPI document even when documentation is enabled.
Backpressure & slow clients
Each connection has a bounded outbound queue (SendBuffer, default 64). A
client that fills it is kicked at once — a WebSocket close
1013 Try Again Later, or an SSE disconnect that triggers EventSource
reconnection. Hub.Stats() exposes the live connection count and cumulative
kick count (counted per kicked client, not per dropped event) for monitoring. A
frame larger than MaxMessageSize (default 64 KiB) is rejected with close
1009.
SendBuffer is the only knob that matters here, because fan-out never
waits. Every client is served from one shared goroutine, so a wait on one is a
wait imposed on all of them: before v0.3.3 the hub paused for up to
SendTimeout (5s) on a client whose buffer was full, during which no other
client received anything, and the events piling up behind it could fill the
bus’s own queue — at which point Publish began refusing events process-wide,
for every subscriber rather than just the hub. SendTimeout is now ignored and
deprecated; raise SendBuffer if your clients need more slack.
Dropping the client rather than the event is deliberate. A kicked client
reconnects and, with a ResumeStore
configured, replays from its cursor — so nothing is lost. A client kept
connected while its events were discarded would have a gap it could never learn
about, which is the worse failure.
Connection & subscription limits
Both limits are unbounded by default. Set them once a hub is exposed to untrusted clients — each connection costs goroutines, a socket, and buffers, and each subscription adds to the per-event fan-out cost.
MaxConnectionscaps the live connection count — the same numberStats().Connectionsreports, so it bounds WebSocket and SSE together. Once full, a new WebSocket upgrade or SSE request is refused with503 Service Unavailablebefore any connection resources are committed; the slot is returned when a connection closes.MaxSubscriptionsPerConncaps how many subscriptions one WebSocket connection may hold at a time. Asubscribepast the cap is answered with aTOO_MANY_SUBSCRIPTIONSerror and the connection stays open; anunsubscribefrees a slot. This bounds a single client’s per-event work, which grows with its subscription count. It has no SSE equivalent — an SSE client subscribes once, at connect, so its fan-out cost is fixed by the patterns in the connecting URL.
Scaling out
The hub is single-process by design; cross-replica fan-out is the bus’s job:
- inproc (single binary) — one hub, all clients local.
- redis / nats / kafka — every replica subscribes to the bus, so an event published anywhere reaches local clients on every replica. Pair with a sticky load balancer so each client stays on one replica (WebSocket affinity).
The hub does not create a consumer group per connection — per-client filtering happens server-side, downstream of one shared bus subscription, so broker load doesn’t scale with connection count.
Graceful shutdown
Hub.Shutdown(ctx) stops accepting connections, signals every client to close
(a 1001 Going Away frame to WebSocket clients), and waits for every connection
goroutine — both WebSocket pumps and SSE handlers — to drain, until ctx
expires. It then cancels the bus subscription.
The hub is mounted by your code, so it is not part of server.Shutdown by
itself. Rather than calling it from your own signal handler, hand it to
Server.AddService and let the server’s lifecycle own it:
// hubService folds Hub.Shutdown into the server's own drain.
type hubService struct{ hub *realtime.Hub }
// Start is a no-op: the hub is already serving by the time it is registered.
func (s hubService) Start(context.Context) error { return nil }
func (s hubService) Stop(ctx context.Context) error { return s.hub.Shutdown(ctx) }
Then register it before starting the server:
server.AddService(hubService{hub})
Services stop in reverse registration order after the HTTP listener has drained,
and Stop receives what remains of the one shutdown budget — so the hub’s drain
is bounded by the same deadline as everything else, instead of racing it from a
separate goroutine.
Do not use
maniflex.ServiceFuncfor this. ItsStopis a deliberate no-op — it adapts a function that winds itself down onctxcancellation — so the hub would never be told to shut down, and connections would be cut by process exit rather than a1001 Going Away. The hub needs aStop, so it needs the full interface.
AddService must be called before Start.
Hub shutdown
AddService alone isn’t enough and can cause a deadlock during shutdow.
An active WebSocket connection is treated as an in-flight HTTP request and
http.Server.Shutdown waits for all in-flight requests to finish without
canceling them - it blocks Hub.Shutdown from running via Service.Stop.
This creates a deadlock: the server drains connections that only Hub.Shutdown
can close. As a result, a single active client can stall the shutdown until
ShutdownTimeout expires, causing Service.Stop, OnShutdown, and the
background-write drain to time out.
Hand the hub Server.ShuttingDown() so it hears about shutdown before the wait
begins:
hub, err := realtime.NewHub(realtime.HubConfig{
Bus: bus,
ShuttingDown: server.ShuttingDown(), // closed before the drain starts
})
Every connection is then told to close as shutdown begins, the handlers return,
and the HTTP drain finishes in milliseconds. Keep the hubService registration:
the signal only tells connections to go, while Hub.Shutdown in Stop is what
waits for them to actually be gone — and it now has a budget to wait with.
Server.ShuttingDown() is a plain <-chan struct{}, closed once. Any streaming
handler you write yourself should select on it too; see
Graceful Shutdown.
Every SSE write carries a bounded deadline, so a client that has stopped reading
cannot pin its handler goroutine — and therefore cannot hold Shutdown open —
past that deadline. This applies to the live stream, the keepalive comment, and
the lastEventId replay backlog alike.
Observability
Set Logger (a *slog.Logger; defaults to slog.Default()) to surface the
events an operator needs. The hub logs the signals that matter and stays quiet
on healthy traffic:
WARN— a slow consumer dropped (its buffer filled), a connection refused because the hub is atMaxConnectionsor itsOriginisn’t allowed, a malformed frame closed as a protocol error, and aShutdownthat timed out before draining. The two refusal cases are throttled — the first, then every 128th, with a running count — so a flood or a scan can’t drown the log.ERROR— a panic recovered while delivering an event, which in practice means a panickingVisibilityhook (it runs inline in the fan-out). The hub recovers per client, so one bad hook is logged and skipped rather than taking down delivery for every other client.
Ordinary disconnect churn — a dead peer reaped on the read deadline, an auth failure, a client that simply went away — is not logged; it’s expected and would only add noise. Log lines carry connection metadata only — transport, remote address, user id, close reason — and never an event payload, matching the redaction rule above.
HubConfig reference
| Field | Default | Purpose |
|---|---|---|
Bus | — (required) | the events.Bus the hub consumes |
Authenticator | AnonymousOnly{} | connection auth |
Visibility | allow-all | per-event authorisation / redaction |
AllowPatterns | allow-all | subscribable topic whitelist |
ResumeStore | nil (disabled) | replay buffer for lastEventId resume |
ResumeBuffer | 0 (disabled) | shortcut: install an in-memory store of this size |
PingInterval | 30s | WS ping / SSE keepalive cadence |
ReadTimeout | 2×PingInterval | WS dead-peer deadline; ReadTimeoutDisabled to opt out |
SendBuffer | 64 | per-client outbound queue depth |
SendTimeout | — | deprecated, ignored; fan-out no longer waits |
MaxMessageSize | 64 KiB | inbound frame size limit |
MaxConnections | 0 (unlimited) | shared WS+SSE connection cap; over it → 503 |
MaxSubscriptionsPerConn | 0 (unlimited) | per-WS subscription cap; over it → TOO_MANY_SUBSCRIPTIONS |
Origins | allow-all | allowed Origin values for both WS and SSE |
Encryption at Rest
A field tagged mfx:"encrypted" is automatically encrypted before it
reaches the database and decrypted on read. The plaintext never appears in
the table; the column stores a self-describing envelope. This page covers
the full subsystem: the tag, the key provider interface, the storage
format, unique-constraint handling, and key rotation.
Declaring an encrypted field
type Patient struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required,filterable,sortable"`
SSN string `json:"ssn" mfx:"encrypted,key:patient-pii"`
}
| Sub-option | Effect |
|---|---|
encrypted | mark the field for envelope encryption |
key:NAME | the key identifier passed to the KeyProvider; defaults to "default" |
The column’s Go and DB types remain string. Storage is the prefix
enc: followed by a base64-encoded binary envelope:
enc:Aa1z... (envelope bytes embed the keyID)
The enc: prefix lets the framework distinguish ciphertext from any
legacy plaintext that may exist in the column — useful for incremental
migration of an existing table.
What encryption costs you
The trade-offs are deliberate and worth being explicit about:
- No filtering. A
WHERE ssn = ?would have to match an envelope that includes a random nonce. Encrypted fields cannot befilterable. - No sorting. Same reason. Encrypted fields cannot be
sortable. - Uniqueness via HMAC. A
mfx:"encrypted,unique"field gets a companion{field}_hmacTEXT UNIQUEcolumn. See the next section. - The KeyProvider is required. Reads degrade to returning the raw
stored ciphertext; writes are rejected with
500 ENCRYPTION_NOT_CONFIGUREDuntil a provider is configured.
For columns that need to be queryable but contain sensitive data, store a non-sensitive lookup key (a hashed identifier) in a separate field and encrypt only the payload.
Access paths
Encryption is applied on every access path, not just HTTP:
-
HTTP pipeline — automatic.
-
Typed helpers (
maniflex.Create/Read/Update/List) andctx.GetModel(name)— encrypt on write and decrypt on read using theServerContext’s KeyProvider (set automatically inside a request). -
Background workers / CLIs using
maniflex.NewBackground(...)— callbg.SetKeyProvider(srv.KeyProvider())so typed access to encrypted models encrypts and decrypts:bg := maniflex.NewBackground(ctx, srv.DB(), srv.Registry()) bg.SetKeyProvider(srv.KeyProvider()) p, _ := maniflex.Read[Patient](bg, id) // decrypted
The raw adapter (srv.DB().Create/FindMany) is deliberately encryption-agnostic
— it stores and returns ciphertext verbatim, which is what RotateEncryptionKey
relies on. Use the typed helpers or ctx.GetModel for transparent crypto.
Configuring a KeyProvider
maniflex.Config.KeyProvider must be set before any model with encrypted
fields is exercised. Two implementations ship in pkg/encryption, both
constructed as plain struct literals.
EnvKeyProvider — keys from environment variables
import "github.com/xaleel/maniflex/pkg/encryption"
server := maniflex.New(maniflex.Config{
KeyProvider: &encryption.EnvKeyProvider{
Prefix: "MYAPP_KEY",
IndexKeyID: "blind-index",
},
// ...
})
The env var name for a given keyID is derived as
{Prefix}_{KEYID_UPPER}, with hyphens replaced by underscores and the
result uppercased:
Prefix | keyID | Env var read |
|---|---|---|
MYAPP_KEY | default | MYAPP_KEY_DEFAULT |
MYAPP_KEY | patient-pii | MYAPP_KEY_PATIENT_PII |
MYAPP_KEY | blind-index | MYAPP_KEY_BLIND_INDEX |
MFX_KEY (default) | billing | MFX_KEY_BILLING |
Each variable holds a base64-encoded 32-byte (256-bit) AES key. Generate one with:
openssl rand -base64 32
The provider accepts either standard or URL-safe base64.
VaultKeyProvider — HashiCorp Vault Transit
server := maniflex.New(maniflex.Config{
KeyProvider: &encryption.VaultKeyProvider{
Address: "https://vault.example.com",
Token: os.Getenv("VAULT_TOKEN"),
Mount: "transit", // optional, default "transit"
IndexKeyID: "blind-index", // dedicated Transit HMAC key
// TokenSource: tokenSource, // optional, refreshed per request
// Client: customHTTPClient, // optional, cloned before use
},
})
keyID maps to a Vault Transit key name; the plaintext is sent to
/v1/{mount}/encrypt/{keyID} and Vault returns its own
vault:v1:... ciphertext, which the provider embeds in the envelope.
A Vault key rotation is transparent — Vault decrypts ciphertexts
encrypted with any prior version of the key automatically.
Vault addresses must use HTTPS. AllowInsecureHTTP is an explicit opt-out for
isolated development or test environments and must not be enabled in production.
The provider uses a private HTTP client with a 10-second whole-operation timeout by
default. An earlier deadline on the request context wins. Set Timeout explicitly
to tune the bound; a negative value disables the client timeout and leaves the
operation bounded only by its context.
For renewable AppRole, Kubernetes, JWT, or other short-lived authentication,
implement VaultTokenSource (or use VaultTokenSourceFunc). It is called before
every Vault request and takes precedence over the static Token.
A custom backend implements the maniflex.KeyProvider interface:
type KeyProvider interface {
Encrypt(ctx context.Context, keyID string, plaintext []byte) ([]byte, error)
Decrypt(ctx context.Context, envelope []byte) ([]byte, error)
KeyIDOf(envelope []byte) (string, error)
HMAC(ctx context.Context, keyID string, data []byte) ([]byte, error)
}
type BlindIndexKeyProvider interface {
BlindIndexKeyID() string
}
Encrypt returns a self-describing binary envelope that embeds the
keyID. Decrypt reads the keyID from the envelope, so callers don’t
supply it. HMAC produces a deterministic keyed digest used for unique
indexes. BlindIndexKeyProvider is an optional capability. The shipped
providers implement it through IndexKeyID; custom providers should implement
it when encrypted+unique models need online key rotation.
Unique encrypted fields
A normal UNIQUE constraint on an envelope is useless — each envelope
contains a random nonce, so two encryptions of the same plaintext are
different ciphertexts. The framework solves this with an HMAC companion
column.
Email string `json:"email" mfx:"encrypted,unique"`
AutoMigrate emits two columns:
| Column | Type | Purpose |
|---|---|---|
email | TEXT | the enc:<base64> envelope |
email_hmac | TEXT UNIQUE | a keyed HMAC of the plaintext |
On every write, the DB step calls KeyProvider.HMAC with IndexKeyID and
stores the result in the companion. The index key must be dedicated to this
purpose and must not rotate with field-encryption keys. The HMAC is
deterministic for a given (index key, plaintext) pair, so the database can
enforce uniqueness without ever seeing the plaintext.
If IndexKeyID is omitted, writes retain the legacy behavior of using the
field-encryption key for compatibility, but online rotation of a model with
encrypted+unique fields is refused. Configure the index key before the first
such row is written. For an existing table, quiesce writes and backfill the
companion digests under the dedicated key before attempting online rotation;
the rotation preflight rejects any legacy digest.
Setting it also separates key material between the two algorithms. On the
fallback the same 32 bytes are the AES-GCM key and the HMAC-SHA256 key; with
IndexKeyID set they resolve to different env vars — or different Transit keys
— holding different secrets. There is no known attack that combines the two, so
this is hygiene rather than an exposure, but it is hygiene that costs one
environment variable and cannot be retrofitted: changing the index key changes
every digest it has already produced.
Because the bill arrives late — writes succeed, and the refusal comes at the first rotation, once the table is full of digests nobody can re-derive — the framework says so at boot:
WARN encrypted unique fields are indexed under the field encryption key because
the KeyProvider names no blind-index key; RotateEncryptionKey will refuse
this model, and existing digests cannot be re-derived
model=Patient fields=ssn hint="set IndexKeyID on the KeyProvider before writing data"
Config.Strict promotes that warning to a startup error, so a production
deployment cannot reach its first write on the fallback by accident.
Reads strip the HMAC column from responses automatically; clients see only
the decrypted plaintext on email and never the digest.
When the unique check fires, the adapter returns *maniflex.ErrConstraint and
the DB step converts it to 409 CONFLICT — same path as any other unique
violation.
If the index cannot be created
The UNIQUE index over the HMAC column is what actually enforces the
constraint. If AutoMigrate cannot create it, the migration fails rather
than starting a server whose mfx:"unique" is claiming a guarantee the database
is not enforcing:
AutoMigrate: could not create unique index uidx_users_email_hmac on
users(email_hmac): UNIQUE constraint failed — the model declares these
column(s) unique, so the server will not start without the constraint. The
usual cause is rows that already violate it; de-duplicate them and start again
The usual cause is exactly that: rows already holding duplicate values in the field. De-duplicate them and start again. A plain (non-unique) index that fails to build is only warned about — a missing one costs a table scan, not a constraint.
Per-domain keys
The key:NAME sub-option routes a field to a specific key identifier:
type Record struct {
maniflex.BaseModel
PaymentToken string `json:"payment_token" mfx:"encrypted,key:billing"`
MedicalNote string `json:"medical_note" mfx:"encrypted,key:medical"`
}
A KeyProvider that backs different keys with different secrets (or a Vault transit mount) lets you scope access by domain — the billing team holds the billing key; medical staff hold the medical key; the application process holds both. Rotating one does not affect the other.
When key: is omitted, the framework uses the keyID "default". Either
configure a key under that name or always tag with an explicit key.
Decryption on the read path
The DB step runs the decryption pass after every read:
- For list and read operations,
decryptFieldsreplaces everyenc:<base64>value with the decrypted plaintext. - HMAC companion columns are always stripped from the response.
- Values that do not have the
enc:prefix are left as-is — important for gradual adoption: enable encryption on a column whose existing rows are plaintext, and only new writes get encrypted. - Rows pulled in by
?include=are decrypted too. A relation’s row reaches the serializer straight from the adapter rather than through the DB step’s own pass, so before v0.2.5 an encrypted field on an included model came back as base64 ciphertext while the same model read directly returned plaintext. If a relation cannot be decrypted the field is left as stored and a warning is logged, rather than failing the whole request over one included row.
If KeyProvider is nil but a model has encrypted fields, reads return
the raw stored ciphertext (so the application still functions in some
read-only sense), but writes are refused. Configuring a provider is the
only way to write encrypted columns.
Key rotation
First update the model’s key:NAME tag (and deploy it) so concurrent writes use
the new encryption key. Then
maniflex.RotateEncryptionKeyWithOptions(ctx, server, modelName, oldKeyID, newKeyID, options) re-encrypts every old-key envelope:
report, err := maniflex.RotateEncryptionKeyWithOptions(
ctx, server, "Patient", "v1", "v2",
maniflex.EncryptionRotationOptions{PageSize: 100},
)
if err != nil {
log.Printf("stopped after %q; row failures: %+v", report.LastID, report.Failures)
log.Fatal(err)
}
log.Printf("re-encrypted %d rows", report.Rotated)
The compatibility wrapper RotateEncryptionKey still returns (int, error).
The detailed API adds Failures, LastID, and Complete.
Rotation resolves the model’s per-model adapter, then preflights the requested range before writing. Malformed base64, invalid envelopes, decrypt failures, and mismatched legacy blind indexes are returned with their row ID, field, and stage. It never reports success after silently skipping corruption.
The operation is not atomic across all rows. On an interruption or adapter
error, resume with AfterID: report.LastID; already-rotated envelopes are
idempotently skipped. If report.Failures is non-empty, repair those rows and
restart from the beginning so the full remaining old-key set is preflighted.
Both encryption keys and the stable index key must remain available until
report.Complete is true.
For large tables, run the rotation as a background job rather than at
startup. Each row is a separate UPDATE, so the operation is bound by
the database’s write throughput.
What an envelope looks like
The exact envelope format is the KeyProvider’s concern. The two
shipped providers use slightly different layouts, but both put a
self-describing header in front of the ciphertext so KeyIDOf can
extract the keyID without decrypting.
EnvKeyProvider — AES-256-GCM with an inline nonce:
[ version:1 ][ keyIDLen:2 (BE) ][ keyID:N ][ nonce:12 ][ gcmCiphertext+tag:M ]
New envelopes use version 0x02, which binds the version byte and keyID into
the GCM tag as additional authenticated data (AAD), so neither can be altered
without failing decryption. Legacy 0x01 envelopes — sealed without that
binding — still decrypt, and a key rotation re-writes them as 0x02.
VaultKeyProvider — Vault returns a vault:v1:... ciphertext that
embeds its own versioning, so the envelope carries no nonce:
[ version:1 ][ keyIDLen:2 (BE) ][ keyID:N ][ vaultCiphertext:M ]
Vault envelopes use version 0x01 (Vault Transit authenticates internally).
The keyID length is a 16-bit big-endian integer, and the framework stores the
binary envelope as the string enc:<base64> in the column.
Encrypt produces the blob; Decrypt parses the header to recover the
keyID, then routes to the right key. KeyIDOf reads the keyID without
decrypting — useful for audit logging and for the rotation loop above.
A custom provider need not follow either format; the framework only
cares that Encrypt and Decrypt are inverses and that KeyIDOf works
on the output of Encrypt.
Compatibility with other features
| Feature | Interaction |
|---|---|
mfx:"encrypted" + unique | HMAC companion column; standard unique violation as 409 |
mfx:"encrypted" + filterable / sortable | not allowed — a query that filters an encrypted field is rejected at query time with ENCRYPTED_FIELD_NOT_FILTERABLE; the tag itself is not stripped |
mfx:"encrypted" + soft-delete | independent — soft-delete operates on a separate marker column |
mfx:"encrypted" + versioning | encrypted fields are excluded from both the diff and the snapshot, along with their _hmac companions. The history row is built from the decrypted record, so anything left in it would be stored as plaintext |
mfx:"encrypted" + audit log | the audit Changes diff does not exclude encrypted fields automatically; use WithExcludeFields to keep them out |
mfx:"encrypted" + relations | a relation FK is never encrypted; relation joins remain unaffected. Encrypted fields on an included relation are decrypted like any other read |
mfx:"encrypted" + export | the export writes decrypted plaintext — it is a portable file of the data, not of the ciphertext. Exclude the column with response.RedactField where that is not wanted |
Operational checklist
- Set
Config.KeyProviderbefore any encrypted-field model is registered. - Back keys with a secret store (env vars from a vault, HashiCorp Vault Transit, AWS KMS). Never commit a key to source control.
- For staged rollout, deploy the schema (
email_hmaccolumn) before the application change that starts encrypting — and the application change before the migration that backfills existing rows. - Keep both keys active throughout a rotation; remove the old key only
after
RotateEncryptionKeyhas reported every row migrated. - Treat
KeyIDOf(envelope)as the source of truth for “which key encrypted this row” — useful for auditing the rotation.
Versioning & History
A model marked Versioned keeps an immutable history of every write to it.
The framework creates a sibling {model}_history table at migration time
and appends one row per Create / Update / Delete. History rows are
queryable through the same REST surface as any other model, with one
restriction: they are read-only.
Opting in
Set Versioned: true in ModelConfig:
server.MustRegister(
Invoice{}, maniflex.ModelConfig{
Versioned: true,
},
)
Equivalent declaration on the embedded BaseModel:
type Invoice struct {
maniflex.BaseModel `mfx:"versioned"`
Number string `json:"number" mfx:"required,unique"`
Amount float64 `json:"amount" mfx:"required,min:0"`
}
Either form triggers two effects at registration:
- A synthetic
InvoiceHistorymodel is added to the registry — same as any other model, but read-only. - Three DB middlewares are attached to
Invoice: a pre-image capture beforeOpUpdate/OpDelete, and an After-DB writer for every write that succeeded.
The history table
The sibling table has a fixed schema, regardless of the source model’s columns:
| Column | Type | Notes |
|---|---|---|
id | TEXT | UUID, primary key of the history row itself |
record_id | TEXT | id of the source row this entry describes |
version | INTEGER | 1-based, monotonic per record_id |
operation | TEXT | "create", "update", or "delete" |
actor_id | TEXT | ctx.Auth.UserID at the time of the write; nullable |
timestamp | TIMESTAMP | UTC, set by the framework |
request_id | TEXT | the X-Request-Id of the producing request |
diff | TEXT | JSON {field: {old, new}} map |
snapshot | TEXT | full row state as JSON — omitted when VersionedDiffOnly is set |
AutoMigrate also adds a unique index uidx_{table}_history_record_version
on (record_id, version DESC) for the standard “list history for one row”
query. The uniqueness guards against two concurrent writes computing the same
(record_id, version); the writer retries on the constraint violation.
What gets diffed
diff records every changed scalar field. The format is:
{
"amount": {"old": 99.0, "new": 105.0},
"status": {"old": "draft", "new": "sent"}
}
OpCreate— every field with a non-nil value is recorded as{"old": null, "new": value}; nil-valued fields are skipped.OpUpdate— only fields whose value differs between pre-image and post-image appear.OpDelete— every field is recorded as{"old": value, "new": null}.
Excluded by default:
- The primary key (
id) — from the diff only; the snapshot keeps it. hiddenfields.writeonlyfields.encryptedfields and their{field}_hmaccompanions.
This avoids leaking secrets into history while still capturing the
business-meaningful changes. The same exclusions apply to the
snapshot — history rows are built from the decrypted row, so an
encrypted column left in the snapshot would sit in the history table as
plaintext and quietly undo the at-rest guarantee. If you need a value
in history, don’t mark it encrypted, hidden or writeonly.
Snapshot vs. diff-only
By default each history row carries both the diff and the full
snapshot of the row state — convenient for “what did the record look
like on date X?” queries:
# The newest history row for one invoice.
curl 'localhost:8080/api/invoices/abc123/history?limit=1'
For high-write models the snapshot is the largest column by far.
VersionedDiffOnly: true skips the snapshot entirely:
server.MustRegister(
EventLog{}, maniflex.ModelConfig{
Versioned: true,
VersionedDiffOnly: true,
},
)
The trade-off: reconstructing the row state at version N requires walking all entries from version 1 to N and applying their diffs. For an audit trail used by humans (reading recent changes) this is fine; for point-in-time recovery, keep the snapshot.
Reading history
History is read through the record it belongs to:
# One invoice's history, newest first.
curl 'localhost:8080/api/invoices/abc123/history'
# Paginated.
curl 'localhost:8080/api/invoices/abc123/history?page=2&limit=50'
The response is the standard list envelope. Rows come back newest-first
(descending version); page and limit are honoured, and the default
page size is 20.
Why not a flat /invoice_history endpoint?
There used to be one, and it was a security hole (audit MS-4). The synthesized history model mounted the full read surface, but per-model middleware is registered against the parent’s name — an app that wrote:
server.MustRegister(Invoice{}, maniflex.ModelConfig{
Versioned: true,
Middleware: &maniflex.ModelMiddleware{Auth: []maniflex.MiddlewareFunc{requireLogin}},
})
protected GET /invoices and left GET /invoice_history open to
anyone. db.Tenancy("org_id", …) scoped with ForModel("Invoice") had
the same gap, so any caller could read every tenant’s history.
Copying the parent’s middleware onto the history model does not fix it.
The history table has none of the parent’s columns — it holds id,
record_id, version, operation, actor_id, timestamp,
request_id, diff and snapshot — so a tenancy filter on org_id
has nothing to filter. It would look scoped and enforce nothing.
So the history model is now Headless (registered and migrated, no
routes of its own) and reached only through the parent. The request runs
the parent’s read pipeline first: if you cannot read the record, you
cannot read its history, and you get the same 404 the record itself
would give you rather than a 403 that confirms it exists. Every auth,
tenancy and force-filter middleware you already registered applies, with
nothing new to configure.
Middleware scoped with ForOperation(OpRead) does match these requests,
and that is what makes the gate work: it reads the request’s forced filters, so
a tenancy middleware that never ran would leave it with nothing to scope by.
The implication runs one way only — ForOperation(maniflex.OpReadHistory)
means history requests alone.
Deleted records
A soft-deleted record keeps its history, including the delete entry.
The gate uses the adapter’s ScopeChecker capability, which counts
soft-deleted rows as present while still applying your scope — so a
deleted record’s history is visible to exactly the callers who could see
the record, and to nobody else.
A hard-deleted record’s history is not reachable over HTTP. The row
that said who was allowed to read it is gone, and answering from the
history table alone would mean showing it either to everyone or to no
one. The rows remain in {model}_history for an admin query or an
offline audit. If you need history to outlive deletion, use soft-delete
(maniflex.WithDeletedAt).
Cascaded records
A record removed by a parent’s onDelete:cascade — or re-pointed by
onDelete:setNull — records the same history a direct DELETE or PATCH
would: a delete entry for the cascade, an update entry for the null.
VersionedRequired holds there too, so a history write that fails rolls the
parent’s delete back with it.
This is the reason a versioned model’s onDelete edge is never handed to a
database ON DELETE clause. The database would remove the rows and tell
nobody, leaving the audit trail with holes precisely on the bulk destructive
operations an auditor looks at first — so the framework walks those edges
itself. See Relations.
A write that bypasses the pipeline entirely — a raw INSERT, a direct adapter
call — still records nothing, as it always has.
Custom adapters
ScopeChecker is optional. An adapter that does not implement it — a
third-party one written against the DBAdapter interface — keeps
working, and the endpoint stays exactly as scoped: the gate falls
back to an ordinary scoped read, so tenancy and force filters apply as
they always did. The one thing it gives up is the soft-delete case: that
read applies the soft-delete condition, so a soft-deleted record’s
history 404s, the same as a hard-deleted one.
There is no warning at startup, because nothing is misconfigured — you did not ask for a capability and fail to get it. To gain it, implement:
func (a *MyAdapter) ExistsInScope(
ctx context.Context, model *maniflex.ModelMeta, id string,
filters []*maniflex.FilterExpr,
) (bool, error)
Report whether a row with that id exists and satisfies filters,
including when it is soft-deleted. Apply the filters in full —
“deleted” must not become “unscoped” — and return only the boolean, never
the row: a method that returned soft-deleted records would be a general
bypass of the soft-delete condition, and this one is deliberately unable
to serve as one. Implementing it on your Tx type as well lets a
history read inside a transaction see the request’s own uncommitted
writes.
Filtering
Query parameters are parsed against the parent model, so
?filter=operation:eq:update is rejected — operation is not a field
of Invoice. Filtering within a record’s history is not currently
supported; a record’s history is bounded by that record’s edit count,
and pagination covers it. For cross-record queries (“everything alice
changed this week”), use audit logging, which is built for
exactly that question.
The history model is Headless, so it contributes a schema to
/openapi.json but no paths; the /{id}/history route is documented on
the parent.
Transactions and history
The history row is written in the same transaction as the source write — both succeed together or neither does. If the primary insert rolls back, no orphan history entry is left behind.
If the history write itself fails after a successful primary write, the
framework logs the error but does not fail the primary response.
Losing one history row is preferable to refusing a write that the user
already saw succeed. The error is logged via ctx.Logger() so an
operator can investigate.
That default is wrong when history is an audit record rather than a
convenience: the gap is silent, and it surfaces only when someone asks what
changed. Set VersionedRequired to make the failure fail the request:
server.MustRegister(Invoice{}, maniflex.ModelConfig{
Versioned: true,
VersionedRequired: true, // no history row → no write
})
Because the history row is written in the same transaction, returning the error rolls the primary write back with it — the change and its audit entry stand or fall together. Note the exception: with no transaction in force there is nothing to roll back, and the error only reports that history is missing after the fact.
Performance notes
- One additional
INSERTper write to a versioned model. Postgres handles this with a write multiplier of ~2x on the affected tables. - The
snapshotJSON is the dominant cost on row size. UseVersionedDiffOnlyfor verbose tables. - The
record_idindex is essential — every “history for one row” query uses it. Don’t drop it. - For very-high-write models, consider routing history to a separate table partition or a write-optimised store (TimescaleDB, ClickHouse) via a custom DB-After middleware instead of the built-in.
Comparison with audit logging
Audit Logging and Versioning solve different problems:
| Versioning | Audit Logging | |
|---|---|---|
| Storage | sibling DB table | configurable sink (DB, syslog, SIEM, …) |
| Granularity | per-row | per-row, optionally with diff |
| Transactional with the write | yes | yes (Before-DB) |
| Reconstruct prior state | yes — via snapshot or diff replay | no — only the change is recorded |
| Read API | the framework’s list/read on {model}_history | up to the sink |
| Best for | “what did this invoice look like a week ago?” | “who did what, when, across the whole system?” |
The two compose cleanly — turn on versioning for models that need reconstructable history, and audit-log everything for compliance.
Operational checklist
- Enable
Versionedon models whose change history matters for compliance, debugging, or undo. Don’t enable it on every model — the write multiplier adds up. - Choose
VersionedDiffOnly: truefor high-write tables where the diff alone is enough. - Plan storage growth: history is monotonic — older rows never go away unless you delete them out of band. Set up a retention job for very active models.
- Restrict access to the history endpoints with
auth.RequireRole— the diff and snapshot may contain values an end user shouldn’t see.
Scheduled Fields & the Runner
A mfx:"scheduled" tag on a *time.Time field declares a time-driven
transition: when the timestamp falls into the past, the framework applies
a configured action to the row. The mechanism is small but covers a
surprising number of real workflows — auto-publish, auto-archive,
soft-delete after expiry, scheduled status transitions.
This page covers both halves: the tag (declarative, per-model) and the runner (the background goroutine that actually applies transitions).
The tag
mfx:"scheduled" must appear on a *time.Time field (the pointer type
is required so “unset” is distinguishable from the zero time). The tag
takes one action and any number of qualifiers, separated by semicolons:
type Post struct {
maniflex.BaseModel
maniflex.WithDeletedAt
Title string `json:"title"`
Status string `json:"status" mfx:"required,enum:draft|published|archived,default:draft"`
// Auto-publish: set status=published when publish_at falls in the past.
PublishAt *time.Time `json:"publish_at" mfx:"scheduled;field=status;from=draft;to=published"`
// Auto-archive: set status=archived once archive_at falls in the past
// (no from= — applies regardless of current status).
ArchiveAt *time.Time `json:"archive_at" mfx:"scheduled;field=status;to=archived"`
// Auto-soft-delete: requires WithDeletedAt above.
ExpiresAt *time.Time `json:"expires_at" mfx:"scheduled;soft-delete"`
}
Actions
Exactly one action per scheduled field:
| Action | Effect when the timestamp passes |
|---|---|
soft-delete | sets the soft-delete marker — requires maniflex.WithDeletedAt or WithIsDeleted |
hard-delete | physically deletes the row, regardless of soft-delete config |
field=NAME;to=VALUE | sets the named field to the value |
Qualifiers
The field=...;to=... action accepts optional qualifiers:
| Qualifier | Effect |
|---|---|
from=VALUE | apply only when the named field currently equals this value |
to=VALUE | the value to assign (required for field=...) |
from= and to= are validated against the field’s enum (if any) at
registration time — a typo aborts the boot, not the first sweep.
Validation at registration
Every scheduled tag is resolved when ScanModel runs. Configurations
that don’t make sense are reported and the field is dropped from the
runner’s scope:
- Field type must be
*time.Time. - Exactly one of
soft-delete,hard-delete,field=is required. soft-deleterequires the model to be soft-deletable.field=requires ato=and references an existing column.from=/to=must be members of the target field’senum, if it has one.
A scheduled column automatically gets an IndexSpec added to the model
so the runner can locate due rows without a full scan.
The runner
The runner lives in maniflex/scheduled (its own satellite-style package).
It is opt-in — declaring scheduled tags makes the rows ready to be acted
on, but nothing happens until a runner is started.
import "github.com/xaleel/maniflex/scheduled"
runner, err := scheduled.New(server, scheduled.Config{
Interval: time.Minute,
BatchSize: 500,
})
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
runner.Start(ctx)
defer runner.Stop()
scheduled.New walks the registry, picks up every model that declares a
scheduled field, and binds them to the runner. A registry with no
scheduled fields produces a usable no-op runner — callers can wire it
unconditionally and pay no cost.
Config
| Field | Default | Purpose |
|---|---|---|
Interval | 1m | how often the loop ticks |
BatchSize | 500 | maximum rows processed per (model, spec) per tick; a larger backlog sets Report.Truncated and logs a WARN |
Logger | slog.Default() | structured log sink |
Clock | time.Now().UTC | injectable; tests override |
Locker | nil | leader election: gate each tick so one replica sweeps per interval (see Distributed runners) |
OnDelete | nil | callback func(model, id string) after a delete commits |
OnSetField | nil | callback func(model, id, field, to string) after a set-field commits |
The two hooks fire once per affected row, after the per-model transaction has committed. They run outside the transaction, so a hook panic does not roll back the write. A panicking hook is also recovered and logged (with a stack trace); it does not strand the model’s remaining hooks, abort later models, or kill the background loop — the sweep continues.
What one tick does
For each registered model with scheduled specs, in turn:
- Run a
SELECT id, <column>, <conditional fields> FROM <table> WHERE <column> <= now() AND ...to find rows due for action. Thefrom=qualifier becomes an additionalAND field = 'value'clause. - Open a per-model transaction.
- For each row in the batch, lock it (
SELECT … FOR UPDATE) and re-check the due predicate against the locked row before acting:soft-delete→UPDATE table SET deleted_at = now() WHERE id = ?hard-delete→DELETE FROM table WHERE id = ?(via the adapter’sHardDeleteif available)field=NAME;to=VALUE→UPDATE table SET name = ? WHERE id = ?
- Commit the transaction.
- Fire
OnDelete/OnSetFieldhooks for each row, in order. - Move to the next model.
The due predicate is read in step 1 outside any transaction, so step 3
re-asserts it under the row lock before mutating. If a user moved the
from= field off its guard value, or nulled/pushed the timestamp to
un-schedule the row, in the window between the read and the write, the
row is no longer due and is skipped — the sweep never clobbers that
concurrent edit. On Postgres the lock is FOR UPDATE; on SQLite the
whole transaction is serialized (BEGIN IMMEDIATE).
A row an action can no longer touch — already deleted this tick by a
prior spec on the same row, already soft-deleted, or removed by a
concurrent replica — matches zero rows. That is an idempotent no-op, not
a failure: the row is skipped (counted in Report.Skipped) and the batch
continues. Without this, a same-row hard-delete + set-field would
delete the row, fail the follow-up update, roll the whole batch back, and
re-read the identical rows next tick — starving the model forever.
The per-model transaction means a single genuinely bad row aborts only
that model’s batch, not the whole sweep. Errors are appended to the
tick’s Report.Errors and logged. A panic inside a model’s sweep (from
an adapter, MapToRecord, or a transaction op) is contained the same
way: it is recovered into a Report.Errors entry, the transaction rolls
back, and the remaining models are still swept.
Sweep for one-shot ticks
runner.Sweep(ctx) runs exactly one tick and returns the Report:
report, err := runner.Sweep(ctx)
log.Printf("deleted %d, updated %d across %d models",
report.Deleted, report.Updated, len(report.PerModel))
Useful in tests and for cron-driven deployments where the framework’s
internal ticker is the wrong fit. Sweep blocks until the pass completes.
Each tick processes at most BatchSize rows per (model, spec); a larger
backlog drains over successive ticks. When that happens the tick sets
Report.Truncated and logs a WARN, so a backlog building faster than it
drains is visible rather than silent. Act on it by raising BatchSize,
tightening Interval, or draining out-of-band with Sweep.
Distributed runners
A single runner per cluster is enough for most workloads. Two runners
sweeping the same batch do redundant work but their writes stay
correct: a soft-delete or set-field that the other replica already
applied matches zero rows and is skipped (Report.Skipped), never
double-applied. The one thing that is not idempotent is the hooks —
a set-field transition fires OnSetField on every replica that commits
it, so N replicas mean up to N duplicate events or audit rows.
For single-firing hooks across replicas you have three options:
-
Run
Startin exactly one replica (a leader-elected pod, a sidecar, a separate deployment). Simplest when your platform already elects a leader. -
Pass a
Config.Locker— the in-process analogue of thejobs/cronlocker. It gates each tick behind an atomic claim keyed on the interval, so only one replica sweeps per tick; the others skip. Nil (the default) keeps every replica sweeping. A lock backend outage fails open — a claim error sweeps anyway rather than stalling all scheduled work.runner, _ := scheduled.New(server, scheduled.Config{ Locker: myLocker, // Acquire(ctx, key, ttl) (bool, error) })Any single-winner primitive works:
SET key val NX PX ttlon Redis, anINSERTon a unique column, a Postgres advisory lock. -
Use the
scheduled/jobsxadapter, which bridges the runner to ajobsqueue so the sweep is enqueued as a durable job and dispatched by the worker pool — exactly one worker picks up any given tick:
import (
"time"
"github.com/xaleel/maniflex/jobs"
"github.com/xaleel/maniflex/jobs/cron"
"github.com/xaleel/maniflex/scheduled"
"github.com/xaleel/maniflex/scheduled/jobsx"
)
// Register the sweep handler on the worker, keyed by jobsx.JobType
// ("maniflex.scheduled.sweep").
w, _ := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Handlers: map[string]jobs.Handler{jobsx.JobType: jobsx.JobHandler(runner)},
})
go w.Run(ctx)
// A fixed-interval ticker enqueues one sweep job per minute; exactly one worker
// picks up any given tick.
sched := cron.New(queue, nil)
sched.Add(cron.Entry{Every: time.Minute, Job: jobs.Job{Type: jobsx.JobType}})
sched.Start(ctx)
In this setup the ticker drives the queue, not the runner directly — exactly one worker processes any given tick, even with many app replicas.
Hooks for events and audit
OnDelete and OnSetField are the natural place to emit events for
scheduled transitions, so downstream systems learn that a row’s status
changed even though no HTTP request caused the change:
runner, _ := scheduled.New(server, scheduled.Config{
OnSetField: func(model, id, field, to string) {
data, _ := json.Marshal(map[string]any{
"model": model, "id": id, "field": field, "to": to,
})
_ = bus.Publish(context.Background(), events.Event{
Type: "scheduled-transition",
Data: data, // Event.Data is json.RawMessage
})
},
})
The hook fires outside the database transaction. For at-least-once delivery semantics, write a row to an outbox table from inside the runner’s transaction (via a custom DB middleware on the affected models) rather than relying on the hook.
Interaction with versioning and audit
A scheduled transition is just an UPDATE (or DELETE) issued by the
runner. It flows through the model’s normal middleware:
Versionedmodels get a history row for the transition, withactor_id = NULL(noctx.Authexists in the runner).db.AuditLogrecords the write the same way.
This is intentional — a status change is a status change, regardless of whether a human or the runner triggered it.
Security: scheduled actions run privileged and un-scoped
The runner sweeps through the raw DB adapter, not the request
pipeline. It has no ServerContext and no authenticated principal, so it
bypasses every request-level scope your middleware normally enforces —
tenant partitioning (ctx.Auth.TenantID), owner/row-level filters, and
auth checks. The consequences:
- A scheduled action applies to every tenant’s due rows, not one
tenant’s. A
mfx:"scheduled;soft-delete"on a tenant-partitioned model soft-deletes all tenants’ expired rows globally; aset-fieldflips the field on every matching row across all tenants. - Hooks receive no tenant/owner context.
OnDelete(model, id)andOnSetField(model, id, field, to)get the model name and row id only — not the principal, tenant, or owner the row belongs to.
This is inherent to a background sweep: there is no request, so there is no caller to scope to. Treat the runner and its hooks as privileged.
What still applies: DB-layer middleware wired into the adapter —
Versioned history, db.AuditLog, mfx:"encrypted" — runs as usual,
because it lives below the request pipeline (versioned rows record
actor_id = NULL, as above). Only request-level scoping is skipped.
If a model needs per-tenant scheduling policy, encode it in the row itself — a per-row timestamp and guard are already tenant-local — or gate the action inside the hook (which knows the id and can look the row up under the right scope). Do not rely on the sweep to honour a tenant boundary; it does not see one.
When to use scheduled fields
| Need | Fit |
|---|---|
| Auto-publish at a fixed time | yes |
| Auto-archive / auto-expire | yes |
| Soft-delete on retention deadline | yes |
| Send an email at 9 AM tomorrow | not directly — use a job queue; the runner only mutates rows |
| Run a multi-step workflow at a deadline | not directly — hook into OnSetField to enqueue the workflow |
The runner is deliberately simple: timestamp + row-local change. For side-effecting work outside the database, use it as a trigger and delegate the actual work to a job queue.
Operational checklist
- One runner per cluster, started once, stopped on shutdown.
Startis idempotent — a duplicate call is a safe no-op, never a second loop. - Set
Intervalto the desired granularity —1mis plenty for most workflows; tighten if you have sub-minute deadlines. - Set
BatchSizeto a value the database can absorb in one transaction without blocking writers. 500 is a safe default; for very high-volume tables tune lower so each batch is shorter. - Use
OnDelete/OnSetFieldhooks for observability — emit events, increment metrics, log structured records. - For deployments with multiple app replicas, gate the runner to one
process or use
scheduled/jobsxto dispatch sweeps through your job queue. - Combine with
maniflex.WithDeletedAtfor the soft-delete-on-expiry pattern; the indexeddeleted_at IS NULLpredicate keeps the sweep query cheap as the table grows.
Audit Logging
db.AuditLog from the catalogue records every mutating operation to a
configured sink. Unlike Versioning — which writes to a
sibling table in the same database — audit log records are designed to be
shipped to an external system (a database table, a structured logger, a
SIEM). This page documents the record shape, the sink contract, and the
options that change what is captured.
Registering
The simplest registration captures the operation without per-field diffs:
import "github.com/xaleel/maniflex/middleware/db"
server.Pipeline.DB.Register(
db.AuditLog(mySink),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
mySink implements db.AuditSink:
type AuditSink interface {
Write(ctx context.Context, record AuditRecord) error
}
The audit record is emitted from a background goroutine with a 5-second timeout. Sink errors are logged but never fail the request — audit writes are fire-and-forget, by design. An audit pipeline that can fail the request is a liveness risk; an audit pipeline that occasionally drops a record is recoverable.
The record shape
Every audited write produces one AuditRecord:
type AuditRecord struct {
Timestamp time.Time `json:"timestamp"`
Model string `json:"model"`
Operation maniflex.Operation `json:"operation"`
ResourceID string `json:"resource_id,omitempty"`
Actor string `json:"actor,omitempty"`
TenantID string `json:"tenant_id,omitempty"`
RequestID string `json:"request_id,omitempty"`
TraceID string `json:"trace_id,omitempty"`
ServiceName string `json:"service_name,omitempty"`
Result any `json:"result,omitempty"`
Changes map[string]FieldChange `json:"changes,omitempty"`
}
type FieldChange struct {
From any `json:"from"`
To any `json:"to"`
}
| Field | Source |
|---|---|
Timestamp | UTC at the moment the record is built |
Model | ctx.Model.Name |
Operation | ctx.Operation |
ResourceID | ctx.ResourceID — empty on create until after the write |
Actor | ctx.Auth.UserID (empty for anonymous requests) |
TenantID | ctx.Auth.TenantID |
RequestID | ctx.RequestID (chi’s X-Request-Id) |
TraceID | ctx.TraceID (W3C traceparent) |
ServiceName | Config.ServiceName |
Result | ctx.DBResult — the row state returned by the adapter |
Changes | populated only when WithChanges() is set |
The minimum shape — Timestamp, Model, Operation, Actor,
RequestID — is enough to answer “who did what, when?” for compliance.
Adding Changes answers “what specifically was modified?”
Tracking changes
WithChanges() enables per-field diffs:
server.Pipeline.DB.Register(
db.AuditLog(sink, db.WithChanges()),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
// No AtPosition — defaults to Before.
)
Important: WithChanges() requires the middleware to run at
maniflex.Before (the default position), not maniflex.After. The middleware
needs to read the row state before the DB step writes, so the diff has
both sides.
With WithChanges(), Changes is populated as:
| Operation | Changes map |
|---|---|
| Create | {field: {from: null, to: new_value}} for each field returned by the write |
| Update | {field: {from: old, to: new}} for each changed field |
| Delete | {field: {from: value, to: null}} for each field on the pre-image |
Fields that didn’t change between pre-image and post-image are omitted. Fields excluded from the diff (see below) are also omitted.
The pre-image is read through ctx.ResolveResourceID(), so a scoped Singleton
diffs against the caller’s own row rather than against nothing. When that row has
not been provisioned yet, the update that provisions it is diffed as a create —
{from: null, to: value} — which is what it is.
Excluding fields from the diff
WithExcludeFields("password", "api_key", "session_token") keeps named
fields out of the Changes map:
server.Pipeline.DB.Register(
db.AuditLog(sink, db.WithChanges(), db.WithExcludeFields(
"password", "ssn", "api_token",
)),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Use this for secrets that shouldn’t reach the audit pipeline even in
hashed form. Field names are matched against the DB column name (e.g.
api_token, not apiToken).
hidden, writeonly, and encrypted fields are not excluded
automatically — the diff is built from the raw row the write returns.
WithExcludeFields is the only mechanism that keeps a column out of the
Changes map, so name every sensitive column explicitly, including any
that carry those tags.
Common sinks
The sink interface is small enough to wire to anything that records structured events.
Database table
type DBAuditSink struct{ db *sql.DB }
func (s *DBAuditSink) Write(ctx context.Context, r db.AuditRecord) error {
changes, _ := json.Marshal(r.Changes)
_, err := s.db.ExecContext(ctx, `
INSERT INTO audit_logs
(timestamp, model, operation, resource_id, actor,
tenant_id, request_id, trace_id, service_name, changes)
VALUES (?,?,?,?,?,?,?,?,?,?)`,
r.Timestamp, r.Model, r.Operation, r.ResourceID, r.Actor,
r.TenantID, r.RequestID, r.TraceID, r.ServiceName, string(changes),
)
return err
}
A separate table — or a separate database — keeps audit volume from affecting the operational schema.
Structured logger
type LogSink struct{ log *slog.Logger }
func (s *LogSink) Write(ctx context.Context, r db.AuditRecord) error {
s.log.LogAttrs(ctx, slog.LevelInfo, "audit",
slog.String("model", r.Model),
slog.String("operation", string(r.Operation)),
slog.String("actor", r.Actor),
slog.String("request_id", r.RequestID),
slog.Any("changes", r.Changes),
)
return nil
}
The simplest sink — ships every audit event to the same log aggregator the rest of the app uses. Good for cold-storage compliance archives.
Async queue
For high-volume systems where the sink might back up, push records to a durable queue and process them out of band:
func (s *KafkaSink) Write(ctx context.Context, r db.AuditRecord) error {
b, _ := json.Marshal(r)
return s.producer.Produce(ctx, "audit-events", b)
}
A failed publish is logged but does not fail the request; the queue itself provides retry semantics.
Failure semantics
The middleware:
- Reads the pre-image (when
WithChanges()is set) before the DB step. - Calls
next(). - Checks the result. If
next()returned a non-nil error, the audit record is not written — we don’t audit failed operations. - Checks
ctx.Response. If status is>= 400, again no audit record. - Builds the record from the captured pre-image and
ctx.DBResult. - Spawns a goroutine that calls
sink.Writewith a 5-second background context.
This means:
- A failed write produces no audit entry. The framework’s other observability — request logs, error metrics — covers failed attempts.
- A successful write whose audit sink fails still succeeds. The audit record is lost.
- The audit write outlives the request context. A long-running audit write doesn’t block the HTTP response.
For at-least-once delivery, the sink must be backed by durable storage
(a database, a queue) — the in-process goroutine can be lost if the
process is killed before Write returns.
Audit log + versioning
Both record changes. Choose by where the records live and how they’re read:
| Concern | Audit log | Versioning |
|---|---|---|
| Storage | external sink | same DB, sibling table |
| Per-record reconstruction | no | yes (snapshot) |
| Compliance archive | yes | possible but awkward |
| Forensics across the whole system | yes | no — history is read one record at a time, through GET /:model/{id}/history |
| Cost | sink-dependent | one extra INSERT per write |
In a production system both are common: audit log feeds a SIEM for “who did what across everything”, versioning provides per-record history inside the app.
Operational checklist
- Pick a sink that matches your audit volume: database table for low volume, structured logs for medium, durable queue for high.
- Register at
maniflex.Beforewhen usingWithChanges(), atmaniflex.Afterotherwise. WithExcludeFieldsevery secret column — includingwriteonly,hidden, andencryptedones, which are not redacted from the diff automatically.- Treat the sink as best-effort. Don’t rely on the in-process goroutine for legal-grade audit retention; use a sink whose own storage is durable.
- Index your audit table on
timestamp,actor,model, andresource_id— they are the columns most queries filter by. - Restrict who can read the audit table. The diffs may contain values
the original endpoint hid behind
RedactField.
Idempotency
Idempotency middleware makes POST requests safe to retry over a flaky
network. The first request runs the pipeline normally; the response is
cached keyed on the Idempotency-Key header. Subsequent requests with
the same key and the same body short-circuit the pipeline and replay
the cached response.
The pattern is borrowed from Stripe’s API. This page documents the
shipped implementation in maniflex/middleware/idempotency.
The contract
An Idempotency-Key identifies one logical operation. Sending the
same key twice with the same body means “I’m retrying — give me the
same result as last time, do not run the operation again.” Sending the
same key with a different body means “I am confused about my own
state and you should refuse me.” Sending no key means “do not apply
idempotency to this request.”
Idempotency-Key: e3b0c442-98fc-1c14-9afb-...
The key is opaque to the framework — any string the client chooses. A UUID per logical operation is the conventional choice.
Registering
The middleware lives on the Deserialize step at maniflex.After
position, scoped to the operations that should be retryable:
import (
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/middleware/idempotency"
)
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{
Store: maniflex.NewMemoryCache(),
TTL: 24 * time.Hour,
}),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
Why After on Deserialize: the middleware needs ctx.RawBody to compute
the body hash, and the default Deserialize handler populates that. Running
after the default ensures the body is present.
Config:
| Field | Default | Purpose |
|---|---|---|
Store | required | the cache backend (anything implementing maniflex.CacheStore) |
TTL | 24h | how long a cached response is replayable |
KeyFunc | ctx.Auth.UserID then ctx.Request.RemoteAddr | derives the per-caller scope |
HeaderRequired | false | when true, requests without Idempotency-Key are rejected with 400 |
Locker | in-process singleflight | serialises concurrent first-misses on the same key. Supply a Redis-SETNX implementation of idempotency.Locker for multi-replica deployments; see Concurrent first-misses |
The cache key
The cache key is composed of four parts:
<KeyFunc(ctx)>:<model>:<operation>:<idempotency-key>
KeyFunc(ctx)— the per-caller scope. Defaults to the authenticated user ID, falling back to the remote IP for anonymous requests. Override to use the API token or any other identifier.model:operation— limits a key’s effect to one (model, op) pair. The sameIdempotency-Keycan be reused safely for, say,POST /api/ordersandPOST /api/refunds— they are different cache keys.idempotency-key— the client-supplied value.
The body hash is not part of the key — it is part of the cached
entry and compared on lookup. This is intentional: it lets the
middleware detect “same key, different body” and respond with
422 IDEMPOTENCY_KEY_REUSED.
What gets cached
Only successful responses (2xx). Failed responses are not cached, on purpose — retrying a failed write is the whole point of idempotency. A first attempt that 5xx’d should be re-run on the retry, not replayed.
The cached entry carries:
type Entry struct {
maniflex.APIResponse // StatusCode, Data, Error, Meta
BodyHash string
StoredAt time.Time
}
StatusCode— replayed verbatim.Data,Meta— replayed verbatim.BodyHash— SHA-256 ofctx.RawBody, used to detect body mismatch.
The replayed response carries the header Idempotent-Replayed: true
so the client can tell a replay from a fresh execution.
What happens on each call
| Request | Effect |
|---|---|
First request with Idempotency-Key: K | runs the pipeline; if 2xx, caches the response |
| Repeat with same key, same body | skips the pipeline; replays cached response; adds Idempotent-Replayed: true |
| Repeat with same key, different body | 422 IDEMPOTENCY_KEY_REUSED |
| Repeat with same key after TTL | runs the pipeline as if it were the first time |
Request with no Idempotency-Key | passes through (unless HeaderRequired is true) |
Choosing a store
Two implementations cover the common cases.
maniflex.NewMemoryCache
In-process, per-replica:
idempotency.Config{Store: maniflex.NewMemoryCache(), TTL: time.Hour}
Suitable for single-replica development. In a multi-replica deployment each replica has its own cache — a retry routed to a different replica gets a fresh run, defeating the purpose.
Redis (or any shared store)
There is no built-in Redis cache. maniflex.CacheStore is a three-method
interface you implement against whatever shared store you run:
type CacheStore interface {
Get(ctx context.Context, key string) (any, bool)
Set(ctx context.Context, key string, value any, ttl time.Duration)
Delete(ctx context.Context, key string)
}
The value the idempotency middleware stores is an idempotency.Entry, so a
cross-process backend must serialise it on Set and decode it back into an
idempotency.Entry on Get. A store that returns the raw bytes (or a decoded
map) instead makes the middleware abort with IDEMPOTENCY_CACHE_CORRUPT.
Register the middleware with your implementation exactly as with the in-process
cache:
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{
Store: myRedisCache, // your maniflex.CacheStore implementation
TTL: 24 * time.Hour,
}),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
Any backend that can store a TTL’d key/value (Redis, Memcached, DynamoDB with TTL) works, provided it round-trips the stored value’s type.
Concurrent first-misses
Two requests carrying the same Idempotency-Key and identical bodies that
arrive at exactly the same moment both miss the cache. Without
serialisation, both would run the full pipeline and both would write —
silently breaking the contract that one key represents one logical
operation. The middleware uses a Locker to serialise these first-misses.
The default Locker is in-process (singleflight-style): the second
goroutine blocks on a channel until the first releases, then re-checks
the cache and replays. This handles single-replica deployments correctly
out of the box.
For multi-replica deployments, supply Config.Locker with a backend that
synchronises across processes — typically Redis SETNX with a short TTL:
type Locker interface {
Acquire(ctx context.Context, key string, ttl time.Duration) (acquired bool, release func(), err error)
}
Acquire returns acquired=true to exactly one caller per key per TTL
window. The caller must invoke release once the cache entry is written
(or the work has failed). acquired=false means another caller holds (or
held) the lock — singleflight-style lockers block first, then return
false so the loser can replay from cache; SETNX-style lockers return
immediately and the loser re-checks the cache itself.
A Locker error (e.g. Redis network blip, request context cancelled)
fails open: the middleware runs the pipeline directly, mirroring the
pre-Locker behaviour rather than returning 503. This trades correctness
under partial outages for availability — appropriate for a feature whose
whole purpose is “make retries safe.”
Scoping to specific endpoints
For most APIs, idempotency belongs only on a handful of write endpoints —
payment, order placement, account creation. Scope with ForModel so
unrelated POST requests are unaffected:
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{Store: store}),
maniflex.ForModel("Payment", "Order"),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
The middleware passes through for unscoped requests with no measurable cost.
Requiring the header
For endpoints where retries without a key are dangerous, set
HeaderRequired: true:
idempotency.Middleware(idempotency.Config{
Store: store,
HeaderRequired: true,
})
A scoped registration is the right shape — make the header mandatory on payment but optional on lower-stakes resources:
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{
Store: store, HeaderRequired: true,
}),
maniflex.ForModel("Payment"), maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
A missing header on a covered endpoint returns
400 IDEMPOTENCY_KEY_REQUIRED.
Use with custom actions
Action endpoints run a trimmed pipeline that skips Deserialize, so
pipeline-level idempotency does not apply automatically. To get the
same behaviour for an action, include the middleware in the action’s
Middleware list:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: placeOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
idempotency.Middleware(idempotency.Config{Store: store}),
},
})
An action skips the Deserialize step that populates ctx.RawBody, so here the
middleware reads the body itself (via ctx.EnsureRawBody) and restores it — the
action handler’s ctx.BindJSON then works exactly as it would without
idempotency, in whatever order. Before this was fixed (v0.2.3), the middleware
hashed an empty body in an action, so the same key, different body check
silently never fired and a reused key replayed the first response regardless of
the payload.
Edge cases
- Same key, different body. Returns
422 IDEMPOTENCY_KEY_REUSED. The contract is that one key represents one logical operation; reusing it for a different payload is almost certainly a client bug. - Request currently in flight when retry arrives. The default
in-process
Locker(singleflight-style) holds the second goroutine until the first finishes, at which point it replays from cache — only one pipeline execution runs per process. For multi-replica deployments, supply aConfig.Lockerthat uses RedisSETNXso two replicas don’t both run the pipeline. See Concurrent first-misses below. - Cache eviction before TTL. The retry runs the pipeline again. This is correct behaviour: the cache is a replay mechanism, not a deduplication mechanism. The application’s own uniqueness constraints handle “this thing was already created.”
- Operation that mutates external state. Idempotency caches the
response, not the side effect. A payment that charged a card once on
the first request will return the same
paidresponse on a retry without charging again — because the first request returned with the payment recorded as committed. The action handler is responsible for being idempotent against the external system; the middleware just prevents the framework from issuing duplicate writes.
Operational checklist
- One shared
Storeacross replicas. Don’t useMemoryCachein multi-replica deployments. - TTL longer than your client’s longest retry window. 24h is generous; for mobile clients on flaky networks, 7d is reasonable.
- Scope to the endpoints that benefit. Don’t blanket-apply.
- Pair with
HeaderRequired: trueon payment-like endpoints where client correctness depends on it. - Surface the
Idempotent-Replayedheader in client SDKs so consumers can tell a replay from a fresh execution. - Combine with a uniqueness constraint at the DB level for defence-in-depth. A retry that hits a different replica after cache eviction will run the pipeline; the DB constraint catches the duplicate.
Outbound Integrations: pkg/integration
maniflex/pkg/integration is a small toolkit for the integration patterns
that sit beside the framework: calling third-party HTTP APIs, polling
hardware, and receiving signed webhooks. It’s not a feature of the
framework — it’s three composable types you call from your own code.
Caller — JSON-over-HTTP with retry
import "github.com/xaleel/maniflex/pkg/integration"
var billing = integration.NewCaller("https://api.billing.example.com")
func init() {
billing.Headers = map[string]string{
"Authorization": "Bearer " + secrets.Billing,
}
}
// Inside a handler / job / cron tick:
var resp struct {
InvoiceID string `json:"invoice_id"`
}
err := billing.Post(ctx, "/invoices", map[string]any{
"amount": total,
"patient": id,
}, &resp)
- Always JSON in, JSON out. Pass
out=nilto discard the response body. - Passing
[]byteorstringas the body skips JSON encoding — useful for upstreams that demand a specific wire format. - Retries fire on network errors, HTTP 5xx, and HTTP 429 with a
configurable backoff (
BackoffFn; default jittered exponential, capped at 2s). A validRetry-Afterheader takes precedence, capped at 30s. 4xx (other than 429) is final. - Zero-valued settings are safe:
Timeoutdefaults to 10s,MaxRetryto 3, andMaxResponseBytesto 4 MiB. Set the corresponding value negative to explicitly disable that protection. - Redirects are same-origin by default so static custom headers such as
X-API-Keycannot be forwarded to another host. An injectedHTTPClientcan opt out only by supplying an explicitCheckRedirectpolicy. - Non-2xx final responses surface as
*integration.ErrHTTPStatus. Useerrors.Asto inspectStatusCode, the parsed JSONBody, or the rawRawBody. - A response exceeding
MaxResponseBytesreturnsintegration.ErrResponseTooLargebefore JSON decoding, including whenout=nilor the response is non-2xx. - Always honours the request context — cancel ctx to abort an in-flight retry loop.
Poller — periodic background work
// Own a cancellable context and cancel it from your shutdown hook — there is
// no ShutdownContext() on Server; the poller stops when the context you pass
// is cancelled.
pollCtx, stopPolling := context.WithCancel(context.Background())
defer stopPolling() // or call it from your graceful-shutdown path
p := &integration.Poller{
Interval: 30 * time.Second,
Fn: func(ctx context.Context) error {
return terminal.SyncFingerprints(ctx)
},
}
go p.Start(pollCtx) // dies cleanly when stopPolling() is called
A failed tick is logged and the schedule continues — Poller is for
best-effort background work, not workflows where missing a tick is a bug.
For those, use pkg/jobs. Set RunOnStart: true to fire immediately rather
than waiting one Interval.
WebhookReceiver — HMAC-signed inbound
wh := &integration.WebhookReceiver{
Secret: secrets.PaymentWebhook,
Algorithm: "sha256", // or "sha512"
Logger: logger, // optional; defaults to slog.Default()
TimestampHeaderKey: "X-Webhook-Timestamp",
TimestampTolerance: 5 * time.Minute,
ReplayCheck: claimPaymentEventID, // optional atomic shared-store hook
// Defaults are GitHub-style: X-Hub-Signature-256 + X-Event-Type
}
http.HandleFunc("/hooks/payments", wh.Handler(map[string]integration.WebhookHandler{
"payment.succeeded": handlePaymentSucceeded,
"payment.refunded": handlePaymentRefunded,
}))
With TimestampHeaderKey enabled, the sender signs
<raw timestamp>.<raw body> and the header must be Unix seconds or RFC3339
within TimestampTolerance (default 5 minutes). ReplayCheck runs only after
that signature and timestamp pass. It should extract an authenticated event ID
from the signed body and atomically claim it in shared storage; return
integration.ErrWebhookReplay when it was already claimed. Keep handlers
idempotent as a final defence.
The handler:
- Reads at most
MaxBodyBytes(default 1 MiB) from the request body; an extra byte rejects the request rather than truncating it. - Computes HMAC over the raw body and compares it constant-time to the
value in
HeaderKey(or over timestamp + body when enabled). Commonalgo=hexprefixes are tolerated. - Applies the optional timestamp and replay checks.
- Looks up the handler by the
EventHeaderKeyvalue. - Calls the handler with the raw body so it can decode whatever shape the upstream sends.
Failure modes:
- 400 — body read error
- 401 — missing/mismatching signature or invalid timestamp
- 409 —
ReplayCheckreturnedErrWebhookReplay - 413 — body exceeds
MaxBodyBytes - 404 — no handler registered for that event
- 500 — handler or replay storage returned a non-replay error; the client receives
internal server error, while the original error is written toLogger
WebhookReceiver.Handler panics if Secret is empty or Algorithm is
neither sha256 nor sha512 — both are configuration mistakes worth
catching at startup.
CSV / XLSX Export
Models can opt into an auto-generated export endpoint that streams CSV or XLSX of the same data the standard list endpoint returns, with the same filter and sort query parameters:
server.MustRegister(Invoice{}, maniflex.ModelConfig{
ExportEnabled: true,
MaxExportRows: 50_000, // optional cap; defaults to 100,000
})
This mounts:
GET /invoices/export → CSV (default)
GET /invoices/export?format=xlsx → XLSX
The endpoint reuses the full request pipeline — Auth, tenancy, soft-delete —
and middleware registered on ForOperation(maniflex.OpList) covers the
export too. An export is a list in another format, so whatever decides which
rows a caller may list decides which rows they may export:
server.Pipeline.Auth.Register(
auth.JWTAuth(secret, auth.JWTOptions{}),
maniflex.ForOperation(maniflex.OpList), // also covers OpExport
)
Naming OpExport as well is harmless but redundant. The implication runs one
way only: ForOperation(maniflex.OpExport) means the export alone, which is
what you want for export-specific middleware such as a rate limiter.
Before v0.2.5 this was not so — a middleware scoped to
OpListdid not run for exports, so tenancy written that way scoped the list and let the export return every tenant’s rows.
Query parameters
The same filter, sort, and include parameters work as on GET /invoices.
page and limit are ignored — the export reads every row that matches the
filters, up to MaxExportRows.
GET /invoices/export?filter=status:eq:posted&sort=created_at:desc
Sorting on created_at requires the model to have opted that column in via
ModelConfig.BaseModelTags —
BaseModel’s columns are readonly and nothing more by default.
Column selection
Each model field appears as one column, named by its json tag (the same
identifier you see in API responses). Hidden, writeonly, and file-typed
columns are excluded:
| Field tag | In export? |
|---|---|
| (default) | yes |
hidden | no |
writeonly | no |
file | no (raw storage keys are useless to the recipient) |
A field masked for this caller by a response middleware
(response.RedactField) is
excluded too — column and values both, so the export does not advertise a field
it will not fill. Those tags are the same for every caller; a masking middleware
decides per request, and the export honours both.
Computed fields registered via Server.AddComputedField are not included
yet — they require runtime evaluation per row and the export is read-only at
the storage layer.
Row cap
MaxExportRows (default 100,000) bounds the result. Requests whose filtered
result would exceed the cap return 413 Request Entity Too Large with a
suggestion to tighten the filters; no partial data is written. Increase the
cap if you have a deliberate need; consider an async job (pkg/jobs) for
multi-million-row exports.
Concurrency cap
An export reads its whole result set into memory and holds it until the last
byte has been written to the client. MaxExportRows bounds one export’s row
count, but not how wide a row is, and not how many exports run at once — so
the memory an export costs is rows × width, and the memory the server costs
is that again times however many are in flight. A handful of concurrent exports
of a wide model is enough to exhaust the heap with every one of them
individually inside its cap.
Config.MaxConcurrentExports (default 4) bounds the product:
server := maniflex.New(maniflex.Config{
MaxConcurrentExports: 8, // 0 uses the default of 4; negative disables the limit
})
The limit is server-wide rather than per-model, because the heap it protects is
shared. An export arriving when every slot is taken is rejected immediately
with 503 Service Unavailable, code EXPORT_BUSY, and a Retry-After header —
it is not queued. Queuing would hold the connection open behind work that is
slow by nature, so the caller is told to come back instead.
The slot is taken before the pipeline runs and released when the request returns, so it spans the database read and the write — the whole window the rows are live. Note that this means the refusal happens before Auth: a request that would have failed authentication gets the 503 rather than a 401 while the server is saturated.
Set a negative value to remove the limit if you have admission control in front of the service.
Response shape
| Format | Content-Type | Content-Disposition |
|---|---|---|
| CSV | text/csv; charset=utf-8 | attachment; filename="<model>-<ts>.csv" |
| XLSX | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | attachment; filename="<model>-<ts>.xlsx" |
The XLSX writer produces a minimal-but-valid .xlsx workbook with one sheet
named Data. Strings are written as inline string cells — no shared-string
table, no styles, no formulas — which keeps the writer dependency-free
(stdlib archive/zip and encoding/xml only) at the cost of slightly
larger files than a heavyweight library would produce.
Unsupported ?format= values return 400 INVALID_FORMAT.
What’s not in v1
- Async / job-backed exports for multi-million-row datasets. Today’s
endpoint streams synchronously; the upper bound is
MaxExportRows. - Constant-memory exports. The CSV and XLSX writers hold nothing per row —
they serialise and write one row at a time — but the rows themselves are read
from the database in one go before the write begins, so one export still costs
memory proportional to its result set.
MaxConcurrentExportsbounds how many of those can coexist; a true row-by-row cursor would remove the per-export cost too, at the price of pinning a database connection for the length of the client’s download and turning a mid-stream database error into a silently truncated file. - Localised column headers. Headers always use the JSON field name.
- Style hints (number formats, frozen headers, autofilter). The XLSX is intentionally plain.
Open an issue if any of these matter for your use case.
Auth & Security Hardening
The defaults are safe to deploy, but production APIs benefit from a few extra layers. This page collects the practical checklist.
Authentication
- Use
auth.JWTAuthwith an asymmetric algorithm (RS256/ES256) when tokens are issued by an external provider. SymmetricHS256works when the signing service and the API share infrastructure. - Publish the JWK Set over
https://. It is the only thing deciding which tokens verify, so plaintext hands anyone on the path the ability to mint their own; a non-loopbackhttp://URL warns at startup. SeeJWKSAuth. - Use
auth.JWKSAuth(jwksURL, opts…)when the issuer publishes a rotating JWK Set (/.well-known/jwks.json). It fetches and caches the keys, selects the signing key by the token’skid, and refetches on an unknownkidso key rotation needs no redeploy. AllJWTOptions(Issuer, Audience, claim mappings, ClockSkew) apply. Prefer this over pinning a single staticPublicKeyagainst an issuer that rotates. RSA (RS256/384/512) and EC (ES256/384/512) supported. - Set
JWTOptions.IssuerandAudienceso tokens issued for another audience are rejected. - Set
JWTOptions.TenantClaimfor multi-tenant APIs — the verified value ends up onctx.Auth.TenantIDand feedsdb.Tenancy. - Never accept anonymous writes by default. Register
auth.JWTAuth(orauth.APIKeyAuth) on the Auth step unscoped, and open the genuinely public routes withauth.AllowAnonymous. Scoping the authenticator onto a list of operations or models instead leaves everything it does not name covered by no auth registration at all — andForModel/ForOperationare inclusion-only, so the next model added is on the wrong side of that line with nothing to say so.
server.Pipeline.Auth.Register(auth.AllowAnonymous(),
maniflex.ForOperation(maniflex.OpList, maniflex.OpRead))
server.Pipeline.Auth.Register(auth.JWTAuth(secret, auth.JWTOptions{
Issuer: "https://accounts.example.com",
Audience: "https://api.example.com",
TenantClaim: "org_id",
}))
AllowAnonymous forgives an absent credential only. One that was presented
and failed — expired, wrongly signed, revoked, malformed — is still 401 on an
exempt route, so a caller cannot shed a restrictive role by corrupting a byte of
their own token.
Verifying tokens from an issuer that rotates its signing keys via JWKS:
server.Pipeline.Auth.Register(auth.JWKSAuth(
"https://accounts.example.com/.well-known/jwks.json",
auth.JWTOptions{
Issuer: "https://accounts.example.com",
Audience: "https://api.example.com",
TenantClaim: "org_id",
}))
Authorisation
- Gate sensitive operations with
auth.RequireRole. Don’t rely on the UI to hide them. - Use
db.Tenancyordb.ForceFilterfor row-level scoping. These run on the DB step so they apply to lists, reads, and writes uniformly — UI code cannot accidentally bypass them. - Strip privileged values with
validate.ForbiddenValuesfor role fields and similar — prevent a normal user from promoting themselves by including"role": "admin"in a payload.
Secrets and PII
- Hash passwords with
service.HashField, never store them raw. - Use
writeonlyon credential fields so they are accepted on input but never returned in responses. - Encrypt sensitive columns with
mfx:"encrypted"and a configuredKeyProvider. Pair with thekey:sub-option for per-domain keys. - Redact in responses with
response.RedactFieldwhen a column is visible to some callers and hidden from others. - Gate writes with
validate.FieldRolewhen a column is writable by some callers and not others (readonlyis all-or-nothing). Without it, a privileged field needs its own endpoint to keep it off the general PATCH.
Input
- Set
Config.QueryTimeoutso a slow query can’t tie up a connection indefinitely. - Cap body sizes with
body.MaxBodySizewhere you know the upper bound. The default 4 MB limit catches accidents, but a 10 KB endpoint should enforce 10 KB. - Strip unknown fields with
body.StripUnknownFieldsin environments where you want a strict contract — every accepted field appears on the model. - Validate beyond tags with
validate.RegexField,validate.UniqueField, andvalidate.CrossFieldValidate. The built-inmfx:rules cover the common cases; everything else belongs in middleware.
Output
- Set security headers globally via
response.AddHeader:Strict-Transport-Security,X-Content-Type-Options,Referrer-Policy. - List CORS origins explicitly with
response.CORSHeaders(origins...)— origins are required (there is no permissive wildcard default; it panics if you pass none), and"*"cannot be combined with credentials. Install it inConfig.HTTPMiddlewares, where preflight runs before Auth. - Cap rate-sensitive endpoints with
db.RateLimitso password resets and similar can’t be brute-forced.
Transport
-
Terminate TLS at the load balancer or reverse proxy, not in the maniflex process. The framework is HTTP/1.1 + HTTP/2 ready.
-
Name your proxies in
Config.TrustedProxies, not justTrustProxyHeaders. Proxy-header resolution is off by default: the client IP is the direct TCP peer, so a caller cannot forge it. Every IP-keyed feature —db.RateLimit, idempotency scoping, and read-audit records — depends on that address, so how you turn resolution on matters:Config{TrustedProxies: []string{"10.0.0.0/8"}} // your LB's CIDRsHeaders are then believed only from those peers, and the
X-Forwarded-Forchain is walked right-to-left past them — so the first address no trusted proxy vouched for wins. A client connecting directly cannot forge its address at all, and one behind the proxy cannot forge it either: a proxy appends the address it saw, so anything the client wrote sits to the left of the truth and is skipped. That holds whether the proxy extends the client’s header line (nginx, AWS ALB) or adds one of its own below it (HAProxy’soption forwardfor) — every line is joined into one chain before the walk. A non-empty list enables resolution on its own;TrustProxyHeadersis not also required.TrustProxyHeaders: truewithout a list is the legacy mode: the leftmostX-Forwarded-Forentry, from any peer — which is the entry a client controls. It is safe only if the proxy strips both inbound headers itself. It warns at startup and fails underConfig.Strict. -
Set
Config.PathPrefixto a non-default value if the proxy mounts the API at a custom path. Don’t rewrite paths inside the application. -
Register
auth.CSRFif — and only if — browsers authenticate with cookies. A bearer token read from JavaScript is not an ambient credential, so a token-authenticated API is not CSRF-vulnerable and the middleware exempts bearer requests by default. Cookie-borne sessions are the case that needs it. See CSRF for both modes. The admin panel carries its own, unconditionally — see Admin Panel — and configuring one does not affect the other.
Operations
- Use a JSON-emitting
sloghandler in production so logs are structured and ingestable by your aggregator. - Set
Config.ServiceName— every log line and audit record carries it. - Point
readinessProbeat{prefix}/readyandlivenessProbeat{prefix}/live; tuneConfig.HealthTimeoutshorter than the probe timeout. A liveness probe aimed at a database-backed endpoint turns a dependency outage into a restart loop across every replica. - Leave
Probes.PublishReadinessChecksoff unless{prefix}/readyis reachable only from inside the cluster. It writes the names of your dependencies and which are failing into a body the probes serve without authentication — they bypassPipeline.Authby design.Config.Probesalso gates or unmounts each probe; see Gating and unmounting the probes. Gate/ready, not/live: a 401 from a liveness probe gets the container killed mid-drain. - Use
Config.PanicLoggerto route panics to a different sink than the rest of the framework logs, so they are easier to alert on.
Audit
- Register
db.AuditLogatmaniflex.Afterfor mutating operations. The records carry actor, model, operation, and a diff of the affected row. - Use
maniflex.ModelConfig{Versioned: true}on sensitive models. Every change writes a row to a sibling{model}_historytable.
Checklist
A reasonable production stack:
// HTTP/router layer — configure before maniflex.New
cfg.HTTPMiddlewares = append(cfg.HTTPMiddlewares,
response.CORSHeaders("https://app.example.com"))
server := maniflex.New(cfg)
// Auth
server.Pipeline.Auth.Register(auth.JWTAuth(secret, jwtOpts))
server.Pipeline.Auth.Register(auth.RequireRole("admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpDelete))
// Body
server.Pipeline.Deserialize.Register(body.MaxBodySize(32<<10),
maniflex.ForModel("PasswordReset"))
server.Pipeline.Validate.Register(body.StripUnknownFields())
// DB
server.Pipeline.DB.Register(db.Tenancy("org_id", tenantFromAuth))
server.Pipeline.DB.Register(db.RateLimit(db.RateLimitConfig{
RequestsPerMinute: 10,
Key: keyByIP,
}), maniflex.ForModel("PasswordReset"))
server.Pipeline.DB.Register(db.AuditLog(auditSink),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After))
// Response pipeline
server.Pipeline.Response.Register(
response.AddHeader("Strict-Transport-Security", "max-age=63072000"))
server.ObserveRequests(response.Logging(slog.Default()))
PostgreSQL in Production
The maniflex/db/postgres adapter is the recommended backend for any deployment
beyond a single process. This page collects production-relevant details that
go beyond the Database Backends overview.
Opening the adapter
import "github.com/xaleel/maniflex/db/postgres"
// Single primary (no replica) — pass "" for the read DSN.
db, err := postgres.Open(
os.Getenv("DB_WRITE_URL"), // primary / write DSN (required)
os.Getenv("DB_READ_URL"), // replica / read DSN ("" → reuse the primary)
server.Registry(),
)
if err != nil {
log.Fatal(err)
}
server.SetDB(db)
Open takes the write DSN, the read DSN, and the registry. The write DSN is
required; pass "" for the read DSN to route reads at the primary. Both are
standard libpq connection strings or URLs
(postgres://user:pass@host:5432/dbname?sslmode=require). MustOpen is the
panic-on-error variant for package-level initialisation.
Tuning pools and session settings
Open applies production defaults. To override them, use OpenWithConfig,
which takes a separate PoolConfig for the write and read pools plus one
SessionConfig. Any zero-value field is replaced by the default Open uses, so
you set only what you want to change:
schema := "orders"
db, err := postgres.OpenWithConfig(
writeDSN, readDSN, server.Registry(),
postgres.PoolConfig{MaxOpenConns: 5}, // write pool
postgres.PoolConfig{MaxOpenConns: 15}, // read pool
postgres.SessionConfig{
StatementTimeout: 10 * time.Second,
ApplicationName: "orders-api",
SchemaName: &schema, // search_path; auto-created on connect if absent
},
)
Connection-pool tuning
The defaults are sized for the smallest tier a managed provider sells.
OpenWithConfig exposes them as PoolConfig fields, set independently for the
write and read pools:
PoolConfig field | Default (write / read) | Considerations |
|---|---|---|
MaxOpenConns | 3 / 6 | (write + read) × processes ≤ max_connections − reserved |
MaxIdleConns | equal to MaxOpenConns | a pool this small keeps every connection; reopening costs a TLS handshake plus the session SET round trip |
ConnMaxLifetime | 30 min | rotate connections to pick up failover or DNS changes |
ConnMaxIdleTime | 5 min | release connections an idle process is no longer using |
Sizing against your server
Open builds both pools even when the read DSN is empty, so the number that
matters is the sum, multiplied by every process that connects — an API, a
worker, a migration job, each replica. Providers also reserve connections for
themselves, and you want a slot left for psql:
| Provider (entry tier) | max_connections | Usable |
|---|---|---|
| Heroku Postgres Essential-0/1/2 | 20 | 20 |
| DigitalOcean Managed PG (1 GiB) | 25 (25 per GiB) | 22 — 3 reserved |
GCP Cloud SQL db-f1-micro | 25 | ~22 |
| Azure Flexible Server B1ms | 50 | 35 — 15 reserved |
| Supabase Nano / Micro | 60 | 60 |
| Neon 0.25 CU | 104 | 97 — 7 reserved |
AWS RDS db.t4g.micro | ~112 (from instance memory) | ~110 |
At Open, the adapter reads the server’s own max_connections and logs a
WARN when the pools claim more than half of it. Set SessionConfig.Logger to
route that warning into your application’s logger; it defaults to
slog.Default(). The check never fails startup.
Raising the ceiling past what your instance needs makes things slower, not
faster: an entry-tier instance has one or two vCPUs, and Postgres throughput
peaks at a low multiple of core count. Extra connections buy queueing headroom
for bursts. Watch Stats().WaitCount and WaitDuration on the pool — sustained
waiting is the signal to size up, and the read pool is almost always the one
that needs it.
If you front Postgres with PgBouncer in transaction-pooling mode:
- Set
MaxOpenConnson the client to roughly match the bouncer’sdefault_pool_size. - Add
binary_parameters=yesto the DSN.lib/pqsends any parameterised query as Parse/Describe/Syncfollowed by Bind/Execute/Sync— two implicit transactions, so the bouncer may hand the server connection to another client in between and the unnamed prepared statement is gone when the Bind lands. The symptom is intermittentprepared statement "" does not existunder load. Withbinary_parameters=yes,lib/pqsends Parse/Bind/Describe/Execute/Sync in one packet, which is a single implicit transaction and safe. The framework caches no named statements of its own; this is the driver’s behaviour and applies to every query, including your raw ones. LISTEN/NOTIFYis not supported under transaction pooling — use the event-bus satellites instead.
Session settings
SessionConfig carries session-level parameters the adapter re-applies (SET …)
on every new physical connection — Postgres does not persist them across
reconnects, so they must be set per connection:
SessionConfig field | Default | Effect |
|---|---|---|
StatementTimeout | 30s | cancels any statement that runs longer (0 = server default) |
LockTimeout | 5s | aborts a statement that waits too long for a lock |
IdleInTransactionTimeout | 60s | aborts transactions left idle — guards against hung app code |
ApplicationName | maniflex | shown in pg_stat_activity and server logs |
TimeZone | UTC | session time zone for TIMESTAMPTZ rendering |
SchemaName | public | schema set as search_path (see below) |
Schema isolation (search_path)
By default the adapter operates in the public schema. Set
SessionConfig.SchemaName to scope every connection to a dedicated schema via
SET search_path — handy for multi-tenant deployments or co-locating several
apps in one database. The schema is created on connect when it does not yet
exist (CREATE SCHEMA IF NOT EXISTS), so AutoMigrate has somewhere to place
its tables; an existing schema is left untouched (a role with USAGE but not
CREATE still connects). The name must be a plain SQL identifier
([A-Za-z_][A-Za-z0-9_$]*); public is assumed to always exist and is never
re-created.
Each schema is migrated independently and gets its own constraints, so foreign
keys and their onDelete actions apply in every one. Before v0.3.2 they did
not: the check for an existing constraint was not scoped to a schema, and
constraint names are derived from table and column, so the same model in a
second schema looked like it had already been migrated. Whichever schema ran
AutoMigrate first got its foreign keys and every later one silently got none —
losing referential integrity and cascade/restrict/setNull with them. If
you ran a multi-schema deployment on an earlier version, verify the constraints
are present before relying on them:
SELECT table_schema, table_name, constraint_name
FROM information_schema.table_constraints
WHERE constraint_type = 'FOREIGN KEY'
ORDER BY table_schema, table_name;
Re-running AutoMigrate on the affected schemas adds what is missing.
Read replicas
When a read DSN is supplied, OpList and OpRead operations are routed to the
read pool; everything else uses the write pool. Trade-offs:
- Reads inside an active write transaction route to the write pool, even when a read replica is configured — read-your-writes is preserved.
- Pure read endpoints get the replica’s spare capacity without any code change.
- The application sees the replica’s normal lag for non-transactional reads. If a workflow depends on read-your-writes outside a transaction, run it inside a write transaction so the read lands on the primary.
FOR UPDATE and pessimistic locking
ctx.LockForUpdate translates to SELECT … FOR UPDATE on Postgres. The lock
is held until the enclosing transaction commits or rolls back. Typical use:
row, err := ctx.LockForUpdate("StockBalance", stockID)
if err != nil {
return err
}
if row["quantity"].(int64) < 1 {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "no inventory")
return nil
}
// safe to subtract — concurrent writers are blocked
Combine with maniflex.WithTransaction (or manual BeginTx) so the lock has a
transaction to scope it.
Isolation levels
maniflex.WithTransaction(&maniflex.TxOptions{Isolation: sql.LevelSerializable}) opens
the request in SERIALIZABLE isolation. Postgres serialisation failures
produce 40001 errors. Note that NormalizeError maps only the constraint
codes 23505 / 23502 / 23503 to *maniflex.ErrConstraint; a 40001
serialisation failure is not normalised — it propagates as a generic error
and surfaces as a 500. If you need transparent retry on serialisation
failures, detect the 40001 SQLSTATE yourself (e.g. in an action handler or a
custom middleware) and retry the transaction.
Most APIs do fine with the default READ COMMITTED plus LockForUpdate on
the contested rows; reach for SERIALIZABLE when the contention pattern is
more complex than a single row.
AutoMigrate at scale
AutoMigrate is enabled by default. For larger production databases, prefer:
server := maniflex.New(maniflex.Config{
DisableAutoMigrate: true,
// other application settings
})
…and run schema changes through a dedicated migration tool (sqlc-migrate, golang-migrate, Atlas, etc.). The framework’s auto-migrator is conservative — it never drops columns and emits straightforward DDL — but coordinating schema changes across replicas, dropped indexes, and rolling deploys is the migration tool’s job.
If you keep AutoMigrate enabled, run the first instance to completion
before scaling out; later instances will see all-up-to-date schema and skip
the work.
TLS and connectivity
- Use
sslmode=require(or stricter) on both the write and read DSN for any production connection. The driver respects the URL parameter. - For Cloud SQL / RDS, the connection string is generated by the cloud console; copy it verbatim and store it as a secret.
- Resolve DNS lookups inside the process — don’t pre-resolve at process
start. The
ConnMaxLifetimesetting then picks up the new endpoint automatically during failover.
Observability
- The adapter exposes pool statistics via
sql.DB.Stats(); export them with theresponse.Metricsrequest observer or a separate collector. - Set
Config.QueryTimeoutto bound slow queries; offending requests return504 TIMEOUTrather than holding a connection open. - Postgres logs (
log_min_duration_statement) andpg_stat_statementsare the canonical way to identify slow queries; the framework does not duplicate that.
Example 3: Order Processing System
This example assembles every advanced topic into one application — actions, raw queries, a custom aggregate endpoint, a transactional outbox, and a background worker. The domain is small (orders and inventory) so the integration is visible.
Domain
type Product struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required,filterable,sortable"`
Price float64 `json:"price" mfx:"required,min:0"`
Stock int64 `json:"stock" mfx:"required,min:0,filterable"`
}
type Order struct {
maniflex.BaseModel
maniflex.WithDeletedAt
CustomerID string `json:"customer_id" mfx:"required,filterable,immutable"`
Total float64 `json:"total" mfx:"required,min:0,filterable,sortable"`
Status string `json:"status" mfx:"required,enum:pending|paid|shipped|cancelled,default:pending,filterable,sortable"`
Lines []OrderLine `json:"lines,omitempty"`
}
type OrderLine struct {
maniflex.BaseModel
OrderID string `json:"order_id" mfx:"required,filterable,immutable"`
ProductID string `json:"product_id" mfx:"required,filterable,immutable"`
Quantity int64 `json:"quantity" mfx:"required,min:1"`
UnitPrice float64 `json:"unit_price" mfx:"required,min:0"`
}
// Transactional outbox row — appended in the same transaction as the order.
type OutboxEvent struct {
maniflex.BaseModel
Kind string `json:"kind" mfx:"required,filterable"`
Payload map[string]any `json:"payload" mfx:"required"`
Status string `json:"status" mfx:"required,enum:pending|done|failed,default:pending,filterable"`
ErrorMsg string `json:"error_msg" mfx:"filterable"`
}
Action: place an order atomically
POST /orders would normally just insert a row. Real order placement needs:
locking the products, decrementing stock, creating the order and lines,
queueing payment — all in one transaction. An action endpoint
handles this explicitly:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: placeOrder,
Middleware: []maniflex.MiddlewareFunc{auth.JWTAuth(secret)},
})
func placeOrder(ctx *maniflex.ServerContext) error {
var req struct {
Lines []struct {
ProductID string `json:"product_id"`
Quantity int64 `json:"quantity"`
} `json:"lines"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
// Reserve stock under a row lock for each product.
var total float64
type line struct {
productID string
qty int64
unit float64
}
var lines []line
for _, l := range req.Lines {
p, err := ctx.LockForUpdate("Product", l.ProductID)
if err != nil {
return err
}
stock := p["stock"].(int64)
if stock < l.Quantity {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK",
fmt.Sprintf("product %s has %d in stock", l.ProductID, stock))
return nil
}
if _, err := ctx.GetModel("Product").Update(l.ProductID, map[string]any{
"stock": stock - l.Quantity,
}); err != nil {
return err
}
unit := p["price"].(float64)
total += unit * float64(l.Quantity)
lines = append(lines, line{l.ProductID, l.Quantity, unit})
}
order, err := ctx.GetModel("Order").Create(map[string]any{
"customer_id": ctx.Auth.UserID,
"total": total,
"status": "pending",
})
if err != nil {
return err
}
for _, l := range lines {
if _, err := ctx.GetModel("OrderLine").Create(map[string]any{
"order_id": order["id"],
"product_id": l.productID,
"quantity": l.qty,
"unit_price": l.unit,
}); err != nil {
return err
}
}
// Outbox row — picked up by the background worker after commit.
if _, err := ctx.GetModel("OutboxEvent").Create(map[string]any{
"kind": "charge-payment",
"payload": map[string]any{"order_id": order["id"], "amount": total},
"status": "pending",
}); err != nil {
return err
}
if err := tx.Commit(); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusCreated,
Data: order,
}
return nil
}
If any step fails, the deferred Rollback reverts the order, the lines, and
the stock decrement together.
Revenue report: a custom action
The dashboard wants GET /revenue to return revenue per day. There is no
SQL-backed “query model”; mount a custom action whose handler runs
the aggregate with ctx.RawQuery:
server.Action(maniflex.ActionConfig{
Method: "GET",
Path: "/revenue",
Handler: func(ctx *maniflex.ServerContext) error {
rows, err := ctx.RawQuery(`
SELECT date(created_at) AS day, SUM(total) AS total
FROM orders
WHERE status IN ('paid', 'shipped')
GROUP BY day
ORDER BY day DESC
LIMIT 30`)
if err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{StatusCode: http.StatusOK, Data: rows}
return nil
},
})
Clients call GET /revenue and get the last 30 days of revenue. For plain
counts/sums over a registered model, the built-in GET /{model}/aggregate
endpoint avoids hand-written SQL — see Raw Queries & Aggregates.
Background worker: process the outbox
A separate goroutine (or process) sweeps OutboxEvent, processes each event,
and updates its status. The worker uses the same registered models:
func runOutboxWorker(server *maniflex.Server) {
// Background workers have no ServerContext — build one with NewBackground,
// then use ctx.GetModel (or the typed maniflex.List/Read helpers).
bg := maniflex.NewBackground(context.Background(), server.DB(), server.Registry())
events := bg.GetModel("OutboxEvent")
for range time.Tick(2 * time.Second) {
rows, _ := events.List(&maniflex.QueryParams{
Filters: []*maniflex.FilterExpr{{
Field: "status", Operator: maniflex.OpEq, Value: "pending",
}},
Limit: 20,
})
for _, ev := range rows {
if err := process(ev); err != nil {
events.Update(ev["id"].(string), map[string]any{
"status": "failed",
"error_msg": err.Error(),
})
continue
}
events.Update(ev["id"].(string), map[string]any{"status": "done"})
}
}
}
The worker is part of the same binary in this example. In production, a
satellite from jobs/redis (see Events & Background Jobs)
replaces the polling loop with a durable queue and at-least-once delivery.
What this example tied together
- Actions for endpoints that don’t fit standard CRUD (
/orders/place). LockForUpdateto safely decrement stock under contention.ctx.BeginTxso the order, lines, and stock change commit atomically.- The transactional outbox pattern for crossing the boundary between the request transaction and external side effects.
- A custom action for the revenue report — a read endpoint built from raw
SQL (
ctx.RawQuery). - A background worker consuming a registered model as its work queue.
Each piece is documented on its own page in the Advanced section: actions, raw queries, batch-saga, and events-jobs.
1. Overview & Scaffolding
This is the start of a ten-part walkthrough. The end product is a small but realistic bookstore API — users sign in, browse books and reviews, place orders, and the system notifies them when an order ships. Each step introduces one capability of the framework; nothing is hand-waved.
Where the reference pages describe a feature in isolation, the tutorial shows how the features compose in a real application.
What you’ll build
bookstore — an HTTP API with:
| Endpoint family | Capability |
|---|---|
/api/users | sign-up, JWT auth, role-based access |
/api/books, /api/authors, /api/genres | the catalogue, with relations and full querying |
/api/reviews | per-book ratings with custom validation |
Book cover upload via multipart/form-data | a file field |
POST /api/orders/place | a transactional action with stock locking |
| Outbox + background worker | email a receipt after each order |
By the end of part 10 the same code base will be deployed to production with PostgreSQL, env-driven configuration, and a health probe.
Prerequisites
- Go 1.25.12 or newer.
- A text editor and a terminal. Nothing else; the development database is pure-Go SQLite, no CGo needed.
Project layout
The app follows the layer-based layout described in App Anatomy. It grows over the tutorial; this is the shape at the end:
bookstore/
├── go.mod
├── main.go # wiring: create, register, set DB, serve
├── config.go # maniflex.Config assembly
├── models/ # one file per model
│ ├── user.go
│ ├── book.go
│ ├── author.go
│ ├── genre.go
│ ├── review.go
│ ├── order.go
│ └── outbox.go
├── middleware/ # custom middleware
│ ├── auth.go
│ ├── validate.go
│ └── register.go
├── actions/ # custom endpoints
│ └── orders.go
├── jobs/ # background workers
│ └── outbox.go
└── static/
└── openapi.html # bundled API viewer
Bootstrap
Create the directory, initialise a module, and add the framework:
mkdir bookstore && cd bookstore
go mod init bookstore
go get github.com/xaleel/maniflex github.com/xaleel/maniflex/db/sqlite
main.go starts as the smallest maniflex app:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
func main() {
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
Documentation: maniflex.DocumentationConfig{Public: true},
})
db, err := sqlite.Open("./bookstore.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
if err := server.Start(); err != nil {
log.Fatal(err)
}
}
Run it:
go run .
The server starts on :8080. Nothing is registered yet, so the only endpoints
that respond are the probes — /api/live, /api/ready, /api/health — but
GET /api/openapi.json already serves a valid (empty) OpenAPI document.
Why the four-step shape
The framework’s lifecycle never deviates from the same four steps — create → register → set DB → serve. Everything we add in the next nine parts plugs into one of those steps:
- Create:
maniflex.Configgrows with logger, file storage, query timeout, and so on. - Register: more models, each carrying their
mfx:tags. - Set DB: SQLite for development, PostgreSQL by part 10.
- Serve: more pipeline middleware, but
Start()itself stays the same.
The structure of main.go will not change between this part and part 10. The
file simply grows new lines.
Next
In Part 2 — Users & Auth we add the User model, a
sign-up endpoint, and JWT-based authentication. By the end of part 2 every
write request will require a valid token.
2. Users & Auth
We start with the User model and the auth layer. By the end of this part,
the API has a sign-up endpoint, password hashing, JWT-based authentication on
all writes, and a role-based admin gate on user deletion.
The model
Create models/user.go:
package models
import "github.com/xaleel/maniflex"
type User struct {
maniflex.BaseModel
Email string `json:"email" mfx:"required,filterable,unique,immutable"`
Password string `json:"password" mfx:"required,writeonly,minlen:8"`
Name string `json:"name" mfx:"required,filterable,sortable"`
Role string `json:"role" mfx:"required,enum:admin|customer,default:customer,filterable"`
}
A few tag choices to notice:
emailisuniqueandimmutable— once a user signs up, the address is the account identity.passwordiswriteonlyso it is accepted on input but never appears in responses, andminlen:8enforces a minimum length of eight characters. (min:/max:bound a number’s value, not a string’s length — declaring one on a string is a registration error naming this tag.)roleis an enum with a safe default; we’ll gateadminwrites separately in middleware.
Register it from main.go:
import "bookstore/models"
server.MustRegister(models.User{})
That alone gives you POST /api/users (sign-up), GET /api/users/{id},
PATCH /api/users/{id}, DELETE /api/users/{id}, and GET /api/users. But
right now anyone can call any of them — we need to hash passwords on the way
in and gate the writes.
Hashing passwords
Add maniflex/middleware/service/bcrypt:
go get github.com/xaleel/maniflex/middleware/service/bcrypt
Then register the hashing middleware on the Service step, scoped to User
create and update:
import (
"github.com/xaleel/maniflex/middleware/service"
"github.com/xaleel/maniflex/middleware/service/bcrypt"
)
server.Pipeline.Service.Register(
service.HashField("password", bcrypt.Hasher()),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
The middleware reads the password field (ctx.Field), replaces it with the
bcrypt hash via ctx.SetField, and lets the DB step write the hash. Nothing else in the application
needs to know that the column is hashed.
JWT authentication
Pull in maniflex/middleware/auth:
go get github.com/xaleel/maniflex/middleware/auth
Register JWTAuth on the Auth step, scoped to writes — we’ll let reads stay
public for now. Auth scoping (ForModel / ForOperation) is
inclusion-only: a middleware runs only for the models and operations you
scope it to, so anything you don’t scope onto stays public.
import "github.com/xaleel/maniflex/middleware/auth"
// Protect updates and deletes on every model…
server.Pipeline.Auth.Register(
auth.JWTAuth("dev-secret", auth.JWTOptions{Issuer: "bookstore"}),
maniflex.ForOperation(maniflex.OpUpdate, maniflex.OpDelete),
)
// …and protect creates only where a session is required. "User" is deliberately
// left out, so POST /api/users (sign-up) needs no token.
server.Pipeline.Auth.Register(
auth.JWTAuth("dev-secret", auth.JWTOptions{Issuer: "bookstore"}),
maniflex.ForModel("Book"),
maniflex.ForOperation(maniflex.OpCreate),
)
JWTAuth verifies the Authorization: Bearer <token> header, parses the
claims, and populates ctx.Auth with the user ID and roles. Tokens fail with
401 UNAUTHORIZED; missing tokens fail the same way.
Sign-up (POST /api/users) is a create on User — a model we never scoped
auth onto — so it stays public with no extra middleware. There is no
AllowPublicWrite helper: public access always comes from not scoping the
authenticator onto an operation, because scoping is inclusion-only.
Role-gated deletes
Only admins should be able to delete users. auth.RequireRole does exactly
that:
server.Pipeline.Auth.Register(
auth.RequireRole("admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpDelete),
)
It runs after JWTAuth, so by the time it fires ctx.Auth.Roles is
populated. Non-admin users receive 403 FORBIDDEN.
Issuing tokens
JWTAuth only verifies tokens — it does not issue them. For development we
add a tiny token endpoint as a custom action:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/auth/login",
Handler: login,
})
func login(ctx *maniflex.ServerContext) error {
var req struct {
Email string `json:"email"`
Password string `json:"password"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
rows, err := ctx.RawQuery(
`SELECT id, password, role FROM users WHERE email = ?`, req.Email,
)
if err != nil || len(rows) == 0 {
ctx.Abort(http.StatusUnauthorized, "INVALID_CREDENTIALS", "bad email or password")
return nil
}
user := rows[0]
if !checkBcrypt(user["password"].(string), req.Password) {
ctx.Abort(http.StatusUnauthorized, "INVALID_CREDENTIALS", "bad email or password")
return nil
}
token := signJWT("dev-secret", user["id"].(string), []string{user["role"].(string)})
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusOK,
Data: map[string]any{"token": token},
}
return nil
}
signJWT and checkBcrypt are small helpers built on github.com/golang-jwt/jwt/v5
and maniflex/middleware/service/bcrypt. In production this endpoint would
issue a refresh token too — for now, a single bearer token is enough.
Trying it out
# Sign up
curl -X POST localhost:8080/api/users \
-H 'Content-Type: application/json' \
-d '{"email":"[email protected]","password":"hunter22!","name":"Alice"}'
# Log in
TOKEN=$(curl -s -X POST localhost:8080/api/auth/login \
-H 'Content-Type: application/json' \
-d '{"email":"[email protected]","password":"hunter22!"}' \
| jq -r .data.token)
# Authenticated read (lists are public, but writes need the token)
curl -X PATCH localhost:8080/api/users/<id> \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"name":"Alice A."}'
What we built
| Capability | How |
|---|---|
| Sign-up | POST /api/users stays public (auth not scoped onto it) |
| Password hashing | service.HashField("password", bcrypt.Hasher()) on the Service step |
| Bearer-token auth on writes | auth.JWTAuth on the Auth step |
| Admin-only delete | auth.RequireRole("admin") |
| Token issuance | /api/auth/login action |
Next
In Part 3 — Modeling Domain Entities & Relations we add
the catalogue: Author, Genre, Book, and Review, wired up with
BelongsTo, HasMany, and many-to-many relations.
3. Modeling Domain Entities & Relations
With users in place, we model the catalogue. A book belongs to one author and many genres, and accumulates many reviews — three relations, two flavours.
The catalogue
Create the models. Each one lives in its own file under models/.
// models/author.go
type Author struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required,filterable,sortable"`
Bio string `json:"bio"`
Books []Book `json:"books,omitempty"` // HasMany
}
// models/genre.go
type Genre struct {
maniflex.BaseModel
Label string `json:"label" mfx:"required,filterable,sortable,unique"`
Books []Book `json:"books,omitempty" mfx:"through:BookGenre"`
}
// models/book.go
type Book struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
ISBN string `json:"isbn" mfx:"required,filterable,unique"`
Price float64 `json:"price" mfx:"required,min:0,filterable,sortable"`
Stock int64 `json:"stock" mfx:"required,min:0,filterable"`
PublishedAt string `json:"published_at" mfx:"filterable,sortable"`
AuthorID string `json:"author_id" mfx:"required,filterable"` // BelongsTo Author
Genres []Genre `json:"genres,omitempty" mfx:"through:BookGenre"` // ManyToMany
Reviews []Review `json:"reviews,omitempty"` // HasMany
}
// models/book_genre.go — the junction model for Book ↔ Genre.
type BookGenre struct {
maniflex.BaseModel
BookID string `json:"book_id" mfx:"required,filterable,immutable"`
GenreID string `json:"genre_id" mfx:"required,filterable,immutable"`
}
// models/review.go
type Review struct {
maniflex.BaseModel
maniflex.WithDeletedAt
BookID string `json:"book_id" mfx:"required,filterable,immutable"` // BelongsTo Book
UserID string `json:"user_id" mfx:"required,filterable,immutable"` // BelongsTo User
Rating int `json:"rating" mfx:"required,min:1,max:5,filterable,sortable"`
Body string `json:"body" mfx:"required"`
}
Three relation styles in one place:
- BelongsTo (convention) —
Book.AuthorID→Author,Review.BookID→Book,Review.UserID→User. No tags required; the framework reads theIDsuffix. - HasMany —
Author.Books,Book.Reviews. A slice of the related struct, not a column on this table. - ManyToMany —
Book.Genres↔Genre.Booksthrough the explicitBookGenrejunction model. Thethrough:tag names the junction; both sides declare it.
Registering
All five models go to MustRegister:
server.MustRegister(
models.User{},
models.Author{},
models.Genre{},
models.Book{},
models.BookGenre{},
models.Review{},
)
AutoMigrate creates the tables. The junction book_genres carries
book_id and genre_id; the framework wires the Book ↔ Genre relation
from the through: tag.
Trying the relations
Create an author, a genre, and a book:
AUTH=$(curl -s -X POST localhost:8080/api/authors -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name":"Ursula K. Le Guin"}' | jq -r .data.id)
SCIFI=$(curl -s -X POST localhost:8080/api/genres -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"label":"Science Fiction"}' | jq -r .data.id)
BOOK=$(curl -s -X POST localhost:8080/api/books -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"title\":\"The Dispossessed\",\"isbn\":\"9780061054884\",\"price\":12.99,\"stock\":10,\"author_id\":\"$AUTH\"}" \
| jq -r .data.id)
# Tag it as sci-fi via the junction model.
curl -X POST localhost:8080/api/book_genres -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"book_id\":\"$BOOK\",\"genre_id\":\"$SCIFI\"}"
Now include the related rows in a read:
curl "localhost:8080/api/books/$BOOK?include=author,genres,reviews"
The response carries author (a single object), genres (an array), and
reviews (an empty array for now). Each include is a separate query against
the related table.
Filtering through relations
Filters can traverse relations using dot notation. The related field must be
filterable:
# All books written by anyone whose name starts with "Ursula"
curl "localhost:8080/api/books?filter=author.name:ilike:Ursula%25"
# All books in the "Science Fiction" genre
curl "localhost:8080/api/books?filter=genres.label:eq:Science+Fiction&include=genres"
Filtering does not require an include; including merely returns the related
rows. You can join one and not the other freely.
Cascading deletes
A deleted author should not orphan books. Update the Book.AuthorID tag to
declare a cascade:
AuthorID string `json:"author_id" mfx:"required,filterable,relation:Author;onDelete:cascade"`
Author Author `json:"author,omitempty"`
The companion Author field is now needed because the explicit relation:
tag must name a companion field of the target type. We also gain a slightly
better OpenAPI: the spec carries the relation explicitly.
onDelete:setNull and onDelete:restrict are the alternatives — see
Relations.
What we built
| Concept | Where |
|---|---|
| BelongsTo (convention) | Book.AuthorID, Review.BookID, Review.UserID |
| HasMany | Author.Books, Book.Reviews |
| ManyToMany | Book.Genres ↔ Genre.Books via BookGenre |
| Explicit relation with cascade | Book.AuthorID after the cascade edit |
| Soft delete | maniflex.WithDeletedAt on Review |
| Filtering through relations | ?filter=author.name:ilike:Ursula% |
Next
In Part 4 — Validation & Business Rules we tighten the rules: ISBNs follow a specific format, a user may not review the same book twice, and reviews on out-of-stock books are blocked.
4. Validation & Business Rules
The mfx: tag rules from Part 3 cover the common case — required fields,
numeric ranges, enums, uniqueness hints. Anything that goes beyond a single
field belongs in Validate-step middleware. This part adds three rules:
- ISBNs must be 13 digits with hyphens optional.
- A user may not review the same book twice.
- A user must have bought a book before reviewing it.
Field-format validation
validate.RegexField is enough for the ISBN check:
import "github.com/xaleel/maniflex/middleware/validate"
server.Pipeline.Validate.Register(
validate.RegexField("isbn", `^(?:97[89])?\d{10}$`),
maniflex.ForModel("Book"),
)
The middleware runs after the mfx: tag rules. A malformed ISBN aborts the
request with 422 VALIDATION_ERROR and the field in details.
We strip hyphens before validating so the client can send the human-readable form. A small Service-step middleware does the rewrite:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if raw, ok := ctx.Field("isbn"); ok {
if v, ok := raw.(string); ok {
ctx.SetField("isbn", strings.ReplaceAll(v, "-", ""))
}
}
return next()
}, maniflex.ForModel("Book"), maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate))
Order matters: Validate runs before Service in the pipeline. We’re rewriting
after validation has confirmed the cleaned-up format would pass. To
validate the cleaned value, swap the registration so the cleanup is on
Validate at maniflex.Before (the default).
“One review per book per user”
Two flavours of uniqueness:
- Schema uniqueness (
mfx:"unique") — adds aUNIQUEconstraint on a single column. Good for an email address. - Cross-column uniqueness — needs custom validation, because no single column is unique.
For reviews we need both book_id and user_id to be unique together. A
small middleware that consults the database:
server.Pipeline.Validate.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
rows, err := ctx.RawQuery(
`SELECT id FROM reviews
WHERE book_id = ? AND user_id = ? AND deleted_at IS NULL`,
bookID, ctx.Auth.UserID,
)
if err != nil {
return err
}
if len(rows) > 0 {
ctx.Abort(http.StatusConflict, "ALREADY_REVIEWED",
"you have already reviewed this book")
return nil
}
return next()
}, maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate))
Two things to notice:
- We use
ctx.RawQueryrather thanctx.GetModel(...).Listbecause we need a count, not the rows. Either works. - The
user_idwe check isctx.Auth.UserID, not the body’suser_id. In the next section we’ll force the body field to match.
“Body owner must match authenticated user”
Letting a client supply user_id is asking for impersonation. service.OwnerScope
from the catalogue forces the field on every create:
import "github.com/xaleel/maniflex/middleware/service"
server.Pipeline.Service.Register(
service.OwnerScope("user_id"),
maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate),
)
OwnerScope reads ctx.Auth.UserID and sets it on the body via ctx.SetField,
overwriting whatever the client sent. A client who omits the field gets it
filled in; a client who sets it to someone else’s ID has the value
overwritten silently.
This is a good place to apply Forbidden values
on role for User too — defence in depth against a privilege-escalation
payload:
server.Pipeline.Validate.Register(
validate.ForbiddenValues("role", "admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
A normal sign-up cannot self-promote to admin; an admin still can, because
they go through PATCH not POST, and the rule is scoped to create only.
Cross-field rules
The third rule — review only books you’ve bought — depends on a model
(Order) we haven’t built yet. We come back to it in Part 7 once orders
exist, using the same Validate.Register shape with a join query:
server.Pipeline.Validate.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
rows, _ := ctx.RawQuery(
`SELECT 1
FROM order_lines ol
JOIN orders o ON o.id = ol.order_id
WHERE o.customer_id = ?
AND ol.book_id = ?
AND o.status IN ('paid','shipped')`,
ctx.Auth.UserID, bookID,
)
if len(rows) == 0 {
ctx.Abort(http.StatusForbidden, "PURCHASE_REQUIRED",
"you may only review books you have bought")
return nil
}
return next()
}, maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate))
We leave this in the codebase as a stub for now and complete it in Part 7.
Where each rule lives
| Rule | Where | Why |
|---|---|---|
| ISBN format | Validate (catalogue) | format check on one field |
| One-review-per-book | Validate (custom) | queries another row |
user_id belongs to caller | Service (catalogue) | mutates body |
role cannot be admin on sign-up | Validate (catalogue) | rejects body value |
| Must have purchased | Validate (custom, deferred) | cross-model query |
The general rule: field-level rules go in Validate; rules that mutate the body go in Service; rules that need the row to exist go in After-DB.
Next
In Part 5 — File Uploads we add a cover image to each book, served from local storage during development and ready to swap for S3 in production.
5. File Uploads
A book needs a cover image. In this part we add a file field to the Book
model, configure local storage for development, and learn how the same code
handles a swap to S3 in production.
Adding the file field
Edit models/book.go:
type Book struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
ISBN string `json:"isbn" mfx:"required,filterable,unique"`
Price float64 `json:"price" mfx:"required,min:0,filterable,sortable"`
Stock int64 `json:"stock" mfx:"required,min:0,filterable"`
PublishedAt string `json:"published_at" mfx:"filterable,sortable"`
AuthorID string `json:"author_id" mfx:"required,filterable,relation:Author;onDelete:cascade"`
Author Author `json:"author,omitempty"`
Cover string `json:"cover" mfx:"file,max_size:2MB,accept:image/png|image/jpeg"`
Genres []Genre `json:"genres,omitempty" mfx:"through:BookGenre"`
Reviews []Review `json:"reviews,omitempty"`
}
The cover field is a string in Go and a string in the database, but the
mfx:"file" tag opts the model into multipart uploads. The column stores the
storage key — a path under whichever backend you have configured.
max_size and accept are enforced in the framework, before the upload
reaches storage. An oversize 5 MB JPEG is rejected with
413 FILE_TOO_LARGE, and a disallowed application/pdf with
415 FILE_TYPE_NOT_ALLOWED — neither is ever written.
Configuring storage
For development we use local disk. The maniflex/storage package ships a ready
implementation:
import "github.com/xaleel/maniflex/storage"
fs, err := storage.NewLocalStorage("./uploads")
if err != nil {
log.Fatal(err)
}
defer fs.Close()
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
FileStorage: fs,
})
./uploads is created if it doesn’t exist. Every uploaded file lands under
uploads/<uuid>/<sanitised-filename> so collisions are impossible.
Local storage uses a directory-scoped root, so a symlink below uploads cannot
redirect file or metadata operations outside it.
Uploading a cover
There are two ways to attach a cover, both supported out of the box.
1. Multipart upload alongside create
The client sends multipart/form-data with one part per field:
curl -X POST localhost:8080/api/books \
-H "Authorization: Bearer $TOKEN" \
-F 'title=The Dispossessed' \
-F 'isbn=9780061054884' \
-F 'price=12.99' \
-F 'stock=10' \
-F "author_id=$AUTH" \
-F 'cover=@./covers/dispossessed.jpg;type=image/jpeg'
The framework parses the multipart envelope, streams cover into the
storage backend, writes the resulting key into the column, and persists the
row. The response is the usual JSON envelope:
{
"data": {
"id": "...",
"title": "The Dispossessed",
"cover": "uploads/3f2b.../dispossessed.jpg",
...
}
}
2. Two-step upload + reference
For large files or out-of-band uploads, hit the standalone
/files endpoint first:
KEY=$(curl -s -X POST localhost:8080/files \
-H "Authorization: Bearer $TOKEN" \
-F 'file=@./covers/dispossessed.jpg;type=image/jpeg' \
| jq -r .data.key)
curl -X POST localhost:8080/api/books \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"title\":\"The Dispossessed\",\"isbn\":\"9780061054884\",\"price\":12.99,\"stock\":10,\"author_id\":\"$AUTH\",\"cover\":\"$KEY\"}"
The file field accepts a plain string in JSON — the storage key returned
by /files. The framework recognises that the value is already a key (not a
new upload) and stores it as-is.
Downloading a cover
Storage keys are served at /files/{key...}:
curl 'localhost:8080/files/uploads/3f2b.../dispossessed.jpg' --output cover.jpg
The handler sets Content-Type, Content-Disposition: inline, and
Content-Length from the metadata stored alongside the file.
For a permission layer in front of downloads — say, only registered users can fetch covers — add Auth middleware to the file route just as you would for any model route.
Automatic cleanup
The framework tracks the row that owns each key. A file is deleted from storage when:
- the owning row is hard-deleted, or
- the field is overwritten by a
PATCHthat supplies a new file or key.
Book does not embed WithDeletedAt, so a delete is a hard-delete and the
cover goes away too. If you want covers to outlive book deletions (for an
audit trail), tag the field with auto_delete:false:
Cover string `json:"cover" mfx:"file,max_size:2MB,accept:image/*,auto_delete:false"`
Swapping in S3
FileStorage is a four-method interface — Store, Retrieve, Delete,
Exists. A drop-in S3 implementation looks like:
type S3Storage struct{ client *s3.Client; bucket string }
func (s *S3Storage) Store(ctx context.Context, key string, r io.Reader, meta maniflex.FileMeta) error {
_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
Bucket: &s.bucket,
Key: &key,
Body: r,
ContentType: &meta.ContentType,
})
return err
}
// Retrieve, Delete, Exists similarly.
Swap storage.NewLocalStorage(...) for the new type in main.go and nothing
else changes. The same model code, the same endpoints, the same multipart
parser. The model never knows.
What we built
| Capability | How |
|---|---|
| File field on Book | mfx:"file,max_size:...,accept:..." |
| Local storage backend | storage.NewLocalStorage("./uploads") |
| Multipart upload | The framework auto-detects multipart/form-data on create/update |
| Pre-uploaded key reference | Plain string in the JSON body |
| Standalone upload | POST /files, returns a key |
| Backend-agnostic | maniflex.FileStorage interface — swap to S3 with no model change |
Next
In Part 6 — Filtering, Sorting & Pagination we build a catalogue browser: lookup books by title, sort by price or publication date, paginate the results, and combine includes with filters.
6. Filtering, Sorting & Pagination
The catalogue is in place. This part stitches together the query parameters
exposed by every list endpoint — filter, sort, include, page,
limit — to build a real browse experience.
Recap: opt-in fields
Every queryable field carries the relevant mfx: tag. Book’s fields are
already tagged from Part 3:
| Field | Tags |
|---|---|
title | filterable,sortable |
isbn | filterable,unique |
price | filterable,sortable |
stock | filterable |
published_at | filterable,sortable |
author_id | filterable |
Untagged fields are deliberately invisible to clients — a query string that
references them is rejected with 400 INVALID_QUERY.
Filter operators
All the operators on one model:
# Title contains "wind" (case-insensitive)
curl 'localhost:8080/api/books?filter=title:ilike:%25wind%25'
# Priced between $10 and $20
curl 'localhost:8080/api/books?filter=price:gte:10&filter=price:lte:20'
# Out of stock
curl 'localhost:8080/api/books?filter=stock:eq:0'
# In any of three genres
curl 'localhost:8080/api/books?filter=genres.label:in:Fantasy,Sci-Fi,Mystery'
# Published after a date, returned newest first
curl 'localhost:8080/api/books?filter=published_at:gte:2020-01-01&sort=published_at:desc'
Multiple filters compose with AND. The framework parses each filter once in
the Deserialize step into ctx.Query.Filters, then the DB step translates
the slice into a WHERE clause.
On a date/time field, a full-timestamp bound is normalised to the same
fixed-width UTC form the write path stores, so the comparison orders
chronologically on SQLite (which compares timestamp columns as text) and a zone
offset is honoured. A value carrying an offset like
created_at:gte:2026-01-01T12:00:00+05:00 is compared in UTC; a date-only bound
like published_at:gte:2020-01-01 above keeps its plain meaning.
Relation filters
?filter=genres.label:in:... filters through the many-to-many junction.
Dot-notation works on any relation whose target field is filterable:
# Books by an author whose name contains "Le Guin"
curl 'localhost:8080/api/books?filter=author.name:ilike:%25Le+Guin%25&include=author'
# Reviews left on books with a specific ISBN
curl 'localhost:8080/api/reviews?filter=book.isbn:eq:9780061054884&include=book'
The include is independent of the filter — you can filter on a relation without returning it, and vice versa.
Sorting
?sort=field:direction for one column, repeat for tie-breakers:
# Cheapest first, oldest among ties
curl 'localhost:8080/api/books?sort=price:asc&sort=published_at:asc'
Only sortable fields work. BaseModel’s created_at and updated_at are
sortable by default.
Pagination
The defaults — page 1, 20 per page — work everywhere. Override per request:
curl 'localhost:8080/api/books?page=2&limit=50'
limit is clamped at 200; oversize requests are silently reduced. List
responses carry pagination metadata in meta:
{
"data": [ ... ],
"meta": { "total": 137, "page": 2, "limit": 50, "pages": 3 }
}
For models where the rows are expensive to render — full audit logs,
analytics tables — register db.Paginate from the catalogue to lower the
ceiling per model:
import "github.com/xaleel/maniflex/middleware/db"
server.Pipeline.DB.Register(db.Paginate(50), maniflex.ForModel("AuditLog"))
Includes
?include=relation1,relation2 populates nested objects in the response.
Includes are separate queries — they do not multiply rows or affect
pagination of the primary list:
curl 'localhost:8080/api/books/<id>?include=author,genres,reviews'
The relation keys come from the model declarations — see
Relations for how they are derived. For a BelongsTo the
result is a single nested object; for HasMany and ManyToMany, an array.
Combining everything
A realistic “browse” call:
curl 'localhost:8080/api/books?filter=genres.label:eq:Science+Fiction
&filter=stock:gt:0
&filter=price:lte:20
&sort=published_at:desc
&include=author,genres
&page=1
&limit=12'
The framework executes this as:
- Parse query →
ctx.Query.Filters,Sorts,Includes,Page,Limit. - Run the main
SELECTwith the WHERE + ORDER BY + LIMIT/OFFSET. - Issue follow-up queries for each include, batched by foreign key.
- Compose the JSON envelope.
Hardcoding tenant scope
In some applications a request from one customer should never see another
customer’s rows. We don’t have multi-tenancy in the bookstore, but the
mechanism is worth knowing. db.Tenancy enforces row-level scoping
unconditionally:
server.Pipeline.DB.Register(
db.Tenancy("organization_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.TenantID
}),
)
Once registered, every list, read, update, and delete is silently filtered to
organization_id = ctx.Auth.TenantID. The client cannot override or escape
it.
Custom filters in middleware
ctx.Query.Filters is a slice — middleware can append to it before the DB
step runs. We’ll use this in Part 7 when we want logged-in customers to see
only their own orders:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
ctx.Query.Filters = append(ctx.Query.Filters, &maniflex.FilterExpr{
Field: "customer_id",
Operator: maniflex.OpEq,
Value: ctx.Auth.UserID,
})
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpList))
The pattern is the same as the bigger Tenancy middleware — write a filter,
let the DB step honour it.
Next
In Part 7 — Custom Endpoints & Actions we add order placement: a transactional action that locks stock, creates the order and its lines, and writes an outbox row that Part 8’s background worker will consume.
7. Custom Endpoints & Actions
Customers need to place orders. A simple POST /api/orders would just
insert one row, but real order placement also has to lock stock, create
line items, and queue a downstream notification — all atomically. This is
the textbook use case for a custom action.
The models
Two new entities. Both opt into soft-delete so an audit trail survives.
// models/order.go
type Order struct {
maniflex.BaseModel
maniflex.WithDeletedAt
CustomerID string `json:"customer_id" mfx:"required,filterable,immutable"`
Total float64 `json:"total" mfx:"required,min:0,filterable,sortable"`
Status string `json:"status" mfx:"required,enum:pending|paid|shipped|cancelled,default:pending,filterable,sortable"`
Lines []OrderLine `json:"lines,omitempty"`
}
// models/order_line.go
type OrderLine struct {
maniflex.BaseModel
OrderID string `json:"order_id" mfx:"required,filterable,immutable"`
BookID string `json:"book_id" mfx:"required,filterable,immutable"`
Quantity int64 `json:"quantity" mfx:"required,min:1"`
UnitPrice float64 `json:"unit_price" mfx:"required,min:0"`
}
// models/outbox.go — Part 8 will consume rows from here.
type OutboxEvent struct {
maniflex.BaseModel
Kind string `json:"kind" mfx:"required,filterable"`
Payload map[string]any `json:"payload" mfx:"required"`
Status string `json:"status" mfx:"required,enum:pending|done|failed,default:pending,filterable"`
ErrorMsg string `json:"error_msg"`
}
Register them. We also tenancy-scope reads of Order to the calling
customer so one user cannot list another’s orders:
server.MustRegister(models.Order{}, models.OrderLine{}, models.OutboxEvent{})
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
ctx.Query.Filters = append(ctx.Query.Filters, &maniflex.FilterExpr{
Field: "customer_id", Operator: maniflex.OpEq, Value: ctx.Auth.UserID,
})
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpList))
Why an action
The standard POST /api/orders would insert one Order row and stop. We
need:
- Lock the books so concurrent buyers don’t oversell stock.
- Decrement stock on every line.
- Create the order.
- Create one
OrderLineper book. - Append an outbox row describing the order, in the same transaction.
A single transaction must cover all five. The Service step on POST /orders
sees only the order body — the lines come from the client. We could write
five middleware functions, but a custom action keeps
the transaction obvious and the trimmed pipeline lighter:
Auth → action handler → Response
Deserialize, Validate, Service, and DB are skipped. Our handler does
its own parsing and database work.
The handler
actions/orders.go:
package actions
func PlaceOrder(ctx *maniflex.ServerContext) error {
var req struct {
Lines []struct {
BookID string `json:"book_id"`
Quantity int64 `json:"quantity"`
} `json:"lines"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
if len(req.Lines) == 0 {
ctx.Abort(http.StatusBadRequest, "EMPTY_ORDER", "an order must contain at least one line")
return nil
}
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
// 1+2: lock each book row and decrement stock.
type planned struct {
bookID string
quantity int64
unitPrice float64
}
var plan []planned
var total float64
for _, l := range req.Lines {
book, err := ctx.LockForUpdate("Book", l.BookID)
if err != nil {
ctx.Abort(http.StatusNotFound, "BOOK_NOT_FOUND",
fmt.Sprintf("book %s does not exist", l.BookID))
return nil
}
stock := book["stock"].(int64)
if stock < l.Quantity {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK",
fmt.Sprintf("book %s has %d in stock", l.BookID, stock))
return nil
}
if _, err := ctx.GetModel("Book").Update(l.BookID, map[string]any{
"stock": stock - l.Quantity,
}); err != nil {
return err
}
price := book["price"].(float64)
total += price * float64(l.Quantity)
plan = append(plan, planned{l.BookID, l.Quantity, price})
}
// 3: the Order row.
order, err := ctx.GetModel("Order").Create(map[string]any{
"customer_id": ctx.Auth.UserID,
"total": total,
"status": "pending",
})
if err != nil {
return err
}
// 4: one OrderLine per book.
for _, p := range plan {
if _, err := ctx.GetModel("OrderLine").Create(map[string]any{
"order_id": order["id"],
"book_id": p.bookID,
"quantity": p.quantity,
"unit_price": p.unitPrice,
}); err != nil {
return err
}
}
// 5: outbox row — picked up by the worker in Part 8.
if _, err := ctx.GetModel("OutboxEvent").Create(map[string]any{
"kind": "order-placed",
"payload": map[string]any{
"order_id": order["id"],
"customer_id": ctx.Auth.UserID,
"total": total,
},
"status": "pending",
}); err != nil {
return err
}
if err := tx.Commit(); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusCreated,
Data: order,
}
return nil
}
Three things worth pointing out:
ctx.LockForUpdateacquires a row-level write lock that lasts until the transaction ends. A concurrent buyer hitting the same book waits at that line until we commit or roll back.- All five inserts share
ctx.Tx.ctx.GetModel(...).Createroutes through the transaction automatically — there is no separate “transactional client” to thread. defer tx.Rollback()is safe after a successfulCommit— rollback becomes a no-op once the transaction has been finalised.
Registering the action
In main.go:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: actions.PlaceOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth("dev-secret"), // identity → ctx.Auth
},
})
auth.JWTAuth is the same middleware registered globally on Auth in Part 2,
but action middleware runs only for the action itself — handy when the action
needs different auth from the generated routes.
Trying it
curl -X POST localhost:8080/api/orders/place \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"lines\":[{\"book_id\":\"$BOOK\",\"quantity\":2}]}"
Response:
{
"data": {
"id": "abc123…",
"customer_id": "user-alice",
"total": 25.98,
"status": "pending",
...
}
}
Re-run it until the book runs out, and the next request gets a clean
409 OUT_OF_STOCK instead of a partial write.
Finishing the “must have bought” review check
Part 4 left a stub: only customers who have bought a book may review it. The
join query needs order_lines and orders — which we now have:
server.Pipeline.Validate.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
rows, _ := ctx.RawQuery(
`SELECT 1
FROM order_lines ol
JOIN orders o ON o.id = ol.order_id
WHERE o.customer_id = ?
AND ol.book_id = ?
AND o.status IN ('paid','shipped')`,
ctx.Auth.UserID, bookID,
)
if len(rows) == 0 {
ctx.Abort(http.StatusForbidden, "PURCHASE_REQUIRED",
"you may only review books you have bought")
return nil
}
return next()
}, maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate))
Next
In Part 8 — Events & Background Jobs we build the background worker that consumes outbox rows and emails order receipts.
8. Events & Background Jobs
Part 7 added a custom action for placing orders. This part defers the post-order work — sending a receipt email — to a background job so that a slow or failing mailer never blocks the purchase response or rolls back the order transaction.
Why a background worker
Running side-effects inside the request transaction creates two problems:
- If the email fails, the whole order rolls back — a transient mail outage breaks the purchase flow entirely.
- If the order rolls back after a successful email, the email can’t be unsent.
Decoupling fixes both: the transaction writes a small job description; a worker outside the transaction carries it out. If the worker crashes mid-task the job is retried automatically.
Wiring up the job queue
The jobs/ package family provides a durable queue with retries and REST-based
status polling. For the bookstore we use jobs/sql, which enqueues inside the
same database transaction as the business write.
Install the queue alongside the server setup in main.go:
import (
"github.com/xaleel/maniflex"
jobsmaniflex "github.com/xaleel/maniflex/jobs/maniflex"
"github.com/xaleel/maniflex/jobs"
jobssql "github.com/xaleel/maniflex/jobs/sql"
)
server := maniflex.New(maniflex.Config{ /* ... */ })
server.MustRegister(Order{}, User{} /* ... */)
db, _ := sqlite.Open("./app.db", server.Registry())
server.SetDB(db)
// jobs/sql takes a database/sql handle, not the maniflex adapter; point it at the
// same database file so jobs live alongside your data. (The "sqlite" driver is
// registered by importing the db/sqlite package above.)
jobsDB, _ := sql.Open("sqlite", "./app.db")
queue := jobssql.New(jobsDB)
jobssql.Migrate(ctx, jobsDB, "sqlite") // "postgres" on PG
// Mount registers the StatusModel and returns a wrapped queue + sink.
// After this, GET /api/job_statuses/:id is available automatically.
sink, queue, err := jobsmaniflex.Mount(server, queue)
if err != nil { log.Fatal(err) }
// Wire up the worker.
w, _ := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Status: sink,
Handlers: map[string]jobs.Handler{
"send_receipt": sendReceiptHandler(mailer),
},
})
go w.Run(ctx)
log.Fatal(server.Start())
Enqueueing from the order action
Modify the order placement action from Part 7 to enqueue a job instead of calling the mailer directly:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders",
Handler: func(ctx *maniflex.ServerContext) error {
// ... validate, insert order, etc. ...
jobID, err := queue.Enqueue(ctx.Ctx, jobs.Job{
Type: "send_receipt",
ActorID: ctx.Auth.UserID,
Payload: mustJSON(map[string]any{"order_id": orderID}),
})
if err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusAccepted,
Data: map[string]any{
"order_id": orderID,
"job_id": jobID, // clients can poll /api/job_statuses/:job_id
},
}
return nil
},
})
The wrapped queue creates an enqueued status row before returning, so the
client can poll immediately — no race between enqueue and the first GET.
The handler
func sendReceiptHandler(mailer Mailer) jobs.Handler {
return func(ctx context.Context, j jobs.Job) (jobs.Result, error) {
var p struct {
OrderID string `json:"order_id"`
}
if err := json.Unmarshal(j.Payload, &p); err != nil {
return jobs.Result{}, err
}
return jobs.Result{}, mailer.SendReceipt(ctx, p.OrderID)
}
}
Handlers return (jobs.Result, error). On error the worker retries with
exponential backoff (default up to 3 attempts). After all retries the job is
marked dead and the status row records the final error message.
Polling for completion
The client receives job_id in the response and polls until done:
POST /api/orders
← 202 {"data": {"order_id": "xyz", "job_id": "01JABC..."}}
GET /api/job_statuses/01JABC...
← 200 {"data": {"status": "enqueued", ...}}
GET /api/job_statuses/01JABC... (retry after a tick)
← 200 {"data": {"status": "succeeded", "completed_at": "2025-01-15T09:01:02Z"}}
No extra endpoint or custom table — the StatusModel is wired up automatically
by Mount.
Emitting events from the pipeline
For lighter-weight fan-out — “notify other services every time an Order is
created” — the events.Emit middleware is a simpler fit than the job queue:
import (
"github.com/xaleel/maniflex/events"
"github.com/xaleel/maniflex/events/redis"
)
bus := redis.New(redisClient, "myapp") // prefix namespaces the Redis stream keys
server.Pipeline.DB.Register(
events.Emit(bus),
maniflex.ForModel("Order"),
maniflex.AtPosition(maniflex.After),
)
Emit publishes order.created (and order.updated, order.deleted) to the
bus on the DB-After step — only when the write succeeded. Subscribers in the
same or other processes consume events independently. For WebSocket fan-out to
connected clients, wire a realtime.Hub to the bus (see
Realtime / WebSockets).
Webhooks
events.Webhook delivers events to external URLs with an HMAC signature —
useful for one-off partner integrations. Unlike events.Emit, it is an event-bus
subscriber, not pipeline middleware, so wire it with bus.Subscribe:
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: events.Webhook(events.WebhookConfig{
URL: "https://partner.example.com/orders",
Secret: os.Getenv("WEBHOOK_SECRET"),
}),
})
What we built
| Capability | How |
|---|---|
| Decoupled post-order email | jobs.Queue — enqueue in action, process in worker |
| Status polling | jobs/maniflex.Mount → GET /api/job_statuses/:id |
| Automatic retries | jobs.Job.MaxRetry + exponential backoff |
| Transactional enqueue | jobs/sql inserts the job row in the same DB transaction |
| Domain event fan-out | events.Emit on DB-After → event bus subscribers |
| External webhook delivery | events.Webhook as a bus subscriber |
Next
In Part 9 — Testing the API we test the whole app end to end, including the job worker and the polling flow.
9. Testing the API
A useful test suite for a Maniflex app exercises the HTTP layer, not just the
database. The supported maniflextest module starts a real HTTP server,
migrates a fresh in-memory SQLite database, and registers all cleanup with the
test:
go get github.com/xaleel/maniflex/maniflextest
A test server
The harness takes the same models and setup function as the application. Calls
such as POST("/widgets", ...) are relative to the configured API prefix
(/api by default).
func TestHarness(t *testing.T) {
server := maniflextest.New(t, maniflextest.Options{
Models: []any{Widget{}},
Setup: func(app *maniflex.Server) {
app.Pipeline.Auth.Register(requireTestUser)
},
})
server.POST("/widgets", map[string]any{"name": "unauthenticated"}).
AssertStatus(http.StatusUnauthorized)
created := server.POST(
"/widgets",
map[string]any{"name": "first"},
maniflextest.As(maniflextest.Human("user-42", "editor")),
).AssertStatus(http.StatusCreated)
widget := maniflextest.DecodeData[Widget](created)
if widget.Name != "first" || widget.ID == "" {
t.Fatalf("unexpected widget: %+v", widget)
}
}
maniflextest.New completes the repetitive integration-test work:
- registers the supplied models and application setup;
- opens a unique in-memory SQLite database and runs
MigrateOnly; - starts an
httptest.Server; - shuts down services, drops test data, and closes the adapter automatically.
Each call to New is isolated, including separate calls in nested t.Run
tests. Use server.App() when a worker or background operation needs the
underlying *maniflex.Server.
Test principals
maniflextest.As injects a complete maniflex.AuthInfo before the
application’s Auth middleware runs:
admin := maniflextest.Human("user-42", "admin")
response := server.POST(
"/orders",
map[string]any{"book_id": bookID},
maniflextest.As(admin),
)
response.AssertStatus(http.StatusCreated)
Use ServiceAccount(id, scopes...) for machine callers. For tenant, claim, or
session-specific tests, construct maniflex.AuthInfo directly and pass it to
As.
The injected principal is only a test transport; it is not production
authentication. Set DisableTestAuth: true and use maniflextest.Bearer when
testing a real JWT or API-key middleware end to end.
Typed responses
Responses are buffered so status, headers, raw bytes, envelopes, and typed models can all be asserted:
created := server.POST("/books", input).AssertStatus(http.StatusCreated)
book := maniflextest.DecodeData[models.Book](created)
books := maniflextest.DecodeDataList[models.Book](
server.GET("/books?sort=title").AssertStatus(http.StatusOK),
)
For error paths, use response.ErrorCode():
response := server.POST("/users", map[string]any{"name": "Alice"})
response.AssertStatus(http.StatusUnprocessableEntity)
if code := response.ErrorCode(); code != "VALIDATION_ERROR" {
t.Fatalf("error code: got %q", code)
}
Fixtures and factories
Fixtures are ordinary API creates, so they pass through validation, middleware, and hooks instead of inserting a database shape the application could never produce:
fixtures := server.Seed(
maniflextest.Fixture{
Name: "alice",
Path: "/users",
Body: map[string]any{"name": "Alice", "email": "[email protected]"},
},
)
aliceID := fixtures.ID(t, "alice")
Use a factory for repeated data:
books := maniflextest.Factory(
"book",
"/books",
10,
func(i int) map[string]any {
return map[string]any{"title": fmt.Sprintf("Book %02d", i)}
},
maniflextest.As(maniflextest.Human("author-1")),
)
fixtures := server.Seed(books...)
firstID := fixtures.ID(t, "book[0]")
Names are stable (book[0], book[1], and so on), making relationships easy
to express without relying on insertion order or hard-coded IDs.
PostgreSQL-specific tests
Most application behaviour should use the fast SQLite default. When SQL dialect, locking, or transaction semantics matter, give the harness a PostgreSQL DSN:
server := maniflextest.New(t, maniflextest.Options{
Models: appModels(),
Setup: registerApplication,
Database: maniflextest.Postgres(os.Getenv("MANIFLEX_TEST_PG_DSN")),
})
Every server receives a randomly named schema. Cleanup drops the entire schema, so tests do not share rows and there is no truncation list to maintain. The database user must be allowed to create and drop schemas.
Contention and workers
For a contention test, create one harness server and send concurrent requests
through it. Do not call t.Fatal from request goroutines; collect each
response’s StatusCode and assert from the test goroutine after they join.
Set StartServices: true when the behaviour under test needs registered
services or lifecycle hooks. Prefer a one-shot worker entry point where
possible; otherwise make the worker accept a context.Context and stop it
before the test returns. Harness cleanup calls Server.Shutdown, so framework
background work is drained rather than abandoned.
See Testing Applications for the complete supported surface and customization points.
Coverage strategy
For a typical bookstore-shaped app, a useful split is:
- per model: happy-path create, read, update, and delete;
- per
mfx:rule: at least one representative rejection; - per custom middleware: one accepted and one rejected request;
- per action: a happy path, a representative failure, and any important contention case;
- per worker: one event is received, processed, and acknowledged.
The framework’s own end-to-end suite covers adapter and query permutations. Application tests should concentrate on the policies and business behaviour the application adds.
Next
In Part 10 — Deploying to Production we swap SQLite for PostgreSQL, drive configuration from environment variables, enable the health probe, and produce a single binary suitable for a container image.
10. Deploying to Production
The bookstore runs cleanly on SQLite + go run . for development. Production
needs three additional things: a real database, configuration from the
environment, and a sensible operational contract — health probes,
structured logs, graceful shutdown. Code changes are minimal; the framework
was already built for this.
Swapping SQLite for PostgreSQL
Add the satellite:
go get github.com/xaleel/maniflex/db/postgres
Change one import in main.go:
- import "github.com/xaleel/maniflex/db/sqlite"
+ import "github.com/xaleel/maniflex/db/postgres"
…and the adapter open call:
// Open(writeDSN, readDSN, registry) is positional. For pool/session tuning use
// OpenWithConfig(writeDSN, readDSN, registry, writePool, readPool, session).
db, err := postgres.OpenWithConfig(
os.Getenv("DB_WRITE_URL"),
os.Getenv("DB_READ_URL"), // optional; "" routes reads to the primary
server.Registry(),
// Both pools count against the same server. Keep
// (write + read) × processes under your max_connections; the defaults
// (3 / 6) are sized for the smallest managed tier.
postgres.PoolConfig{MaxOpenConns: 4, MaxIdleConns: 4, ConnMaxLifetime: 30 * time.Minute}, // write pool
postgres.PoolConfig{MaxOpenConns: 10, MaxIdleConns: 10, ConnMaxLifetime: 30 * time.Minute}, // read pool
postgres.SessionConfig{ApplicationName: "bookstore"},
)
Models, middleware, actions, and tests all carry over unchanged. The shared
db/sqlcore adapter means SQL emitted by AutoMigrate is portable between
the two backends. See PostgreSQL in Production for
pool tuning, read replicas, and migration choices.
Configuration from environment
A single Config populated from os.Getenv:
// config.go
func loadConfig() maniflex.Config {
return maniflex.Config{
Port: envInt("PORT", 8080),
PathPrefix: envStr("PATH_PREFIX", "/api"),
ServiceName: envStr("SERVICE_NAME", "bookstore"),
DisableAutoMigrate: envStr("AUTO_MIGRATE", "true") != "true",
QueryTimeout: envDuration("QUERY_TIMEOUT", 30*time.Second),
ShutdownTimeout: envDuration("SHUTDOWN_TIMEOUT", 30*time.Second),
HealthCheckDB: true,
HealthTimeout: 3 * time.Second,
Logger: slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
})),
}
}
maniflex also ships a helper, maniflex.ConfigFromEnv(prefix), that reads a
conventional set of environment variables (PORT, DB_WRITE_URL,
QUERY_TIMEOUT_MS, … — see Configuration).
It returns an error if one of them is set to something it cannot read, so a
mistyped PORT stops the deploy instead of booting on 8080. Pick whichever style
fits your team — both produce the same maniflex.Config.
Production-safe migrations
Migration runs by default (convenient in development). In production, the prevailing pattern is:
- Disable
AutoMigrateon every instance. - Run schema changes through a dedicated migration tool
(
golang-migrate, Atlas, sqlc-migrate) executed as a separate one-shot step in your deploy pipeline. - Roll out the new application code afterwards.
AUTO_MIGRATE=false
The framework’s auto-migrator never drops columns, but it isn’t aware of your release strategy — splitting “deploy” and “migrate” into two steps lets you stage them deliberately.
Health probes
Two endpoints, two questions. GET /api/live asks whether the process is
alive and touches nothing to answer. GET /api/ready asks whether it should
receive traffic: it pings the database, runs any Config.ReadinessChecks, and
answers 503 while the server is still starting or already draining.
livenessProbe:
httpGet:
path: /api/live
port: 8080
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /api/ready
port: 8080
periodSeconds: 5
timeoutSeconds: 5
Do not point livenessProbe at /api/ready. A database outage would then
restart every replica — none of which can fix the database by dying — and the
restarts arrive precisely when the dependency is least able to take a
reconnection storm.
Set HealthTimeout (default 3s) shorter than the probe’s timeoutSeconds
so the handler can return a clean 503 before the probe gives up.
GET /api/health still exists and still follows HealthCheckDB. It is a
compatibility alias for deployments written before the split; new ones should
use /api/live and /api/ready.
terminationGracePeriodSeconds on the pod should be longer than
Config.ShutdownTimeout, otherwise Kubernetes will SIGKILL the process
before in-flight requests have finished. With the defaults (30s shutdown),
60s grace is a comfortable buffer.
Logging and tracing
The JSON handler above turns every line into a structured record that an
aggregator can index. ctx.Logger() automatically adds request_id and
trace_id per request, so a single trace can be reconstructed end-to-end.
Set Config.ServiceName so every log line and every audit record carries
the service identifier — invaluable when a single aggregator collects logs
from several services.
For a debugging spike, enable pipeline tracing:
TRACE=1
if envStr("TRACE", "") != "" {
cfg.Trace = maniflex.PipelineTrace{Enabled: true, Skips: true}
}
Steps, Timings, and Aborts produce DEBUG-level records that show
every middleware enter/exit and the file:line of every Abort call.
Disable in normal operation — they are high-volume.
The Dockerfile
A typical Dockerfile for the binary:
FROM golang:1.25-alpine AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -o /out/bookstore ./
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/bookstore /bookstore
COPY static /static
EXPOSE 8080
USER 65532:65532
ENTRYPOINT ["/bookstore"]
CGO_ENABLED=0 works because maniflex/db/postgres uses lib/pq (pure Go) and
nothing else in the framework requires a C toolchain.
static/ is copied so the Scalar OpenAPI viewer is reachable at
/static/openapi.html — provided the app opts into static serving with
maniflex.Config{StaticDir: "static"} (serving is opt-in; an unset StaticDir
publishes nothing).
Production checklist
| Setting | |
|---|---|
| Database | maniflex/db/postgres with WriteURL and optional ReadURL |
| Migrations | DisableAutoMigrate: true + external migration tool |
| Logger | JSON handler |
Config.ServiceName | the service name |
Config.QueryTimeout | bounded (e.g. 30s) |
Config.ShutdownTimeout | matches the slowest legitimate request |
| Probes | livenessProbe → /api/live, readinessProbe → /api/ready |
K8s terminationGracePeriodSeconds | larger than ShutdownTimeout |
| TLS | terminated at the load balancer |
| Auth | auth.JWTAuth with an asymmetric key from your IdP |
| Rate limits | db.RateLimit on password-reset / sign-up / login |
| Audit log | db.AuditLog on OpCreate / OpUpdate / OpDelete |
| File storage | swap LocalStorage for S3 / R2 / GCS |
| Outbox worker | run alongside the API, or as a separate deployment |
After every model, middleware, action, and optional route is registered, run
server.ValidateProduction() before Start(). It checks the bounded settings
above and refuses any mounted data route without an explicit protected/public
access decision. See Production Validation.
Where to go from here
The tutorial finishes here. From this point, the reference pages cover everything in more depth, and the code base is small enough to grow in any direction:
- More middleware from the Middleware Catalogue.
- Customisation via Writing Middleware.
- Advanced workflows in Custom Endpoints, Raw Queries & Query Models, and Batch Operations & Sagas.
- Hardening with Auth & Security Hardening.
The shape of main.go has not changed in ten parts. Add models, add
middleware, add actions — the wiring is the same.
Stability & Compatibility
This page is the contract behind the version number: what will not change while maniflex is v1, what may change, and what happens to something once it is deprecated.
It takes effect at v1.0.0. While the project is v0.x, any minor release may
break anything. Nothing below applies retroactively to v0.x.
maniflex has two distinct sets of consumers, and they depend on different things:
| Contract | Who depends on it | Where it lives |
|---|---|---|
| Go API | applications importing github.com/xaleel/maniflex/... | exported identifiers |
| HTTP contract | clients calling the generated API | routes, query parameters, envelopes, error codes |
Both are covered. The HTTP contract matters most in practice — a mobile app or a generated SDK talks to the API, not to the Go types — and it is the one a Go compatibility promise alone would leave unaddressed.
Which modules are covered
The core module and the fourteen released alongside it:
github.com/xaleel/maniflex, admin, db/postgres, db/sqlite,
events/kafka, events/nats, events/rabbitmq, events/redis, jobs/redis,
maniflextest, middleware/auth/redis, middleware/db/redis,
middleware/service/bcrypt, pkg/otel, storage/s3.
Everything below is scoped to those. examples and tests are directories in
the repository rather than modules to import, and are not published.
The Go API contract
Within v1.x, in a covered module:
- Exported identifiers keep their names, kinds, and signatures.
- Exported struct fields keep their names, types, and meanings.
- Zero values keep their behaviour. A
Configfield that means “unlimited” when unset does not quietly start meaning “disabled”. Most maniflex configuration is a zero-value default, so this is load-bearing. - Sentinel errors (
ErrNotFound,ErrIncrementOutOfBounds, and the rest) keep their identity and stayerrors.Is-comparable.
New exported API may be added in any minor. That has one consequence you must plan for:
Use keyed struct literals.
maniflex.Config{Port: 8080}, nevermaniflex.Config{8080, ...}. Adding a field to an exported struct is an additive change under this policy and it breaks unkeyed literals. Unkeyed literals of maniflex structs are not supported.
Not covered
- Anything under
internal/, which the Go toolchain already prevents you from importing. - The text of error messages.
codein the error envelope and the sentinel error values are the contract; the human-readable string is not. Do not match on message prose. - Log output — format, wording, levels, and which events are logged at all.
- Metric and span names emitted by
pkg/otel, until a future release freezes them explicitly. - Struct field order, which matters only to unkeyed literals, already excluded above.
- Unsafe or reflective access to unexported state.
Extension interfaces are closed
DBAdapter, Tx, FileStorage, CacheStore, KeyProvider, the events and
jobs broker and queue interfaces, auth.Revoker, the Locker interfaces, and
their siblings are covered for callers: if you accept or call one, its
existing methods will not change.
They are closed to outside implementation. Methods may be added to them in a minor release, which breaks any type implementing the interface from outside this repository.
This is a deliberate trade. maniflex is built around pluggable backends, and a
fully frozen DBAdapter would mean no adapter could gain a capability until v2 —
so the interface that exists to enable extension would be the one thing blocking
it. If you maintain an out-of-tree adapter, pin the minor version and expect to
add methods when you upgrade; opening an issue about it is the fastest way to get
a compatibility shim considered.
Deprecation and removal
Nothing exported is removed during v1.x.
A Go module’s import path encodes its major version, so removing an exported identifier requires a /v2 path — there
is no mechanism for “removed in v1.4” that does not break every importer. A
policy promising removal after some number of minor releases would be a promise
Go will not let this project keep.
So a deprecation is a signal, not a countdown:
- The identifier gains a
// Deprecated:godoc comment naming its replacement, or stating plainly that it has none. Editors and linters surface these. - The CHANGELOG entry records it.
- It keeps working, unchanged, for all of v1.x. A deprecated field does not become a silent no-op — that is a behaviour change, and behaviour changes are breaking whether or not the symbol survives.
- It is removed in v2.0.0, at a new module path, alongside a migration guide.
The practical cost lands on new API rather than old: anything exported in v1 is permanent, so it is worth being slower to export. Prefer an unexported type with an exported constructor, and prefer a concrete struct over an interface, unless there is a reason to widen.
Breaking changes are marked in the CHANGELOG as **(breaking)** on the relevant
category, and a behaviour change that keeps every signature intact is marked
**(behaviour change)** — the more dangerous of the two, because it compiles.
The HTTP contract
Within v1.x, for the routes maniflex generates:
- Route shapes for every generated operation — collection, item, and the
mounted sub-resources (
/export,/aggregate,/{field}/upload-url,/{id}/restore,/{id}/history,/{id}/{field}) — keep their paths and methods, under whateverPathPrefixyou configure. - Query parameters keep their names and semantics:
filter(and itsfilter[N]group form),sort,page,limit,cursor,count,include,select,format,aggregate, andq/modelson global search. See Querying. - Envelopes keep their shape:
{"data": ...}on success,{"meta": ...}on lists,{"error": {"code", "message", "details"}}on failure, anddetailsis an array wherever it appears. See Response Envelope. - Error codes are permanent. A code is never removed, and never repurposed to mean something else.
- Status codes for each generated operation stay as documented.
- The DDL contract in Field Schema & Nullability and the identity contract in Record Identity keep their existing “out of contract for v1” carve-outs.
What may still change, and what your client must tolerate
Minor releases may add. Specifically, they may add response fields, query parameters, error codes, routes, and headers.
That is only safe if clients are written for it, so it is part of the contract in the other direction — the API is additive, and a conforming client must:
- ignore JSON fields it does not recognise, rather than rejecting the response;
- treat an unrecognised
error.codeas a generic failure of its HTTP status class, rather than crashing; - not depend on JSON key ordering.
A strict-by-default generated SDK, or a deserialiser configured to error on unknown fields, will break on a minor release. That is the client’s configuration, not a breach of this policy.
Also outside the contract: the exact prose of error.message, the byte-level
output of the OpenAPI document (its semantic content is covered), and the
admin panel’s HTML, which is a UI rather than an API.
Your API’s own evolution is yours
This page covers what maniflex generates, not what your application exposes. If you rename a model field, maniflex will faithfully rename the JSON key and break your clients — the framework has no field-deprecation or API-versioning mechanism of its own, and shipping one is not planned for v1.
Until it exists, the tools are ordinary ones: keep the old column and populate
both during a transition, add a computed field with
Server.AddComputedField to serve the old name from the new data, or mount a
versioned PathPrefix per API generation.
Minimum versions
Go: 1.25.12. Every module declares it, and CI builds each release against both that exact version and current stable. The minimum may be raised in a minor release — this is the convention across the Go ecosystem, including the standard library’s own support window — but never in a patch release.
The declaration is patch-precise (go 1.25.12, not go 1.25), so a consumer on
an earlier 1.25 patch cannot build maniflex at all. That is stricter than most Go
libraries and is worth knowing before you pin a toolchain.
Databases: what CI proves. Continuous integration runs the full end-to-end
suite against PostgreSQL 17 and against modernc.org/sqlite, the pure-Go driver
used by db/sqlite. Other PostgreSQL versions are widely expected to work — the
adapter uses lib/pq and no version-gated syntax — but they are not tested here,
so this page does not promise them.
Redis: 7.0. Four modules speak Redis, and the supported floor is the highest
requirement among them. middleware/db/redis sets it: its rate-limit counter
pins a new key’s TTL with EXPIRE … NX, so that a failure between the increment
and the expiry cannot leave a counter with no TTL — one that never resets, and
therefore bars a client for ever. The NX argument arrived in Redis 7.0.
The others ask for less. events/redis and jobs/redis reclaim abandoned
messages with XAUTOCLAIM, which is 6.2, and middleware/auth/redis uses only
GET and SET. A deployment that leaves out middleware/db/redis will work on
6.2 — but that is a configuration this page does not promise, for the same
reason the PostgreSQL versions above are not promised.
Nothing here is proven by CI, which runs no job against a real Redis at all. This floor is derived from the commands the modules issue, not observed from a suite, so read it as the version the code requires rather than one it is known to run on. Like the Go minimum, it may be raised in a minor release.
Security support
Security fixes land on the current minor release first. Once v2.0.0 ships, the
final v1 minor continues to receive security fixes for 12 months. Everything
else is fixed forward.
Report vulnerabilities privately — see SECURITY.md — not as public issues.
Getting to v1.0.0
The stable tag is preceded by at least one release candidate (v1.0.0-rc.1,
rc.2, …). Release candidates carry no compatibility promise; their purpose is
to find the breaks while breaking is still free.
Covered modules are released in lockstep at a single version, so v1.2.0 of
core and v1.2.0 of db/postgres are always built and tested together. Mixing
versions across covered modules is unsupported even though Go permits it.
Enforcement
| Guarantee | Enforced by |
|---|---|
| Documented Go examples compile | doccheck, over every fence in docs/src, plus anchored includes of real files |
| The README quickstart compiles | examples/quickstart plus TestREADMEQuickstartMatchesCompiledExample |
| Every mounted route is in the OpenAPI spec | TestOpenAPIRouteParity |
| The DDL contract holds | tests/e2e/schema_contract_test.go |
| Modules build from a clean consumer environment | release smoke tests in CI |
doccheck parses Go fences and cannot type-check them, so a fence naming a field
that no longer exists still passes — only compiled examples catch that. This is
why important examples live in .go files.
Testing Applications
github.com/xaleel/maniflex/maniflextest is the supported integration-test
harness for Maniflex applications. It is a separate module because its SQLite
and PostgreSQL drivers should not become dependencies of the core framework.
Starting a server
server := maniflextest.New(t, maniflextest.Options{
Config: maniflex.Config{PathPrefix: "/v1"},
Models: []any{
User{},
maniflex.ModelConfig{TableName: "app_users"},
Order{},
},
Setup: func(app *maniflex.Server) {
registerMiddleware(app)
registerActions(app)
},
})
Setup runs after model registration and database injection but before route
validation and migration. Config.DB must remain nil; select the test adapter
with Options.Database.
By default, New:
- opens a unique shared in-memory SQLite database;
- installs support for request-scoped test principals;
- invokes
Setup; - validates and migrates the application;
- starts an
httptest.Server; - registers HTTP, lifecycle, database, and fixture cleanup with
t.Cleanup.
Options.StartServices additionally starts registered services and lifecycle
hooks. server.App() exposes the application for background contexts and
direct assertions.
Requests
GET, POST, PUT, PATCH, DELETE, and Do resolve paths beneath
Config.PathPrefix. DoRoot targets a root-mounted endpoint such as a
standalone file route.
Request bodies may be nil, []byte, an io.Reader, or any JSON-encodable
value. Request options include:
maniflextest.Header("If-Match", etag)
maniflextest.Bearer(token)
maniflextest.As(principal)
A response exposes StatusCode, Header, and Body, plus AssertStatus,
Decode, JSON, Data, DataList, ID, and ErrorCode. Use
DecodeData[T] and DecodeDataList[T] to keep response assertions typed.
Authentication
Human(id, roles...) and ServiceAccount(id, scopes...) construct common
principals. As carries one into the Auth pipeline, where the application’s
authorization middleware sees it as ctx.Auth.
To test production authentication itself, set DisableTestAuth: true and send
the real credential with Bearer or Header.
Fixtures
Server.Seed creates named fixtures through HTTP:
records := server.Seed(
maniflextest.Fixture{Name: "free", Path: "/plans", Body: freePlan},
maniflextest.Fixture{Name: "pro", Path: "/plans", Body: proPlan},
)
proID := records.ID(t, "pro")
Factory builds repeated fixtures from an index. Seed preserves input order,
rejects duplicate names, requires every create to return 201, and stores each
response’s data object under its name.
Time
Behaviour that turns on the passage of time — an idempotency replay window, a rate-limit window, a cached read — used to be untestable from the outside: every path to it measured against the wall clock, so asserting that a one-hour window closes meant waiting an hour.
maniflextest.NewClock returns a clock that only moves when you say so.
clock := maniflextest.NewClock(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC))
clock.Advance(90 * time.Minute) // or clock.Set(someInstant)
Hand its Now method to whatever measures the time. The method value satisfies
maniflex.Clock and stays bound to the clock, so it can be passed anywhere one
is accepted:
| Where | What it governs |
|---|---|
maniflex.WithCacheClock(clock.Now) on NewMemoryCache | idempotency replay windows, db.CacheQuery response caches, anything else backed by that CacheStore |
db.RateLimitConfig.Clock | the in-process rate-limit window (no effect when Backend is set — there the window belongs to the backend) |
scheduled.Config.Clock | when the scheduled runner considers a field’s instant to have arrived |
A nil clock anywhere is the wall clock, so nothing changes for code that does not set one.
The whole shape, from a test that runs in CI:
func TestIdempotencyWindowClosesOnceTheClockPasses(t *testing.T) {
clock := maniflextest.NewClock(time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC))
cache := maniflex.NewMemoryCache(maniflex.WithCacheClock(clock.Now))
server := maniflextest.New(t, maniflextest.Options{
Models: []any{Widget{}},
Setup: func(app *maniflex.Server) {
app.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{Store: cache, TTL: time.Hour}),
maniflex.AtPosition(maniflex.After),
maniflex.ForOperation(maniflex.OpCreate),
)
},
})
key := maniflextest.Header("Idempotency-Key", "order-1")
body := map[string]any{"name": "widget"}
server.POST("/widgets", body, key).AssertStatus(http.StatusCreated)
server.POST("/widgets", body, key).AssertStatus(http.StatusCreated) // replayed
assertWidgetCount(t, server, 1)
clock.Advance(2 * time.Hour) // the replay window closes
server.POST("/widgets", body, key).AssertStatus(http.StatusCreated)
assertWidgetCount(t, server, 2)
}
What it does not reach. created_at and updated_at are stamped by the
database adapter, and job scheduling — NotBefore, lease expiry, cron ticks —
runs on its own wall clock. Neither takes an injected clock, so a test that
depends on either still has to work around it.
Databases
SQLite()is the default and creates a distinct in-memory database.SQLite(path)uses a file-backed database when persistence behaviour matters.Postgres(dsn)creates and later drops an isolated random schema.- A custom
DatabaseFactorycan return anymaniflex.DBAdapterplus an optional cleanup callback.
The harness owns adapters returned by its database factory and closes them after cleanup. Do not share one adapter between concurrently running harness servers.
Middleware order
Options.RecordPipeline records which middleware runs for each request, and
Server.PipelineSteps reports it in execution order:
server := maniflextest.New(t, maniflextest.Options{
Models: []any{Order{}},
RecordPipeline: true,
Setup: func(app *maniflex.Server) {
app.Pipeline.Auth.Register(tenantGuard, maniflex.WithName("tenant-guard"))
},
})
server.GET("/orders")
// [Auth/tenant-guard Auth/default Deserialize/default … DB/default Response/default]
steps := server.PipelineSteps()
Each entry is Step/middleware. A middleware registered without
WithName is reported
as [unnamed], and a step’s built-in handler as default.
The framework reports order by logging one record per middleware, which the
harness captures. Matching those log records yourself is the thing to avoid: a
log message is not part of the API, so a test that greps for one can break on a
patch release. Recording wraps Config.Logger rather than replacing it, so a
logger you configured keeps receiving everything.
The report covers the most recent request. Requests issued concurrently against one server share the recording, so assert on one at a time, or give each its own server.
Production Validation
Development defaults favor a short feedback loop: generated routes are public
until Auth middleware is registered, QueryTimeout is unlimited, and
AutoMigrate is enabled. Before deployment, audit the fully assembled server:
cfg := maniflex.Config{
Strict: true,
DisableAutoMigrate: true,
QueryTimeout: 30 * time.Second,
}
server := maniflex.New(cfg)
server.MustRegister(User{}, Order{})
server.Pipeline.Auth.Register(auth.JWTAuth(publicKey, auth.JWTOptions{}))
if err := server.ValidateProduction(); err != nil {
log.Fatal(err)
}
log.Fatal(server.Start())
Call ValidateProduction after registering models, middleware, actions, global
search, and documentation, but before Start or Handler. It reports every
problem together and changes no runtime behavior.
What it requires
Config.Strictis enabled.Config.QueryTimeoutis positive.Config.MaxConcurrentRequestsis positive, so a burst is refused rather than queued on the database pool.- Every global and per-model effective
QueryLimitsfield remains positively bounded, as does global search’sMaxLimitwhen search is mounted. - AutoMigrate is disabled when models are registered.
- Every generated model operation has matching
Pipeline.Authmiddleware or an explicit public declaration. - Standalone files, custom actions, and global search each have a protected or explicitly public access decision.
It also runs every registry check Start runs, so one call reports the whole
startup posture rather than only the part specific to production:
- Encrypted
uniquefields have a blind-index key, so their uniqueness digests can still be re-derived after a key rotation. - Fields tagged
file_acl:signedhave aFileStoragethat can mint a time-limited URL, rather than degrading to a permanent one. - Every model field has a Go type the OpenAPI generator can describe.
- Relations,
lock_scopedeclarations, andPipelinemiddleware wiring resolve.
Several of those are findings Config.Strict turns fatal, and production
validation requires Strict, so they surface here rather than at the first boot.
Framework outbound calls made through integration.Caller already have bounded
timeout, retry, and response-size defaults. The validator cannot inspect
arbitrary http.Client instances created by application code.
Proxy-header resolution remains off by default, which is safe. TrustedProxies
names the peers whose forwarding headers may be believed, and the validator can
check that the entries parse but not that they describe your actual topology.
TrustProxyHeaders: true with no TrustedProxies is the legacy allowlist-free
mode — the explicit assertion that the service sits behind a proxy which replaces
client-supplied forwarding headers itself. It warns at startup and, because
Config.Strict is required for production validation to pass, fails there.
Declaring public model operations
Server.AllowPublic marks only the scopes you name:
// Public sign-up; every other User operation still needs Auth coverage.
server.AllowPublic(
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate),
)
This is a validation declaration, not middleware. It does not make a protected route public or alter request handling.
Other route types
Standalone files use FilesConfig.BeforeMiddlewares; set
FilesConfig.AllowPublic only when /files is deliberately public.
Custom actions normally inherit matching Pipeline.Auth middleware. If access
is enforced inside ActionConfig.Middleware or the handler, set
ActionConfig.AccessControlled. Set ActionConfig.AllowPublic for an
intentionally public action.
Global search likewise uses Pipeline.Auth for OpSearch, or an explicit
GlobalSearchConfig.AllowPublic.
If a router-level middleware protects every route before dispatch, set
Config.HTTPAccessControlled alongside non-empty Config.HTTPMiddlewares.
This flag is an assertion and does not install authentication.
Generated documentation is already explicit: its zero value mounts nothing,
Documentation.Middleware protects it, and Documentation.Public deliberately
publishes it. Static serving requires an explicit non-empty StaticDir.
The probe endpoints — /live, /ready, and /health — are public by default
and the sweep exempts them, because an orchestrator’s probe is the canonical
unauthenticated request. Config.Probes gates or unmounts them when that is not
what you want; setting it does not change what ValidateProduction asks for.
Startup Validation & Strict Mode
maniflex validates your configuration once, at startup, and reports everything
wrong with it in a single message. Most problems are fatal on their own;
Config.Strict adds the handful that are merely suspect.
The principle: a directive that parses but cannot be honoured is an error, not a silent no-op. A misspelt tag that is quietly ignored leaves the protection you wrote absent, and nothing at runtime says so.
Always fatal
These fail whether or not strict mode is on. Each one used to warn and then discard what you asked for.
| Problem | Why it cannot be a warning |
|---|---|
A ModelConfig at argument position 0 | It has no model to attach to, so the whole config was dropped — a ModelConfig{Headless: true} silently did not apply and the model mounted routes you thought were suppressed. |
Two ModelConfigs in a row | The second has no fresh model to bind to; same silent drop. |
An invalid mfx:"scheduled" tag | The field was dropped from the sweep, so the scheduled transition you configured simply never ran. |
mfx:"relation" on a field not ending in ID | The target model is derived by stripping that suffix. Without one it is inferred from the whole field name — almost never a real model. Write mfx:"relation:Target". |
| A middleware registered on a step its operations never reach | It is frozen into the chain and never runs. When it is an authorisation check, that is a silent hole. |
A RequiresField declaration no model can satisfy | The gate watches for a field name nothing has — see below. |
The first three are registration errors, returned from Register (so
MustRegister panics). The last two need the complete registry — a relation’s
target may be registered after the model pointing at it — so they are raised
when the router is built.
Declaring the fields a middleware gates
A middleware that gates a field by name has a nasty failure mode: misspell the name and the gate watches for a body key nothing sends, while the real field keeps its name and nothing gates it. It fails open, silently.
Nothing at runtime can detect this. From inside a request, “this model has no such field” looks identical to a gate deliberately registered across models where only some carry it. Only the registration knows which it is — so that is where you say so:
server.Pipeline.Validate.Register(
validate.RestrictField("document_quota_bytes", isSuperuser),
maniflex.ForModel("User"),
maniflex.RequiresField("document_quota_bytes"), // ← checked at startup
)
Writing the name twice is not the tautology it looks like: the declared name is checked against the model, not against the middleware’s argument, so a misspelling is caught either way.
The ForModel scope makes the check exact:
- With
ForModel— every named model must have every declared field. The gate was aimed at those models specifically, so a model that lacks the field is a mistake. - Without it — at least one registered model must have the field. A gate no model can trigger cannot be doing anything.
Use it for any middleware that gates or reads a field by name, not just
validate.RestrictField — response.RedactField and hand-written gates have
the same failure mode.
It is opt-in: a registration that declares nothing is not checked, so existing
code is unaffected. validate.RestrictField keeps its first-request warning as
a fallback for undeclared registrations, but that warning cannot fire for an
endpoint nobody exercises, and cannot tell a typo from a deliberate broad
registration. Declare the field.
Gated by Config.Strict
maniflex.New(maniflex.Config{Strict: true})
These stay warnings by default because each has a legitimate reading:
| Problem | The legitimate reading |
|---|---|
mfx:"relation" whose target model is not registered | The field may be a plain foreign id that wants no relation tag. The FK column works either way. |
The standalone /files endpoints mounted with no auth middleware | A deliberately public upload endpoint is conceivable, if rarely wise. |
Config.StaticDir names a directory that does not exist | Static serving degrades to 404s. Failing the boot would let a missing frontend asset bundle take down a working API. |
Config.TrustProxyHeaders set with no Config.TrustedProxies | The service may genuinely sit behind a proxy that replaces client-supplied forwarding headers itself. |
Encrypted unique fields with no blind-index key on the KeyProvider | It is the documented legacy behaviour, and an application that never rotates its encryption keys never pays for it. |
mfx:"file_acl:signed" against a FileStorage that cannot sign | LocalStorage in development is the common case, where a permanent path costs nothing. |
| A model field whose Go type the OpenAPI generator cannot describe | The field still serialises correctly; it is the published contract that is weaker, so a generated client loses the field’s type rather than the server losing the field. |
One proxy case is on neither list: a Config.TrustedProxies entry that does
not parse fails the boot whether or not strict mode is on. A malformed CIDR is
unambiguously a typo, and dropping it silently would narrow what is trusted —
failing open on exactly the requests the entry was written to cover.
Turn it on in CI and staging, where a boot failure costs a re-run rather than an outage. Leave it off in production if you would rather serve a degraded API than none.
One report, not one restart at a time
Every problem found is listed together:
maniflex: 3 startup problems:
1. [relation] Invoice.Owner is tagged mfx:"relation" but its name does not
end in "ID", so the target model was inferred from the whole field name
as "Owner" — write mfx:"relation:Target" to name the target explicitly
2. [middleware] middleware "audit" is registered on the db step for
operation(s) [search], all of which skip that step — it will never run;
register it on a step those operations run, or widen ForOperation
3. [static] Config.StaticDir "./pubic" does not exist, so nothing would be
served under /static (Config.Strict)
Issues that are only fatal because strict mode is on are marked (Config.Strict),
so you never hunt for a bug in configuration that is legal by default.
When validation runs
Startup validation happens before migrations and before services start:
Start()
├── validate + build router ← fails here
├── migrate
├── start services
└── open listener
A configuration error therefore costs nothing to find — it cannot leave a half-migrated schema behind.
Start and StartWithContext return the error. Handler() panics,
because it has no error return; if you mount maniflex inside another router,
that is the form you will see.
Migration failures
Startup validation checks configuration. AutoMigrate separately refuses to
proceed when it cannot build a unique index — the model declares a
constraint and the database would not be enforcing it, so every write that
should have been refused would be accepted silently. This is not gated by
Config.Strict: a constraint that does not exist is wrong in every environment.
The usual cause is data that already violates it, and the error says so, naming the table, the index and the columns. De-duplicate and start again.
A failed plain index stays a warning: it costs a table scan, not a guarantee.
What stays a warning
Some warnings describe a legitimate operational state and are not promoted even under strict mode:
- A column in the database that is not on the model. Normal during a staged field removal; failing here would break rolling deploys.
- A column whose type differs from the model’s. Promoting this would brick startup against an already-drifted production database.
- Two scheduled fields targeting the same column where the later omits
from=. A real hazard, but a legal design.
Glossary
Every framework term used in these docs, in one place. Links point to the page where each term is defined in depth.
A
Action. A custom endpoint registered with server.Action(...). Runs a
trimmed pipeline (Auth → action middleware → handler → Response); the
Deserialize, Validate, Service, and DB steps are skipped. See
Custom Endpoints.
Adapter. An implementation of the maniflex.DBAdapter interface. Two ship:
db/sqlite and db/postgres. Custom backends implement the same
interface. See Database Backends.
After (position). A middleware position that places the function after
the step’s default handler. The middleware sees the result of the default
(ctx.DBResult, ctx.Response). See Writing Middleware.
AuditRecord. The structured record produced by db.AuditLog — fields
include Timestamp, Model, Operation, ResourceID, Actor,
TenantID, RequestID, TraceID, ServiceName, and optionally a
per-field Changes diff. See Audit Logging.
AuthInfo. The struct populated by Auth middleware and stored on
ctx.Auth. Carries UserID, Roles, Claims, TenantID,
IdentityType, Scopes, SessionID, AuthMethod. See
ServerContext.
AutoMigrate. The startup phase that creates and alters tables to match
registered models. Runs by default; disable with Config.DisableAutoMigrate. See
Database Backends.
B
BaseModel. The struct every registered model must embed. Contributes
id (UUID, framework-assigned), created_at, and updated_at. See
Models & BaseModel.
Batch. Multiple inserts or updates inside a single request, usually through a custom action that opens one transaction. See Batch Operations & Sagas.
Before (position). The default middleware position; the function runs before the step’s default handler. See Writing Middleware.
BelongsTo. A relation where this model carries the foreign key. Declared
either by convention (UserID field → User) or explicitly with
mfx:"relation:Name" plus a companion field of the target type. See
Relations.
C
Cache (CacheStore). The generic key/value cache interface used by
several middlewares (idempotency, rate limit, response cache). maniflex
ships MemoryCache; satellite modules provide Redis-backed
implementations. See cache.go.
Catalogue. The set of ready-made middleware shipping under
maniflex/middleware/. See Middleware Catalogue.
ServerContext. The single per-request struct threaded through every
pipeline step. Fields include Request, Writer, Ctx, Model,
Operation, ResourceID, RawBody, ParsedBody, Query, DBResult,
Response, Auth, Tx. See ServerContext.
Companion field. On an explicit BelongsTo relation, the struct field
of the target type that accompanies the FK column. Required by
mfx:"relation:Name"; the field is named Name and typed as the target
model. See Relations.
Config (maniflex.Config). The single struct passed to maniflex.New. Every
field has a sensible default. See Configuration.
D
DBAdapter. The interface implemented by every database backend.
Methods: FindByID, FindMany, Create, Update, Delete,
FindByIDForUpdate, AutoMigrate, BeginTx, Raw, Close. Ping is
not on DBAdapter — it is the separate, optional Pinger interface.
See Database Backends.
Diff (versioning). The per-field {old, new} map written into the
diff column of a history row. Hidden, writeonly, and encrypted fields
are excluded. See Versioning & History.
E
Embed. A BaseModel, WithDeletedAt, or WithIsDeleted value
included in a model struct as an anonymous field. Embeds contribute
columns and turn on framework behaviour. See Models & BaseModel
and Soft Delete.
Envelope (storage). The binary blob produced by KeyProvider.Encrypt,
stored in the database as enc:<base64(envelope)>. The envelope embeds
its keyID so Decrypt can route to the right key. See
Encryption at Rest.
Envelope (response). The JSON shape {"data": ...} (success) or
{"error": {...}} (failure). Customisable via response.Envelope. See
Response Envelope.
F
FieldMeta. The framework’s per-field metadata, derived from a struct
field’s json / db / mfx tags by parseFieldTags. Stored on
ModelMeta.Fields. See Models & BaseModel.
FileStorage. The interface for file backends. maniflex/storage ships
LocalStorage; custom implementations cover S3, R2, GCS. See
File Fields & Uploads.
Filter / FilterExpr. A parsed ?filter= expression. A slice of
these on ctx.Query.Filters becomes the SQL WHERE clause. See
Querying.
G
Generate (OpenAPI step). The middle step of the OpenAPI pipeline that
builds the *OpenAPISpec from the registry. After-position middleware
customises the spec. See OpenAPI Spec.
H
Handler (action). The func(ctx *maniflex.ServerContext) error registered
with server.Action(...) as an action’s body. See
Custom Endpoints.
HasMany. A relation declared as a slice field of the related struct. No column on this table; the related table carries the FK. See Relations.
HMAC column. A {field}_hmac companion column auto-created for
mfx:"encrypted,unique" fields, storing a keyed digest so uniqueness can
be enforced without comparing ciphertexts. See
Encryption at Rest.
History table. The {model}_history sibling table created for a
maniflex.ModelConfig{Versioned: true} model. Receives one row per write to
the source model. See Versioning & History.
I
Idempotency-Key. The HTTP header consumed by
middleware/idempotency to deduplicate retries. The first request runs;
subsequent requests with the same key replay the cached response. See
Idempotency.
IdentityType. The AuthInfo field classifying the principal — Human,
ServiceAccount, or Anonymous. See ServerContext.
Immutable (tag). A mfx: directive that accepts a value on create but
strips it on update. See Field Tags Reference.
Include. The ?include=relationKey,... query parameter that populates
related rows inline in the response. See Relations and
Querying.
Index (IndexSpec). A declared database index, created during
AutoMigrate. Declared in ModelConfig.Indices or auto-generated for
mfx:"scheduled" columns. See model.go.
J
Junction (model). The third model in a many-to-many relation, carrying
the two FKs. Named via mfx:"through:JunctionModel" on both sides. See
Relations.
K
KeyProvider. The interface that the encryption subsystem uses to
encrypt, decrypt, and HMAC field values. Implementations: EnvKeyProvider,
VaultKeyProvider. See Encryption at Rest.
L
LockForUpdate. A *ServerContext method that acquires a row-level
write lock inside an active transaction. SELECT ... FOR UPDATE on
Postgres; transaction-level lock on SQLite. See Transactions.
M
MiddlewareFunc. The signature every pipeline middleware must
satisfy: func(ctx *maniflex.ServerContext, next func() error) error. See
Writing Middleware.
ModelAccessor. The CRUD helper returned by ctx.GetModel(name).
Exposes List, Read, Create, Update, Delete for any registered
model, routed through ctx.Tx when set. See ServerContext.
ModelConfig. The per-model options passed alongside a struct in
MustRegister. Fields: TableName, SoftDelete, Middleware,
Versioned, VersionedDiffOnly, Indices. See Models & BaseModel.
ModelMeta. The framework’s runtime description of a registered
model. Carries Name, TableName, Fields, Relations, SoftDelete,
Config, Indices, and resolved scheduled specs.
O
OnDelete. The referential action attached to a foreign key —
cascade, setNull, restrict, or unset. See Relations.
Operation (maniflex.Operation). The CRUD verb identifying a request:
OpList, OpRead, OpCreate, OpUpdate, OpDelete, OpOptions, OpAction.
A HEAD request runs as the GET it mirrors (OpRead / OpList). See
Pipeline Overview.
Outbox. The transactional outbox pattern: a row written in the same transaction as the primary write, consumed by a background worker for external side effects (emails, webhooks, events). See Batch Operations & Sagas.
P
Pipeline. The six-step request pipeline (Auth → Deserialize → Validate → Service → DB → Response) and its sibling OpenAPI pipeline.
See Pipeline Overview.
Position. Where in a step’s chain a middleware sits — Before
(default), After, or Replace. See Writing Middleware.
Q
QueryParams. The parsed ?page=&limit=&filter=&sort=&include= of
a request, stored on ctx.Query. See Querying.
R
Registry. The in-memory map of every registered *ModelMeta. Built
by MustRegister, consumed by the adapter and the router. See
Architecture.
Relation. A connection between two models — BelongsTo, HasMany,
or ManyToMany. See Relations.
Replace (position). A middleware position that substitutes the step’s
default handler entirely. The last matching Replace middleware wins.
See Writing Middleware.
S
Saga. A multi-step workflow composed of forward steps and compensating undos, usually implemented via a transactional outbox and a worker. See Batch Operations & Sagas.
Scheduled (tag / runner). mfx:"scheduled;..." declares a
time-driven transition on a *time.Time column. The
scheduled.Runner sweeps the rows and applies the transition. See
Scheduled Fields.
Service name. Config.ServiceName. Identifies the service in logs,
audit records, and the X-Service-Name response header. See
Configuration.
Soft delete. Marking a row as deleted instead of removing it. Opt in
via maniflex.WithDeletedAt (timestamp) or maniflex.WithIsDeleted (boolean). See
Soft Delete.
Step (StepRegistry). One of the six pipeline steps. Exposes
Register(fn, opts...) to attach middleware. See
Pipeline Overview.
T
Tag (mfx: directive). A comma-separated list in a struct field’s
mfx tag declaring per-field behaviour. See
Field Tags Reference.
TenantID. The AuthInfo field that scopes a principal to one tenant.
Populated by JWT auth (TenantClaim) or by custom Auth middleware. See
Auth Middleware.
Through (tag). mfx:"through:JunctionModel" declares a many-to-many
relation via a named junction model. See Relations.
Trace (PipelineTrace). Debug-level pipeline tracing controlled by
Config.Trace. Sub-flags: Steps, Timings, Aborts, Bodies,
Skips. See Configuration.
Tx. A transaction handle returned by ctx.BeginTx (or the underlying
adapter). Stored on ctx.Tx; the default DB step routes through it
automatically. See Transactions.
V
Versioned (config). ModelConfig.Versioned = true writes a row to
the sibling {model}_history table on every change to the source model.
See Versioning & History.
W
WithTransaction. The catalogue middleware that wraps the DB step in
a transaction, committing on success and rolling back on error. See
Transactions.
writeonly. A tag directive that accepts the field on input but
strips it from responses. Standard choice for passwords. See
Field Tags Reference.
FAQ & Troubleshooting
Common questions and the pitfalls that catch new users. Each entry links to the page that covers the underlying concept in depth.
Registration & startup
“I registered a model after SetDB and nothing happens.”
The adapter is built from the registry; once it’s open, later registrations don’t reach it. Register every model before opening the database:
server.MustRegister(User{}, Order{}, Invoice{}) // 1. registry populated
db, _ := sqlite.Open("./app.db", server.Registry()) // 2. adapter built
server.SetDB(db) // 3. adapter wired in
See Architecture and Models & BaseModel.
“AutoMigrate didn’t add my new column.”
AutoMigrate adds missing columns but never drops or alters existing ones.
If your struct says string and the table column is INTEGER, the
migrator leaves it alone and logs a drift warning. Inspect the warning
log, then either fix the struct or run a manual ALTER TABLE.
“Why does the framework panic when a struct doesn’t embed BaseModel?”
Because every model needs an id and timestamp columns, and BaseModel
provides them. The check is deliberate; embedding BaseModel is one
line. See Models & BaseModel.
“Can I rename the id column?”
No. The framework hard-codes id as the primary-key column name across
the adapter, the relation resolver, and the OpenAPI generator. Pick a
table prefix or rename the table instead.
“Can my records use integer ids, or ids the client supplies?”
No to both. Identity is one string column holding a framework-generated
UUIDv4, and no route accepts an id from a request — an "id" in a write
body is ignored rather than rejected. Server-side code writing through
ctx.GetModel(...).Create may assign its own id, and then owns
uniqueness and URL-safety. Composite keys are not representable at all.
The full contract, including what a natural key should look like instead,
is Record Identity.
Pipeline & middleware
“My middleware doesn’t fire — I scoped it to OpAction.”
OpAction requests run a trimmed pipeline: Auth → action middleware → handler → Response. Middleware registered on Validate, Service, or DB
with ForOperation(OpAction) is never reached. Move per-action logic
into the action’s own middleware list:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: placeOrder,
Middleware: []maniflex.MiddlewareFunc{auth.JWTAuth(secret), checkStock},
})
See Custom Endpoints.
“ctx.Abort was called but the database still got written to.”
Probably called next() after Abort. Abort only populates
ctx.Response; it does not stop the chain. Return without next() and
the chain unwinds. See the “Calling next() after Abort” section of
ServerContext.
“Why doesn’t After-position middleware see my error?”
It does — through ctx.Response:
func afterDB(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { return err }
if ctx.Response != nil && ctx.Response.StatusCode >= 400 {
return nil // skip the side effect
}
record(ctx)
return nil
}
A non-nil error from next() and a 4xx/5xx ctx.Response are distinct
signals — both mean “the default step refused to succeed.”
“I want the same middleware on every model except one.”
Register it without ForModel, then register a passthrough that
short-circuits for the excluded model — or guard inside the middleware:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if ctx.Model.Name == "PublicResource" {
return next()
}
// ... usual work ...
return next()
})
Transactions
“LockForUpdate returned an error: ‘requires an active transaction’.”
Pessimistic locks only make sense inside a transaction. Wrap the call:
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil { return err }
ctx.Tx = tx
defer tx.Rollback()
row, err := ctx.LockForUpdate("StockBalance", id)
// ...
return tx.Commit()
…or register maniflex.WithTransaction(nil) on the Service step so the
transaction is already open by the time the lock fires.
“WithTransaction registered twice on SQLite — got a ‘tx already active’ error.”
SQLite does not support nested transactions. WithTransaction is
idempotent — registering it twice is fine — but calling ctx.BeginTx
manually inside an already-open SQLite transaction will fail. Reuse
ctx.Tx instead of starting a new one.
“My transaction committed even though I returned an error.”
Three things to check:
- The error was returned from
next(), not swallowed inside the middleware. - The middleware did not call
next()and return its own error — the framework treats those independently. ctx.Response.StatusCodeis>= 400ornext()returned non-nil. Both abort the commit.
Querying
“?filter=foo:eq:bar returns 400 INVALID_QUERY.”
The field isn’t filterable. Add the tag:
Foo string `json:"foo" mfx:"filterable"`
The check is intentional — exposing every column to client filters lets clients build arbitrary indexes against you. Opt in per field.
“Nested filter ?filter=author.name:eq:X doesn’t work.”
Two conditions:
- The relation must be declared on the current model (
AuthorIDfor convention,relation:Authorfor explicit). - The target field must itself be
filterableon the related model.
“I want a default sort order.”
There isn’t a built-in “default sort” tag. Register a Deserialize
middleware that appends to ctx.Query.Sorts when the client doesn’t
supply one:
server.Pipeline.Deserialize.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { return err }
if len(ctx.Query.Sorts) == 0 {
ctx.Query.Sorts = append(ctx.Query.Sorts, maniflex.SortExpr{
Field: "created_at", Direction: maniflex.SortDesc,
})
}
return nil
}, maniflex.ForModel("Article"), maniflex.ForOperation(maniflex.OpList), maniflex.AtPosition(maniflex.After))
“Soft-deleted rows show up in my admin tool list.”
They don’t — the framework filters them out everywhere. To see them, filter on the marker explicitly:
curl '...?filter=deleted_at:not_null'
See Soft Delete.
Files & encryption
“My multipart upload returns 501 NO_STORAGE.”
Config.FilesConfig.Storage is nil. Configure a backend:
fs, _ := storage.NewLocalStorage("./uploads")
server := maniflex.New(maniflex.Config{
FileStorage: fs,
// other application settings
})
“Encrypted field rejected my write: ENCRYPTION_NOT_CONFIGURED.”
Same shape — Config.KeyProvider is nil. Configure one (e.g.
encryption.EnvKeyProvider) before any model with mfx:"encrypted" is
exercised. See Encryption at Rest.
“I added mfx:"unique" to an encrypted field and got a duplicate-key error on legit data.”
The framework stores a keyed HMAC of the plaintext in a {field}_hmac
companion column. Two plaintexts that hash to the same digest is a
collision — astronomically unlikely with SHA-based HMAC. More likely you
have legacy rows whose digests were made with a field-encryption key. Configure
the provider’s stable IndexKeyID, quiesce writes, and backfill those companion
digests before online encryption-key rotation. RotateEncryptionKeyWithOptions
preflights the digests and reports every mismatched row instead of changing
them during rotation.
Auth
“I added auth.JWTAuth and sign-up stopped working.”
Sign-up is a write — a global auth.JWTAuth rejects it because no token
exists yet. There is no AllowPublicWrite helper; instead scope the
authenticator so it never runs on the sign-up path. Auth scoping
(ForModel / ForOperation) is inclusion-only: a middleware runs only
for the models and operations you scope it to, so anything you leave out of
every auth registration stays public.
// Protect updates and deletes everywhere…
server.Pipeline.Auth.Register(
auth.JWTAuth(secret),
maniflex.ForOperation(maniflex.OpUpdate, maniflex.OpDelete),
)
// …and protect creates only on the models that need a session. "User" is
// deliberately not listed, so POST /users (sign-up) is covered by no auth
// registration and stays public.
server.Pipeline.Auth.Register(
auth.JWTAuth(secret),
maniflex.ForModel("Post", "Comment"),
maniflex.ForOperation(maniflex.OpCreate),
)
Scoping the authenticator this way is safe only while the model list is. Because
the filters are inclusion-only, a model registered later is named in no
registration and so is covered by no auth at all — silently. Prefer
auth.AllowAnonymous for anything but the smallest API: it leaves JWTAuth
registered globally and makes the exemption the thing you enumerate, so an
unlisted route refuses rather than opens.
server.Pipeline.Auth.Register(auth.AllowAnonymous(),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
server.Pipeline.Auth.Register(auth.JWTAuth(secret))
It is also the only way to have a route that authenticates a token when one is
sent and still serves a caller who sends none — a JWTAuth scoped away from the
route does not run there, so even a valid token leaves ctx.Auth nil.
auth.AllowPublicRead is a passthrough, not an exemption: an aborting
authenticator answers 401 before it is reached, so it needs AllowAnonymous in
front of the authenticator to do anything at all.
“JWT keeps returning 401 — the token validates manually.”
Three usual suspects:
- Algorithm mismatch —
HS256token verified against anRS256public key (or vice versa). SetJWTOptions.PublicKeyfor asymmetric keys. Issuer/Audiencemismatch — the framework rejects tokens whoseiss/auddoesn’t match the configured value.- Clock skew — the token’s
nbfis in the future orexpis in the past on the server clock. Sync NTP.
Database
“Why does Tenancy middleware not apply to my reads?”
It does, including reads — Tenancy filters every operation, including
list and read. If you’re seeing rows from another tenant, check that the
middleware is registered without a ForOperation filter and that
ctx.Auth.TenantID is being populated by the upstream Auth middleware.
“My Postgres reads have stale data after a write.”
Read replicas have replication lag. The framework routes reads to the
replica only outside an active transaction; reads inside a
WithTransaction-managed request go to the primary. For
read-your-writes outside a transaction, run the query through
ctx.RawQuery against an explicit primary connection, or shorten the
client’s expected window.
“I dropped a column from my struct — AutoMigrate didn’t remove it.”
By design. The migrator never drops; it logs a drift warning so you see
the column still exists. Remove it with an explicit ALTER TABLE … DROP COLUMN
during a maintenance window.
“I changed a field’s type — the column still has the old one.”
Also by design. AutoMigrate adds columns; it never rewrites one, because
converting a column’s type can lose data and locks the table while it runs. It
logs a drift warning naming the table, the column, the type the database has and
the type the model wants, at every startup. Convert the column with an explicit
versioned migration (ALTER TABLE … ALTER COLUMN, plus whatever backfill the
new type needs).
“504 TIMEOUT on a query that used to work.”
Config.QueryTimeout fired. The deadline is per request, applied to
ctx.Ctx. Either:
- Raise the timeout (
30sis a common ceiling). - Speed up the query (missing index, expensive include, large LIMIT).
- Add a
db.Paginatecap on the offending list endpoint.
A 504 always means a server-side deadline expired. A client that gives up and
disconnects mid-request is logged as 499 instead, so the two do not share a
line in your metrics.
OpenAPI
“/openapi.json is empty.”
You didn’t register any models before calling Handler or Start. The
generator reads the registry at request time, but that registry is sealed when
the router is built; late model registration is rejected so the specification
and mounted routes cannot disagree.
“I customised the spec but my changes don’t appear.”
Register the customisation at maniflex.After position on the Generate step,
not Before. Before runs against an empty spec; After mutates the
just-generated document.
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetTitle("My API"),
maniflex.After, // <-- not the default Before
)
Production
“My pod terminated mid-request on deploy.”
Kubernetes sent SIGTERM, gave you terminationGracePeriodSeconds,
then SIGKILL’d the process. Config.ShutdownTimeout defaults to 30s;
set the pod’s grace period larger (60s is comfortable) so the graceful
path has time to complete. See Graceful Shutdown.
“Readiness returns 503 intermittently.”
/ready pings the database and runs every Config.ReadinessChecks entry, so
503 means a dependency did not answer in time. Tune:
Config.HealthTimeout— should be shorter than the probe’stimeoutSeconds.- The database — if the pool is exhausted,
db.Ping()waits for a connection. - Your own checks — they run on every probe request, so a full round-trip through a dependency will eventually time one out.
The same applies to /health when Config.HealthCheckDB is true.
Which dependency failed is in the log, not the response — the body reports only
{"status":"not_ready"} unless Config.Probes.PublishReadinessChecks is set,
so that a public probe does not name what you depend on.
“Kubernetes restarts my pods whenever the database blips.”
The liveness probe is pointed at a database-backed endpoint — /api/ready, or
/api/health with HealthCheckDB on. Point it at /api/live, which answers
200 for as long as the process is alive and never touches a dependency.
Restarting a replica cannot fix a database, and doing it to every replica at
once is how an outage gets worse. See
Configuration.
“My auth middleware doesn’t run on /health or /ready.”
By design. The probes are mounted straight onto the router and never enter the
model pipeline, so no Pipeline.Auth middleware sees them and
authx.AllowPublic has nothing to exempt them from — an orchestrator’s probe is
the canonical unauthenticated request.
Use Config.Probes when that is not what you want. It gates or unmounts each
probe independently, and it is the only lever scoped to the probes alone
(Config.HTTPMiddlewares reaches them, but reaches every other route too):
cfg.Probes = maniflex.ProbesConfig{
Ready: maniflex.ProbeConfig{Middleware: []maniflex.HTTPMiddleware{probeToken}},
Health: maniflex.ProbeConfig{Disabled: true},
}
Gate /ready, not /live. A liveness probe that gets a 401 is a liveness
probe that fails, and the answer to that is a SIGKILL mid-drain. See
Gating and unmounting the probes.
“Logs are noisy with debug records.”
Config.Trace.Enabled = true was left on, or the Logger accepts DEBUG.
Set the handler’s level to INFO in production. Trace flags are opt-in
specifically because they’re high-volume.
Library design questions
“Why reflection instead of codegen?”
To avoid the regeneration step. A mfx: tag change is in effect on the
next process start; nothing to rebuild. The cost is one reflection pass
per model at boot — usually under a millisecond per model. See
Architecture.
“Can I use maniflex with GraphQL?”
The generated routes are REST. You can put GraphQL in front (gqlgen,
graphql-go) and have resolvers call ctx.GetModel(...).List / .Read
/ etc. — the model accessor doesn’t care which HTTP layer is on top.
“Can I use multiple databases?”
One adapter per *maniflex.Server. For multi-database setups, run two servers
(possibly in the same process) and route between them at the application
layer, or use raw queries from custom actions that target the secondary
database directly.
“Is there a CLI?”
Not yet. The framework is intentionally library-only — no project-scaffolding command, no migration runner. Use the standard Go toolchain and an external migration tool. The App Anatomy page describes the recommended project layout.
Where to find more
If your question isn’t here, three places to look next:
- The page for the concept involved — every link in this FAQ points to it.
- The framework’s own e2e tests under
tests/e2e/— they exercise every edge case the docs describe. - The source. The
maniflexpackage is small; reading the implementation of a step or a middleware is often faster than guessing.
AI Agents
A condensed, self-contained reference for AI coding agents working on maniflex
projects. Copy the block below into CLAUDE.md, AGENTS.md, or an equivalent
context file. Everything an agent needs to write correct maniflex code is in it;
no prose links to chase.
# maniflex — Reference for AI coding agents
Reflection-driven Go REST framework. Annotated structs in → full REST API out:
filtering, pagination, relations, soft-delete, file uploads, OpenAPI 3.1 spec.
No codegen.
## Modules
- `maniflex` — core (chi + uuid only).
- `maniflex/db/sqlite` — pure-Go SQLite (modernc.org/sqlite). No CGo.
- `maniflex/db/postgres` — lib/pq.
- `maniflex/middleware/{auth,body,validate,service,db,response,openapi,idempotency}` — catalogue.
- `maniflex/middleware/service/bcrypt` — password hashing.
- `maniflex/middleware/db/redis` — Redis rate-limit backend (`db.RateLimitConfig.Backend`).
- `maniflex/events/{kafka,nats,rabbitmq,redis}` — event publishers.
- `maniflex/jobs/redis` — background job queue.
- `maniflex/scheduled` — runner for `mfx:"scheduled"` fields.
- `maniflex/storage` — local file storage; ships `LocalStorage`.
- `maniflex/pkg/encryption` — `EnvKeyProvider`, `VaultKeyProvider`.
Each is its own module. Import only what you use.
## The four-step lifecycle (fixed order)
```go
server := maniflex.New(maniflex.Config{Port: 8080, PathPrefix: "/api"})
server.MustRegister(User{}, Post{}) // 1. populate registry
db, _ := sqlite.Open("./app.db", server.Registry()) // 2. adapter reads registry
server.SetDB(db) // 3. inject adapter
log.Fatal(server.Start()) // 4. serve
```
Models registered after `SetDB` do not reach the adapter. The adapter is built
from the registry at `Open` time.
## Models
Every model embeds `maniflex.BaseModel`:
```go
type Post struct {
maniflex.BaseModel // adds id (UUID), created_at, updated_at
// all three are mfx:"readonly" and nothing
// more — not filterable, not sortable
maniflex.WithDeletedAt // optional — adds deleted_at (timestamp soft delete)
// maniflex.WithIsDeleted // alternative — adds is_deleted (bool)
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,enum:draft|published"`
UserID string `json:"user_id" mfx:"required,filterable"` // BelongsTo User
}
```
Table name = pluralised snake-case of struct name (`BlogPost` → `blog_posts`).
Override with `ModelConfig.TableName`.
Validation:
- Struct must be a struct type and embed `BaseModel`, else `MustRegister` panics.
- Field types: scalars, `*time.Time`, slices for relations, `map[string]any`,
structs for companions.
## The `mfx:` tag — complete list
Comma-separated. Whitespace trimmed. Unknown directives ignored.
**Validation:**
- `required` — must be present on create
- `enum:a|b|c` — pipe-separated allowed values
- `min:N`, `max:N` — numeric bounds
- `default:V` — used when field absent on create (cast to type)
**Write access:**
- `readonly` — stripped from all writes
- `immutable` — accepted on create, rejected on update
**Response visibility:**
- `writeonly` — accepted on write, hidden in responses (e.g. password)
- `hidden` — hidden in responses **and** stripped from create/update schemas
**Query:**
- `filterable` — usable in `?filter=`
- `sortable` — usable in `?sort=`
- `searchable` — full-text search hint
**Schema:**
- `unique` — `UNIQUE` constraint at the column
**Relations** (semicolon-separated sub-options):
- `relation:Name` — explicit FK; requires companion field `Name` of target type
- `relation:Name;onDelete:cascade|setNull|restrict` — referential action
- `through:JunctionModel` — declares M2M via junction (on slice fields)
**Files:**
- `file` — multipart file field; column stores storage key
- `max_size:N` — `KB`/`MB`/`GB` suffix or bytes
- `accept:pattern1|pattern2` — MIME-type patterns
- `auto_delete:false` — keep stored file when row deleted or field replaced
**Encryption:**
- `encrypted` — AES-256-GCM at rest; not `filterable`/`sortable`
- `key:NAME` — keyID for KeyProvider (default `"default"`)
- `encrypted,unique` — adds `{field}_hmac` companion column for uniqueness
**Versioning** (on embedded `BaseModel` field):
- `mfx:"versioned"` — adds `{model}_history` sibling table
- `mfx:"versioned:diff_only"` — store diffs only, skip snapshots
**Scheduled** (on `*time.Time` fields):
- `scheduled;soft-delete` — needs `WithDeletedAt`/`WithIsDeleted`
- `scheduled;hard-delete`
- `scheduled;field=NAME;to=VALUE` — required pair
- `scheduled;field=NAME;from=OLD;to=NEW` — guarded transition
**Exclusion:** `json:"-"`, `db:"-"`, or `mfx:"-"` removes the field entirely.
## Relations
| Kind | Declared by | Relation key |
|---|---|---|
| BelongsTo (convention) | `UserID string` field → `User` | `user` |
| BelongsTo (explicit) | `ManagerID string \`mfx:"relation:Manager"\`` + `Manager User` companion | `manager` |
| HasMany | `Posts []Post` slice field | `posts` |
| ManyToMany | `Tags []Tag \`mfx:"through:ProductTag"\`` on both sides + junction model registered | `tags` |
Include in queries: `?include=user,posts,tags`. Nested filter: `?filter=user.role:eq:admin`.
Junction models for M2M are registered like any model.
## The 6-step pipeline
`Auth → Deserialize → Validate → Service → DB → Response`. Each step has a
default + `*StepRegistry` on `server.Pipeline`.
Middleware signature:
```go
type MiddlewareFunc func(ctx *maniflex.ServerContext, next func() error) error
```
Register:
```go
server.Pipeline.Service.Register(myFn,
maniflex.ForModel("User", "Order"), // optional, by struct name
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate), // optional
maniflex.AtPosition(maniflex.Before), // Before (default) | After | Replace
maniflex.WithName("my-fn"), // optional, for traces
)
```
Operations: `OpList`, `OpRead`, `OpCreate`, `OpUpdate`, `OpDelete`, `OpOptions`, `OpAction`. A `HEAD` request runs as the `GET` it mirrors (`OpRead` / `OpList`); `OpHead` is never set.
`OpAction` uses a trimmed pipeline: `Auth → action middleware → handler → Response`.
Validate/Service/DB middleware never fires for actions.
**Short-circuit pattern (always pair these):**
```go
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing token")
return nil // do NOT call next()
```
Calling `next()` after `Abort` lets downstream steps run and possibly overwrite `ctx.Response`. Always return without `next()`.
**Default step behaviours:**
- Auth: passthrough. Populate `ctx.Auth` here.
- Deserialize: parses query params → `ctx.Query`; body → `ctx.ParsedBody`; multipart → `ctx.Files`. 4 MB JSON body limit (multipart is `FilesConfig.MaxUploadBytes`, default 32 MB).
- Validate: enforces `mfx:` tag rules on create/update. Strips `readonly`/`id`; strips `immutable` on update.
- Service: passthrough. Business logic goes here.
- DB: dispatches to adapter. Routes through `ctx.Tx` when set. Maps `ErrNotFound`→404, `*ErrConstraint`→409, `context.DeadlineExceeded`→504, client disconnect→499 (`maniflex.StatusClientClosedRequest`, no body).
- Response: builds `APIResponse` from `ctx.DBResult`. List adds `meta`. Delete returns 204.
## ServerContext (per-request, not goroutine-safe)
Common fields:
- `Request *http.Request`, `Writer http.ResponseWriter`, `Ctx context.Context`
- `Model *ModelMeta`, `Operation Operation`, `ResourceID string`
- `RequestID string`, `TraceID string`
- `RawBody []byte`, `ParsedBody *RequestBody` (read-only — `ctx.Field` to read, `ctx.SetField`/`DeleteField` to mutate), `Query *QueryParams`, `Files map[string]*UploadedFile`
- `DBResult any` (`*ListResult` for list; the record otherwise — a typed `*T` on reads)
- `Response *APIResponse`
- `Auth *AuthInfo`, `Tx Tx`
Methods:
- `Abort(status int, code, message string)` — sets `ctx.Response`; caller returns nil without `next()`.
- `BindJSON(v any) error` — decode body into `v`; calls Abort on error.
- `URLParam(name) string`, `QueryParam(name) string`
- `Set(k, v)` / `Get(k) (any, bool)` — cross-step storage
- `Logger() *slog.Logger` — pre-seeded with request_id, trace_id, service
- `HasRole(role string) bool`
- `BeginTx(ctx, opts *TxOptions) (Tx, error)` — start a transaction
- `LockForUpdate(modelName, id) (map[string]any, error)` — requires `ctx.Tx`
- `RawQuery(sql, args...) ([]map[string]any, error)` — routes through ctx.Tx
- `RawExec(sql, args...) (int64, error)`
- `GetModel(name) *ModelAccessor` — CRUD on any registered model; routes through ctx.Tx
ModelAccessor methods: `List(q)`, `Read(id)`, `Create(data)`, `Update(id, data)`, `Delete(id)`.
`AuthInfo`:
```go
type AuthInfo struct {
UserID string
Roles []string
Claims map[string]any
TenantID string
IdentityType AuthIdentityType // IdentityHuman | IdentityServiceAccount | IdentityAnonymous
Scopes []string
SessionID string
AuthMethod string // "jwt" | "api_key" | "session" | ...
}
```
## Transactions
```go
// Option A — middleware-wrapped, automatic commit/rollback
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil), // nil opts = default isolation
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
// Option B — manual
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil { return err }
ctx.Tx = tx
defer tx.Rollback() // no-op after Commit
// ... ctx.GetModel(...).Create/Update/Delete all routed through tx ...
return tx.Commit()
```
`WithTransaction` commits if `next()` returned nil AND `ctx.Response` is nil or 2xx. Otherwise rolls back. Idempotent — registering twice is fine, second call sees existing `ctx.Tx`.
SQLite does not support nested transactions. Its write connections open with `BEGIN IMMEDIATE` (`_txlock=immediate`, set by `sqlite.Open`), so read-then-write transactions serialise instead of racing.
## Errors
Sentinels (use `errors.Is` / `errors.As`):
```go
maniflex.ErrNotFound // 404 NOT_FOUND
*maniflex.ErrConstraint // 409 CONFLICT (Table, Column, Detail)
```
Built-in error codes the framework emits:
`INVALID_JSON`, `EMPTY_BODY`, `BODY_READ_ERROR`, `INVALID_QUERY`,
`MULTIPART_ERROR`, `NOT_FOUND`, `CONFLICT`, `VALIDATION_ERROR`,
`DB_ERROR`, `TX_BEGIN_ERROR`, `TX_COMMIT_ERROR`, `NO_STORAGE`,
`TIMEOUT`, `PANIC`, `ENCRYPTION_NOT_CONFIGURED`.
Envelope: `{"error": {"code": "...", "message": "...", "details": ...}}`
Success envelope: `{"data": ...}`; list adds `"meta": {total, page, limit, pages}`.
## Querying (only on opt-in fields)
Operators: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `like`, `ilike`, `contains`, `starts_with`, `ends_with`, `has`, `not_has`, `in`, `not_in`, `between`, `is_null`, `not_null`.
`like`/`ilike` take a raw SQL pattern (`%`, `_` are wildcards). `contains`/`starts_with`/`ends_with` take a literal value — `%` and `_` are escaped and match themselves — and are case-insensitive. Use the latter for user-typed text.
`has`/`not_has` ask whether a JSON column holds a value, and need the column tagged `mfx:"json_array"` (element membership: `tags:has:urgent`) or `mfx:"json_object"` (a key=value pair: `meta:has:tier=gold`). `contains` on such a column is refused: it substring-matches the serialised document.
Bare `?filter=` clauses AND. `?filter[N]=` puts a clause in OR group N: same index ORs, different indexes AND, and a bare clause is its own AND term — so the expressible shape is an AND of ORs, with no nesting and no OR across groups. The index must be a non-negative integer (`filter[recent]` is `400 INVALID_QUERY`). In Go the equivalent is `FilterExpr.Group`, where `0` means ungrouped, so URL `filter[0]` is `Group: 1`.
```
?filter=status:eq:published
?filter=views:gte:100&filter=status:eq:published # ANDed
?filter[0]=status:eq:draft&filter[0]=status:eq:published # OR within a group
?filter[0]=a:eq:1&filter[0]=b:eq:2&filter[1]=c:gte:3 # (a OR b) AND (c)
?filter=tag:in:go,rust,zig
?filter=title:contains:intro # literal, case-insensitive
?filter=author.name:ilike:%ursula% # relation dot notation
?sort=created_at:desc&sort=title:asc # created_at needs BaseModelTags
?include=user,comments,tags
?page=2&limit=20 # default 20, max 200
```
Soft-deleted rows are filtered out of list/read/include automatically. To see them: `?filter=deleted_at:not_null`.
## Custom Actions
```go
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Handler: cancelOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
// any maniflex.MiddlewareFunc — runs between Auth and the handler
},
})
```
```go
func cancelOrder(ctx *maniflex.ServerContext) error {
id := ctx.URLParam("id")
var req MyReq
if err := ctx.BindJSON(&req); err != nil { return nil }
// ... work via ctx.GetModel / ctx.RawExec / ctx.BeginTx ...
ctx.Response = &maniflex.APIResponse{StatusCode: http.StatusOK, Data: result}
return nil
}
```
Action handler owns body parsing, validation, transactions. Validate/Service/DB pipeline steps do NOT run for actions.
OpenAPI for an action comes from `ActionConfig.OpenAPI` (`ActionOpenAPI`): `RequestSchema`/`ResponseSchema` reflect Go structs into JSON schemas, `Description`, `Security`, and `QueryParams []OASParameter`. Path parameters are extracted from the route; query parameters are not discoverable and must be declared. A `QueryParams` entry's `In` defaults to `"query"`; set it explicitly only for `"header"`/`"cookie"`. Every other field is emitted as written. Declarations are documentation only: nothing validates or binds them at runtime, so the handler still checks `ctx.QueryParam` itself.
## File uploads
Tag a string field `mfx:"file,max_size:2MB,accept:image/*"`. Configure `maniflex.Config.FilesConfig.Storage`.
Two upload styles:
1. Multipart POST/PATCH to the model endpoint — fields become `ctx.ParsedBody`, file parts become `ctx.Files`, storage key is written into the column.
2. Two-step: `POST /files` (multipart, field name `file`) returns `{"data":{"key":...}}`. Pass the key as a JSON string on the file field.
Standalone routes (when `FileStorage` set):
- `POST /files` — upload
- `GET /files/{key...}` — download (sets Content-Type, Content-Disposition, Content-Length)
- `DELETE /files/{key...}`
Without `FileStorage`, multipart and `/files/*` return 501 NO_STORAGE.
Auto-cleanup: stored file deleted on hard-delete or field overwrite. `auto_delete:false` opts out. Soft-delete does not trigger cleanup.
## Catalogue middleware
<!-- doccheck:ignore reason="catalogue entries are independent call shapes with application-defined placeholders" -->
```go
import (
"github.com/xaleel/maniflex/middleware/auth"
"github.com/xaleel/maniflex/middleware/body"
"github.com/xaleel/maniflex/middleware/validate"
"github.com/xaleel/maniflex/middleware/service"
"github.com/xaleel/maniflex/middleware/db"
"github.com/xaleel/maniflex/middleware/response"
"github.com/xaleel/maniflex/middleware/openapi"
"github.com/xaleel/maniflex/middleware/idempotency"
)
// AUTH (Auth step)
auth.JWTAuth(secret, auth.JWTOptions{Issuer, Audience, TenantClaim, ScopesClaim, PublicKey})
auth.APIKeyAuth("X-API-Key", auth.APIKeyEntry{Key, Auth: maniflex.AuthInfo{...}}, ...)
auth.RequireRole("admin")
auth.RequireScope("posts:read", "posts:write") // ALL of them; RequireAnyScope(...) for any one
auth.AllowAnonymous() // register BEFORE JWTAuth/JWKSAuth + ForModel/ForOperation:
// no credential -> served, ctx.Auth nil; bad credential -> still 401
auth.AllowPublicRead() // passthrough on read/list; needs AllowAnonymous in front of the
// authenticator to be reachable at all
auth.BlockOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete)
// (no AllowPublicWrite — scope AllowAnonymous onto the op instead. Prefer that to scoping JWTAuth off it:
// filters are inclusion-only, so a model added later is in no registration and gets no auth at all)
// BODY (Deserialize / Validate steps)
body.MaxBodySize(16 << 20) // override 4MB JSON default (also caps multipart, default 32MB — this LOWERS it)
body.StripUnknownFields()
body.CoerceTypes()
// VALIDATE (Validate step)
validate.UniqueField(sqlDB, maniflex.SQLite, "email") // driver: maniflex.Postgres | maniflex.SQLite
validate.RegexField("phone", `^\+?[0-9]{7,15}$`)
validate.ForbiddenValues("role", "superadmin")
validate.RequireAtLeastOne("name", "email")
validate.CrossFieldValidate(func(body map[string]any) error { return ... })
validate.DateRange("start_date", "end_date") // end must not be before start
validate.RequireWhen("reason", "status:eq:rejected") // conditional required
// SERVICE (Service step — usually Before)
service.HashField("password", svcbcrypt.Hasher()) // import middleware/service/bcrypt
service.SlugifyField("title", "slug")
service.SetField("user_id", func(ctx) any { return ctx.Auth.UserID })
service.StripField("password_confirm")
service.CopyField("email", "billing_email")
service.Timestamp("last_seen_at")
service.TimestampWhen("published_at", "status", "published")
service.OwnerScope("user_id") // unconditional SetField on create
// Side effects are in package events (NOT service):
events.Emit(bus) // DB-step middleware, AtPosition(After)
// Webhook/SendEmail are event-bus SUBSCRIBERS, not middleware — wire with bus.Subscribe:
bus.Subscribe(ctx, events.Subscription{Patterns: []string{"order.*"},
Handler: events.Webhook(events.WebhookConfig{URL: u, Secret: s})})
// DB (DB step)
db.ForceFilter("org_id", func(ctx) any { return ctx.Auth.Claims["org_id"] })
db.Tenancy("org_id", func(ctx) string { return ctx.Auth.TenantID })
db.Paginate(50)
db.RateLimit(db.RateLimitConfig{RequestsPerMinute: 10, KeyFunc: func(ctx) string {...}}) // Backend: for a cross-replica counter
db.AuditLog(sink) // AtPosition(After), or default Before with db.WithChanges()
db.Invalidate(cache, func(ctx) []string { return ["keys", ...] }) // AtPosition(After)
// HTTP ROUTER (Config.HTTPMiddlewares; before route dispatch/Auth)
response.CORSHeaders("https://app.example.com") // validates preflight; origins required; "*" panics with credentials
// REQUEST OBSERVERS (Server.ObserveRequests; router entry through response writing)
response.Logging(slog.Default())
response.Metrics(collector)
// RESPONSE (Response step)
response.Cache(response.CacheConfig{MaxAge: 300}) // private by default; AtPosition(After)
response.TransformField("avatar_url", func(v any) any { return cdn+v.(string) })
response.RedactField("phone", func(ctx) bool { return !ctx.HasRole("support") })
response.Envelope(func(ctx, data, meta) any { return ... })
response.AddHeader("Strict-Transport-Security", "max-age=63072000")
// OPENAPI (OpenAPI.Generate step, maniflex.After position)
openapi.SetTitle("My API")
openapi.SetDescription("...")
openapi.AddServer("https://api.example.com", "Production")
openapi.AddSecurityScheme("bearerAuth", maniflex.OASSecurityScheme{Type: "http", Scheme: "bearer"})
openapi.AddExtension(func(spec *maniflex.OpenAPISpec) { /* mutate freely */ })
// IDEMPOTENCY (Deserialize step, AtPosition(After), scoped to OpCreate)
idempotency.Middleware(idempotency.Config{
Store: maniflex.NewMemoryCache(), // or any maniflex.CacheStore (e.g. Redis)
TTL: 24 * time.Hour,
KeyFunc: func(ctx) string { return ctx.Auth.UserID },
HeaderRequired: false,
})
// Reads Idempotency-Key header. Replays cached 2xx for same key+body.
// Same key + different body → 422 IDEMPOTENCY_KEY_REUSED.
// Adds "Idempotent-Replayed: true" on replay.
```
## Encryption at rest
```go
import "github.com/xaleel/maniflex/pkg/encryption"
server := maniflex.New(maniflex.Config{
KeyProvider: &encryption.EnvKeyProvider{Prefix: "MYAPP_KEY"},
// or: &encryption.VaultKeyProvider{
// Address: vaultAddress, Token: vaultToken, Mount: "transit",
// }
})
type Patient struct {
maniflex.BaseModel
SSN string `json:"ssn" mfx:"encrypted,key:patient-pii"` // → MYAPP_KEY_PATIENT_PII env var
}
```
- Storage: `enc:<base64(envelope)>`.
- `EnvKeyProvider` needs base64-encoded 32-byte keys in env vars.
- Encrypted fields cannot be `filterable`/`sortable`.
- `encrypted,unique` adds `{field}_hmac TEXT UNIQUE` companion column.
- Set `EnvKeyProvider.IndexKeyID` / `VaultKeyProvider.IndexKeyID` to a dedicated,
non-rotating HMAC key before writing `encrypted,unique` data.
- `RotateEncryptionKeyWithOptions` preflights envelopes and blind indexes,
reports row/field failures, and provides `LastID` for interruption resume.
Keep old, new, and index keys active until `Complete` is true.
## Versioning
```go
server.MustRegister(Invoice{}, maniflex.ModelConfig{Versioned: true})
// or VersionedDiffOnly: true to skip snapshots
```
Creates `invoice_history` table (the source name + `_history`, not pluralised) with columns: `id, record_id, version, operation, actor_id, timestamp, request_id, diff, [snapshot]`.
Both the diff **and** the snapshot exclude hidden/writeonly/encrypted fields and HMAC companions — a history row is built from the decrypted record, so anything left in would be stored as plaintext (audit MS-3).
Read history at `GET /invoices/{id}/history` — newest first, `?page=`/`?limit=` (default 20). The history model is `Headless`: there is **no** `/invoice_history` endpoint. It runs the parent's read pipeline, so a caller who cannot read the record gets the same `404` for its history; scope with `ForOperation(maniflex.OpRead, maniflex.OpReadHistory)`. A soft-deleted record keeps readable history; a hard-deleted one does not (nothing left to authorise against).
## Scheduled runner
```go
import "github.com/xaleel/maniflex/scheduled"
runner, _ := scheduled.New(server, scheduled.Config{
Interval: time.Minute,
BatchSize: 500,
OnDelete: func(model, id string) { /* application hook */ },
OnSetField: func(model, id, field, to string) { /* application hook */ },
})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
runner.Start(ctx)
defer runner.Stop()
```
Scans `*time.Time` columns with `mfx:"scheduled"` tag. Per-model transactional batches. Hooks fire after commit. Use `runner.Sweep(ctx)` for one-shot ticks.
For multi-replica deployments: run runner in one replica, or use `maniflex/scheduled/jobsx` to dispatch sweeps through a job queue.
## Configuration
```go
type Config struct {
Port int // default 8080
PathPrefix string // default "/api"
Documentation DocumentationConfig // zero value mounts no specs
ServiceName string // adds "service" attr to logs, X-Service-Name header
DB DBAdapter // required before Start
DisableAutoMigrate bool // migration runs by default; set to skip it
QueryTimeout time.Duration // per-request DB deadline; 0 = unlimited
QueryLimits QueryLimits // URI/query/aggregate complexity caps
HTTPAccessControlled bool // asserts global HTTP middleware protects every route
ShutdownTimeout time.Duration // default 30s
Logger *slog.Logger
PanicLogger *slog.Logger
Trace PipelineTrace // {Enabled, Steps, Timings, Aborts, Bodies, Skips}
FileStorage FileStorage
KeyProvider KeyProvider
HealthCheckDB bool // legacy GET {prefix}/health pings DB
ReadinessChecks []ReadinessCheck // {Name, Check} app dependencies on {prefix}/ready
HealthTimeout time.Duration // default 3s; budget shared by all dependency checks
Probes ProbesConfig // {Middleware, Live, Ready, Health, PublishReadinessChecks}
}
```
Probes are mounted under `PathPrefix`: `GET {prefix}/live` is liveness (always
`200`, no I/O, stays `200` during the drain), `GET {prefix}/ready` is readiness
(`503` while starting or stopping, then `{"status":…}` from the DB ping plus
every `ReadinessChecks` entry — the per-dependency `checks` map is withheld
unless `Probes.PublishReadinessChecks` is set, since it names what the app
depends on and which part is failing; concurrent requests share one run of the
checks, coalesced not cached), and `GET {prefix}/health`
is the legacy alias whose meaning follows `HealthCheckDB`. Never point a
liveness probe at a database-backed endpoint. Check errors are logged, never
echoed; a check name that is empty, duplicated, or `db` panics at router build.
Probes never enter the pipeline, so `Pipeline.Auth` does not run for them and
they are public by default. `Config.Probes` is the only probe-scoped override:
`Probes.Middleware` wraps all three, `Probes.{Live,Ready,Health}.Middleware`
wraps one (after the shared chain, not instead of it), and
`Probes.{Live,Ready,Health}.Disabled` unmounts one so the router answers `404`
and no middleware runs. Gate `/ready`, not `/live` — a rejected liveness probe
gets the container killed mid-drain; gate `/live` only with a credential the
probe URL can carry.
After registration and before `Start`/`Handler`, call
`server.ValidateProduction()`. Generated operations need matching Auth
middleware or a scoped `server.AllowPublic(...)` declaration; custom actions,
global search, and standalone files have corresponding explicit access flags.
`maniflex.ConfigFromEnv(prefix) (Config, error)` reads PORT, DB_WRITE_URL, DB_READ_URL, QUERY_TIMEOUT_MS, SHUTDOWN_TIMEOUT_S, SERVICE_NAME, HEALTH_CHECK_DB — those and no others. A non-empty prefix is applied with an underscore (`ORDERS_PORT`). Unset variables are left zero for `ApplyDefaults`; a variable that is set but unreadable (`PORT=808O`) returns an error naming it. Do not discard the error.
## Database adapters
```go
// SQLite (dev) — write connections get _txlock=immediate automatically.
db, err := sqlite.Open("./app.db", server.Registry())
db, err := sqlite.Open(":memory:", server.Registry())
// PostgreSQL (prod) — Open(writeDSN, readDSN, registry) is positional.
db, err := postgres.Open(os.Getenv("DB_WRITE_URL"), os.Getenv("DB_READ_URL"), server.Registry())
// Pool/session tuning → OpenWithConfig(writeDSN, readDSN, registry, writePool, readPool, session):
db, err := postgres.OpenWithConfig(
os.Getenv("DB_WRITE_URL"), os.Getenv("DB_READ_URL"), server.Registry(),
postgres.PoolConfig{MaxOpenConns: 4, MaxIdleConns: 4, ConnMaxLifetime: 30 * time.Minute}, // write
postgres.PoolConfig{MaxOpenConns: 10, MaxIdleConns: 10, ConnMaxLifetime: 30 * time.Minute}, // read
postgres.SessionConfig{ApplicationName: "myapp"},
)
```
Pool defaults are 3 write / 6 read — both pools open even when readDSN is `""`, so budget `(write + read) × processes ≤ max_connections − reserved` (entry tiers: Heroku 20, DigitalOcean/GCP 25, Azure B1ms 50, Supabase 60, Neon 104). `Open` WARNs when the pools claim over half the server's `max_connections`; route it with `SessionConfig.Logger`. Behind PgBouncer in transaction mode, add `binary_parameters=yes` to the DSN.
Reads route to ReadURL outside an active transaction; reads inside a tx go to write primary. AutoMigrate adds missing columns; never drops. Logs drift warnings.
## ModelConfig (per-model options)
```go
server.MustRegister(MyModel{}, maniflex.ModelConfig{
TableName: "custom_table",
SoftDelete: maniflex.SoftDeleteConfig{Enabled: true, Field: "deleted_at", FieldType: maniflex.SoftDeleteTimestamp},
Middleware: &maniflex.ModelMiddleware{
Auth: []maniflex.MiddlewareFunc{/* configured middleware */},
Deserialize: []maniflex.MiddlewareFunc{/* configured middleware */},
Validate: []maniflex.MiddlewareFunc{/* configured middleware */},
Service: []maniflex.MiddlewareFunc{/* configured middleware */},
DB: []maniflex.MiddlewareFunc{/* configured middleware */},
Response: []maniflex.MiddlewareFunc{/* configured middleware */},
},
Versioned: true,
VersionedDiffOnly: false,
QueryLimits: maniflex.QueryLimits{MaxFilterClauses: 16},
Indices: []maniflex.IndexSpec{{Name, Columns, Unique}},
})
```
`Register` accepts `...any`. Slice arguments are flattened one level — so you can do:
```go
var AuthModels = []any{User{}, Role{}}
var OrderModels = []any{Order{}, OrderLine{}}
server.MustRegister(AuthModels, OrderModels) // both flattened
```
A `ModelConfig` value applies to the model immediately preceding it.
## Common patterns
### Public sign-up (scope auth away from it; inclusion-only)
```go
// Protect updates/deletes everywhere; protect creates only where needed.
// "User" is not scoped, so POST /users (sign-up) stays public.
server.Pipeline.Auth.Register(auth.JWTAuth(secret),
maniflex.ForOperation(maniflex.OpUpdate, maniflex.OpDelete))
server.Pipeline.Auth.Register(auth.JWTAuth(secret),
maniflex.ForModel("Post", "Comment"), maniflex.ForOperation(maniflex.OpCreate))
```
### Hash password on User create/update
```go
server.Pipeline.Service.Register(service.HashField("password", svcbcrypt.Hasher()), // import middleware/service/bcrypt
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate))
```
### Multi-tenancy
```go
server.Pipeline.DB.Register(db.Tenancy("org_id",
func(ctx *maniflex.ServerContext) string { return ctx.Auth.TenantID }))
```
### Audit every write
```go
server.Pipeline.DB.Register(db.AuditLog(sink, db.WithChanges()),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete))
// Note: WithChanges() requires default Before position, NOT After.
```
### Transactional create with stock lock
```go
server.Pipeline.Service.Register(maniflex.WithTransaction(nil),
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
book, err := ctx.LockForUpdate("Book", bookID.(string))
if err != nil { return err }
if book["stock"].(int64) < 1 {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "")
return nil
}
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
```
### Custom default sort
```go
server.Pipeline.Deserialize.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { return err }
if len(ctx.Query.Sorts) == 0 {
ctx.Query.Sorts = append(ctx.Query.Sorts, maniflex.SortExpr{
Field: "created_at", Direction: maniflex.SortDesc,
})
}
return nil
}, maniflex.ForModel("Article"), maniflex.ForOperation(maniflex.OpList), maniflex.AtPosition(maniflex.After))
```
### Background worker reading registered models
```go
// In a goroutine launched alongside server.Start(). Background code has no
// ServerContext — build one with NewBackground, then use ctx.GetModel.
bg := maniflex.NewBackground(context.Background(), server.DB(), server.Registry())
events := bg.GetModel("OutboxEvent")
rows, _ := events.List(&maniflex.QueryParams{
Filters: []*maniflex.FilterExpr{{Field: "status", Operator: maniflex.OpEq, Value: "pending"}},
Limit: 20,
})
for _, ev := range rows {
// ... process ...
events.Update(ev["id"].(string), map[string]any{"status": "done"})
}
```
## Project layout (recommended)
Small app — single `main.go`. Past ~5 models, split by responsibility:
```
main.go wiring only (4 steps)
config.go maniflex.Config assembly
models/ one file per model
middleware/ custom middleware + register.go
actions/ custom action handlers
internal/ framework-free business logic
```
Large monolith — split by domain:
```
domains/auth/ models.go + middleware.go + register.go (exports Models []any and Register(s))
domains/orders/ ditto
domains/catalog/ ditto
main.go server.MustRegister(auth.Models, orders.Models, catalog.Models)
auth.Register(server); orders.Register(server); catalog.Register(server)
```
## Hard rules (ordered by frequency of being violated)
1. **Register models before `sqlite.Open` / `postgres.Open`.** Adapter reads
registry at open time.
2. **After `ctx.Abort`, return without `next()`.** Abort only sets `ctx.Response`;
it does not stop the chain.
3. **Soft-deleted rows are auto-filtered from list/read/include.** To see them,
filter on the marker (`?filter=deleted_at:not_null`).
4. **`OpAction` skips Validate/Service/DB.** Per-action middleware list runs
between Auth and the handler. The handler owns body parsing.
5. **`WithTransaction` on Service step requires `maniflex.Before` position** (default).
Or use `maniflex.Replace` on the DB step.
6. **`db.AuditLog(sink, db.WithChanges())` must be registered at `maniflex.Before`
position**, not After — it needs the pre-image.
7. **Encrypted fields can't be `filterable` or `sortable`.** For uniqueness use
`encrypted,unique` (adds HMAC companion).
8. **`AutoMigrate` never drops columns.** Removes need explicit `ALTER TABLE`.
9. **Configure `FileStorage` before any model with `mfx:"file"` is exercised.**
Without it, multipart uploads return 501.
10. **SQLite has no nested transactions.** Calling `BeginTx` inside an active
SQLite tx fails. Reuse `ctx.Tx`.
11. **`Handler()` does not migrate.** Only `Start()` runs auto-migration. When
mounting `Handler()` yourself, call `MigrateOnly`/`AutoMigrate` first.
## Testing pattern
```go
func newTestServer(t *testing.T) (*httptest.Server, *maniflex.Server) {
t.Helper()
server := maniflex.New(maniflex.Config{Port: 0, PathPrefix: "/api"})
server.MustRegister(User{}, Post{})
db, err := sqlite.Open(":memory:", server.Registry())
if err != nil { t.Fatal(err) }
t.Cleanup(func() { db.Close() })
server.SetDB(db)
middleware.Register(server)
// Handler() does not migrate — only Start() does. Migrate explicitly.
if err := server.MigrateOnly(context.Background()); err != nil { t.Fatal(err) }
ts := httptest.NewServer(server.Handler())
t.Cleanup(ts.Close)
return ts, server
}
```
In-memory SQLite per test. `server.Handler()` returns the chi router but does
**not** migrate — `MigrateOnly` validates and seals the fully configured app,
then creates the tables up front.
## Sentinels & constants
<!-- doccheck:ignore reason="symbol catalogue lists independent expressions and types rather than one executable fragment" -->
```go
maniflex.OpCreate, maniflex.OpRead, maniflex.OpUpdate, maniflex.OpDelete, maniflex.OpList, maniflex.OpOptions, maniflex.OpAction
maniflex.Before, maniflex.After, maniflex.Replace
maniflex.OpEq, maniflex.OpNeq, maniflex.OpGt, maniflex.OpGte, maniflex.OpLt, maniflex.OpLte, maniflex.OpLike, maniflex.OpILike, maniflex.OpIn, maniflex.OpNotIn, maniflex.OpIsNull, maniflex.OpNotNull, maniflex.OpBetween, maniflex.OpContains, maniflex.OpStartsWith, maniflex.OpEndsWith
maniflex.SortAsc, maniflex.SortDesc
maniflex.OnDeleteCascade, maniflex.OnDeleteSetNull, maniflex.OnDeleteRestrict, maniflex.OnDeleteNoAction
maniflex.IdentityHuman, maniflex.IdentityServiceAccount, maniflex.IdentityAnonymous
maniflex.SoftDeleteTimestamp, maniflex.SoftDeleteBool
maniflex.SchedSoftDelete, maniflex.SchedHardDelete, maniflex.SchedSetField
maniflex.ErrNotFound // sentinel error
*maniflex.ErrConstraint // typed error with Table, Column, Detail
maniflex.ErrNoAdapter
maniflex.ErrFileNotFound
```
## What maniflex does NOT do
- No codegen. Reflection at registration only.
- No GraphQL.
- No built-in CLI. Use standard `go` tooling.
- No multi-database per server instance. One adapter per `*maniflex.Server`.
- No automatic column drops in AutoMigrate. Manual `ALTER TABLE` required.
- No SQLite nested transactions.
- No magic — every behaviour is a function in the `maniflex` package or a
catalogue middleware. Read the source when in doubt.