maniflex
Annotated Go structs in. A full REST API out.
maniflex is a Go framework that turns a plain struct into a production REST API —
filtering, pagination, relations, soft-delete, file uploads, and an OpenAPI 3.1
spec — with no generated code and no per-endpoint boilerplate. Behaviour is
declared with mfx: struct tags and customised through a composable
six-step middleware pipeline.
type Post struct {
maniflex.BaseModel
maniflex.WithDeletedAt
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,enum:draft|published|archived"`
UserID string `json:"user_id" mfx:"required,filterable"`
}
Register that struct and you get GET/POST /posts, GET/PATCH/DELETE /posts/{id},
?filter=status:eq:published, ?sort=created_at:desc, ?page=2&limit=20,
?include=user, soft-delete semantics, and an entry in /openapi.json.
Why maniflex
- Reflection, not codegen. Register a struct at startup; routes, schema, and validation are derived from it at runtime. Nothing to regenerate when a field changes.
- One module, two dependencies. The core
maniflexmodule depends only on chi and uuid. Postgres, Redis, Kafka, NATS, bcrypt and friends live in satellite modules you pull in only if you import them. - A pipeline you can reach into. Every request flows through six ordered
steps — Auth → Deserialize → Validate → Service → DB → Response. Hook
middleware
Before,After, orReplaceat any step, scoped by model or operation. - Batteries included, swappable. Ready-made middleware for JWT auth, unique validation, password hashing, multi-tenancy, audit logging, CORS, and more — each one just a function you can replace.
- SQLite or Postgres. Both backends share one SQL adapter. Develop against pure-Go SQLite (no CGo, no external service), deploy on Postgres.
Core concepts
Five ideas carry the whole framework. Understand these and the rest of the docs slot into place.
Model
A Go struct that embeds maniflex.BaseModel and declares its fields with mfx:
struct tags. The struct is the single source of truth: it defines the database
table, the JSON request and response shapes, the validation rules, and which
fields are filterable or sortable. Optional embeds add behaviour — maniflex.WithDeletedAt
turns on soft-delete — and naming conventions like a UserID field declare
relations.
→ Models & BaseModel, Field Tags Reference,
Relations
Registry
The collection of every registered model, built by MustRegister at startup.
It is consumed in two places: the HTTP router reads it to mount routes, and the
DB adapter reads it to run migrations and resolve relations. This is why
MustRegister must run before sqlite.Open / postgres.Open — the adapter
is handed the populated registry.
→ Getting Started
Pipeline
Six ordered steps every request flows through: Auth → Deserialize → Validate → Service → DB → Response. The pipeline is the unit of customisation — instead of writing handlers, you attach middleware to the step where your logic belongs. → Pipeline Overview, ServerContext
Middleware
A func(ctx *maniflex.ServerContext, next func() error) error registered on a pipeline
step. Registration is scoped with maniflex.ForModel(...) and maniflex.ForOperation(...),
and positioned with maniflex.AtPosition(maniflex.After) (or maniflex.Before, the default, or maniflex.Replace). Set
ctx.Response and return without calling next() to short-circuit the request.
→ Writing Middleware, Middleware Catalogue
Adapter
The database backend implementing the storage interface. Two ship in-tree —
db/sqlite (pure-Go, no CGo) and db/postgres — and both share one SQL core.
Inject one with server.SetDB(db), which patches the pipeline’s DB step in
place.
→ Database Backends, Transactions
Where to go next
- Getting Started — install, define your first model, run the server.
- Models & Tags — the full
mfx:tag reference and relation conventions. - The Pipeline — how requests flow and where to hook in.
- Middleware Catalogue — ready-made middleware for every step.
- Querying the API — filtering, sorting, pagination, includes.
maniflex requires Go 1.25 or newer.
Getting Started
This guide takes you from an empty directory to a running REST API with a
filterable, paginated posts resource — in about five minutes and roughly
thirty lines of Go.
Prerequisites
- Go 1.25 or newer — check with
go version. - That’s it. The first example uses the pure-Go SQLite backend, so there is no database server to install and no CGo toolchain to configure.
1. Create the project
mkdir blog && cd blog
go mod init blog
go get github.com/xaleel/maniflex
maniflex itself pulls in only two dependencies — chi
and uuid. The SQLite adapter lives in its own
satellite module, so add it explicitly:
go get github.com/xaleel/maniflex/db/sqlite
2. Define a model
A model is a plain struct that embeds maniflex.BaseModel. The mfx: struct tags
declare how each field behaves — what’s required, what can be filtered, what
can be sorted.
Create main.go:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,enum:draft|published|archived"`
}
maniflex.BaseModel contributes the id, created_at, and updated_at fields, so
you never declare them yourself. See Field Tags Reference for every
tag and Models & BaseModel for the embeds.
3. Wire up the server
Registration order matters: models must be registered before the database is opened, because the SQLite adapter needs the registry to run migrations and resolve relations.
func main() {
// 1. Create the server — no DB yet.
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
// 2. Register models — this populates the registry.
server.MustRegister(Post{})
// 3. Open SQLite with the populated registry.
db, err := sqlite.Open("./blog.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
// 4. Inject the adapter into the pipeline.
server.SetDB(db)
// 5. Serve.
log.Fatal(server.Start())
}
Migration (on by default) creates and updates tables to match your structs on
startup — convenient for development. For an in-memory database that resets on
every run, pass ":memory:" to sqlite.Open.
4. Run it
go run .
The server is now listening on :8080, and Post{} has a full set of routes
mounted under the /api prefix:
| Method | Path | Action |
|---|---|---|
POST | /api/posts | create a post |
GET | /api/posts | list posts |
GET | /api/posts/{id} | read one post |
PATCH | /api/posts/{id} | update a post |
DELETE | /api/posts/{id} | delete a post |
5. Make some requests
Create a post:
curl -X POST localhost:8080/api/posts \
-H 'Content-Type: application/json' \
-d '{"title":"Hello","body":"First post","status":"published"}'
List, filter, sort, and paginate — all from the query string:
# Only published posts
curl 'localhost:8080/api/posts?filter=status:eq:published'
# Newest first, ten per page
curl 'localhost:8080/api/posts?sort=created_at:desc&page=1&limit=10'
Filtering and sorting only work on fields tagged filterable / sortable —
that’s why Title and Status carry those tags above. The full filter grammar
is in Querying.
What you get for free
From that one struct, maniflex derived:
- Five REST endpoints with JSON request/response handling.
- Field validation (
required,enum) on every write. - Query-string filtering, sorting, and pagination.
- A generated table kept in sync by
AutoMigrate. - An OpenAPI 3.1 entry at
/api/openapi.json.
No generated code, no per-endpoint handlers.
Where to go next
- Quickstart Tutorial — build a small app end to end.
- Models & BaseModel — relations, soft-delete, file fields.
- The Request Pipeline — the six steps every request flows through, and where to hook in your own middleware.
- Querying — the full filter, sort, and
includegrammar. - Database Backends — switching from SQLite to PostgreSQL.
App Anatomy
A maniflex app has very little machinery of its own — the framework derives routes, schema, and validation from your structs, so the code you write is mostly models and middleware. This page shows how to lay that code out, starting from a single file and growing into packages as the app gets bigger.
The smallest app
A maniflex app is just a package main that does four things in order:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
type Message struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
}
func main() {
server := maniflex.New(maniflex.Config{Port: 8080, PathPrefix: "/api"})
server.MustRegister(Message{})
if db, err := sqlite.Open("./app.db", server.Registry()); err == nil {
defer db.Close()
server.SetDB(db)
} else {
log.Fatal(err)
}
log.Fatal(server.Start())
}
Four steps — create server → register models → open and set DB → serve.
Everything below is about where the code for each step lives as the app grows.
The ordering is load-bearing: MustRegister must run before sqlite.Open,
because the adapter is handed the populated registry. See
Getting Started.
A typical project layout
Once an app has more than a handful of models, split it into packages by responsibility, not by model. A layout that scales well:
myapp/
├── go.mod
├── main.go # wiring only: create, register, set DB, serve
├── config.go # maniflex.Config assembly, env-var reading
├── models/ # one file per model — the structs and their tags
│ ├── user.go
│ ├── post.go
│ └── comment.go
├── middleware/ # custom pipeline middleware
│ ├── auth.go
│ ├── audit.go
│ └── register.go # attaches all middleware to the pipeline
└── internal/ # non-framework code: services, clients, helpers
└── mailer/
└── ...
Nothing here is enforced by the framework — maniflex never scans directories. It is a convention that keeps each file answering one question.
What goes in each file
main.go — wiring, nothing else
main.go should read top-to-bottom as the four-step sequence and contain no
business logic. Its whole job is to assemble the pieces and call Start():
func main() {
server := maniflex.New(config.Load())
server.MustRegister(
models.User{},
models.Post{},
models.Comment{},
)
db, err := sqlite.Open("./app.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
middleware.Register(server) // all pipeline hooks, in one place
log.Fatal(server.Start())
}
If you can’t see the four steps at a glance, something belongs in another file.
config.go — building maniflex.Config
Keep maniflex.Config construction — and any environment-variable reading — out of
main.go. A single Load() function makes the app’s knobs easy to find:
package config
func Load() maniflex.Config {
return maniflex.Config{
Port: envInt("PORT", 8080),
PathPrefix: "/api",
DisableAutoMigrate: env("APP_ENV", "dev") == "production",
}
}
See Configuration for every maniflex.Config field.
models/ — one file per model
Each file holds one struct, its mfx: tags, and the relation fields that point
at other models. This is the heart of the app — the struct is the table, the
JSON shape, and the validation rules all at once.
// models/post.go
package models
type Post struct {
maniflex.BaseModel
maniflex.WithDeletedAt // opt-in soft-delete
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,enum:draft|published|archived"`
UserID string `json:"user_id" mfx:"required,filterable"` // BelongsTo User
Comments []Comment `json:"comments,omitempty"` // HasMany
}
Put model-spanning relations in whichever file is the “owning” side and let Go’s
package scope resolve the rest — all models share the models package, so
Post can reference Comment freely. See Models & BaseModel,
Field Tags Reference, and Relations.
middleware/ — your pipeline hooks
A middleware is a func(ctx *maniflex.ServerContext, next func() error) error. Group
related hooks per file (auth.go, audit.go), and keep one register.go that
attaches them all — so there is exactly one place to see how the request
pipeline has been customised:
// middleware/register.go
package middleware
func Register(s *maniflex.Server) {
s.Pipeline.Auth.Register(bearerToken,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete))
s.Pipeline.Service.Register(hashPassword,
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate))
s.Pipeline.DB.Register(auditLog,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After))
}
The middleware functions themselves live in their topic files. See Writing Middleware and the Middleware Catalogue for hooks that already ship with the framework.
internal/ — everything that isn’t maniflex
Code that has nothing to do with the framework — a mail client, a payment SDK
wrapper, domain calculations — goes under internal/. Middleware in the
Service step calls into these packages; the packages themselves never import
maniflex. This keeps the framework-facing layer thin and your business logic
unit-testable on its own.
Structuring a large monolith
The layer-based layout above (models/, middleware/) stays readable up to a
few dozen models. Past that, a flat models/ directory with sixty files and a
main.go that names every one of them becomes the bottleneck. Large maniflex
codebases switch from splitting by layer to splitting by domain.
Domain packages
Give each business domain its own package that owns all of its code — models, middleware, and business logic together:
myapp/
├── main.go
├── domains/
│ ├── auth/
│ │ ├── models.go # User, Role, Session, ApiKey
│ │ ├── middleware.go # token auth, password hashing
│ │ └── register.go # exports Models and Register(s)
│ ├── catalog/
│ │ ├── models.go # Product, Category, Variant
│ │ ├── middleware.go
│ │ └── register.go
│ └── orders/
│ ├── models.go # Order, LineItem, Invoice, Refund
│ ├── middleware.go
│ └── register.go
└── internal/
A new feature now touches one directory instead of being smeared across
models/, middleware/, and internal/.
Registering models in groups
server.Register (and MustRegister) is variadic and flattens any slice
argument — so each domain can export its models as a single slice, and
main.go registers the slices side by side:
// domains/auth/register.go
package auth
// Models is every model this domain owns. One list, one place to update.
var Models = []any{
User{},
Role{},
Session{},
ApiKey{},
}
// main.go
server.MustRegister(
auth.Models, // each argument is a []any —
catalog.Models, // Register flattens them into individual models
orders.Models,
)
main.go no longer grows when a domain gains a model; only that domain’s
Models slice changes. To register everything as one list instead, concatenate
the slices: append(append(auth.Models, catalog.Models...), orders.Models...).
Registering middleware in groups
Apply the same idea to the pipeline. Each domain exposes its own
Register(s *maniflex.Server), and the top-level middleware registration just calls
each one — exactly the shape in the request from a growing app:
// domains/orders/register.go
package orders
// Register attaches every pipeline hook this domain needs.
func Register(s *maniflex.Server) {
s.Pipeline.Validate.Register(checkStock, maniflex.ForModel("Order"))
s.Pipeline.Service.Register(chargePayment,
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
s.Pipeline.DB.Register(emitOrderEvent,
maniflex.ForModel("Order"), maniflex.AtPosition(maniflex.After))
}
// main.go (or a thin top-level middleware/register.go)
func registerMiddleware(s *maniflex.Server) {
auth.Register(s)
catalog.Register(s)
orders.Register(s)
}
Each domain controls its own pipeline hooks; the top-level function is just a table of contents.
Co-locating per-model middleware
For middleware that belongs to exactly one model, skip the separate Register
call entirely — ModelConfig.Middleware lets you attach hooks at registration
time, scoped to that model automatically:
server.MustRegister(
Order{}, maniflex.ModelConfig{
Middleware: &maniflex.ModelMiddleware{
Validate: []maniflex.MiddlewareFunc{checkStock},
Service: []maniflex.MiddlewareFunc{chargePayment},
},
},
)
This keeps a model and its rules in one declaration — useful when the hook is meaningless without the model.
Conventions that keep a big monolith honest
- One registration point per concern. Exactly one place lists the model groups, and one lists the middleware groups. If you can’t find where a model is registered, the structure has drifted.
- Domains depend inward, never sideways. A domain may import
internal/andmaniflex; it should not import a sibling domain. Cross-domain relations are expressed by FK fields (a stringUserID), which need no import. internal/holds framework-free logic. Payment, mail, and pricing code lives here and never importsmaniflex, so it stays unit-testable in isolation.main.gostays a fixed size. Adding a domain adds one line to each registration list and nothing else. Ifmain.gogrows with the app, logic has leaked into it.
How a request moves through the files
Tracing one POST /api/posts shows why the layout is split this way:
- The router (built from the registry in
main.go) matches the route. - The request enters the pipeline —
Auth → Deserialize → Validate → Service → DB → Response. - At each step, the hooks from
middleware/register.gorun, scoped by model and operation. Validatechecks themfx:tags declared inmodels/post.go.Servicemiddleware may call intointernal/for business logic.- The adapter injected via
SetDBruns the SQL at theDBstep.
Each file owns one stage of that journey — which is exactly why a growing app stays readable.
Where to go next
- Example 1: Simple Blog — this layout filled in for a real three-model app.
- The Request Pipeline — the six steps in depth.
- Configuration — every
maniflex.Configfield.
Example 1: Simple Blog
This is the first worked example: a small blog API built end to end. It uses
only what the previous pages covered — models and mfx: tags,
the four-step setup, and SQLite. No relations, no middleware, no pipeline
customisation yet; those arrive in later chapters. The goal is to see a complete,
runnable app with nothing unexplained in it.
What we’re building
A blog with two independent resources:
- Post — an article with a title, body, and a publication status.
- Subscriber — an email address signed up for the newsletter.
The two are unrelated — each is its own table with its own endpoints — which keeps the example to concepts already introduced.
The whole app
The blog is small enough to live in a single main.go, the
smallest-app shape from App Anatomy:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
// Post is a blog article.
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,filterable,sortable,enum:draft|published|archived"`
}
// Subscriber is a newsletter sign-up.
type Subscriber struct {
maniflex.BaseModel
Email string `json:"email" mfx:"required,filterable"`
Name string `json:"name" mfx:"filterable,sortable"`
}
func main() {
// 1. Create the server.
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
// 2. Register both models — populates the registry.
server.MustRegister(Post{}, Subscriber{})
// 3. Open SQLite with the populated registry, then inject it.
db, err := sqlite.Open("./blog.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
// 4. Serve.
log.Fatal(server.Start())
}
That is the entire blog. Run it:
go run .
Reading the models
Every field choice maps to a tag covered in Getting Started:
| Field | Tags | Effect |
|---|---|---|
Title | required,filterable,sortable | must be present; usable in ?filter= and ?sort= |
Body | required | must be present; not queryable |
Status | required,...,enum:draft|published|archived | rejected unless one of the three values |
Email | required,filterable | must be present; filterable but not sortable |
Name | filterable,sortable | optional; queryable and sortable |
maniflex.BaseModel adds id, created_at, and updated_at to both structs, so
those are never declared by hand. With migration on by default, the posts and
subscribers tables are created to match on startup.
The endpoints you get
Registering the two structs mounts a full REST surface under /api:
| Method | Path | |
|---|---|---|
POST | /api/posts | create a post |
GET | /api/posts | list posts |
GET | /api/posts/{id} | read one post |
PATCH | /api/posts/{id} | update a post |
DELETE | /api/posts/{id} | delete a post |
Subscriber gets the identical five routes under /api/subscribers.
Using the API
Create a post
curl -X POST localhost:8080/api/posts \
-H 'Content-Type: application/json' \
-d '{"title":"Hello World","body":"My first post","status":"draft"}'
The response echoes the stored row, including the id and timestamps that
BaseModel filled in.
Validation in action
Leave out a required field, or send a status outside the enum, and the
write is rejected before it reaches the database:
# Missing body, and status is not in the enum
curl -X POST localhost:8080/api/posts \
-H 'Content-Type: application/json' \
-d '{"title":"Broken","status":"weekly"}'
# → 400, the response names the offending fields
Update and delete
# Publish the post — PATCH only sends the fields that change
curl -X PATCH localhost:8080/api/posts/<id> \
-H 'Content-Type: application/json' \
-d '{"status":"published"}'
curl -X DELETE localhost:8080/api/posts/<id>
List, filter, sort, paginate
All from the query string, on the fields tagged filterable / sortable:
# Only published posts
curl 'localhost:8080/api/posts?filter=status:eq:published'
# Newest first
curl 'localhost:8080/api/posts?sort=created_at:desc'
# Page two, five per page
curl 'localhost:8080/api/posts?page=2&limit=5'
# Combine them
curl 'localhost:8080/api/posts?filter=status:eq:published&sort=title:asc&limit=5'
Add a couple of subscribers and the same querying works there too:
curl -X POST localhost:8080/api/subscribers \
-H 'Content-Type: application/json' \
-d '{"email":"[email protected]","name":"Ada"}'
curl 'localhost:8080/api/subscribers?sort=name:asc'
The API documents itself
You never wrote a schema, yet the server already publishes one. Alongside the
model routes, maniflex auto-generates an OpenAPI 3.1 specification describing
every endpoint, field, and validation rule — derived from the same structs and
mfx: tags:
curl localhost:8080/api/openapi.json
The spec updates itself whenever a model changes; there is nothing to
regenerate. To browse it as interactive documentation, the framework ships an
HTML viewer in static/openapi.html that loads /api/openapi.json — open
http://localhost:8080/static/openapi.html while the server is running.
The OpenAPI step is fully customisable later; see OpenAPI Spec.
What this example showed
- A complete, runnable app from two plain structs and a four-step
main. mfx:tags driving validation (required,enum) and queryability (filterable,sortable).- Multiple models registered in one call, each with an independent REST surface.
- Filtering, sorting, and pagination with no query code written.
- A self-updating OpenAPI 3.1 spec at
/api/openapi.json.
Where to go next
Everything here treated the two models as separate islands. Real apps connect
them — a post belongs to an author, a comment belongs to a post. That, and
the mfx: tags beyond the basics, are the next chapters:
- Models & BaseModel — the embeds and what they contribute.
- Field Tags Reference — every
mfx:tag. - Relations — connecting models with foreign keys.
Architecture
This page explains how maniflex is put together. The reference pages describe each piece in isolation; here we look at how they fit. A reader who finishes this page should be able to point at any other doc and predict roughly what it covers.
The five pieces
┌──────────────────────┐
register → │ Registry │ ← models discovered here
└──────────────────────┘
│ │
│ ▼
│ ┌──────────────────────┐
│ │ Database adapter │
│ │ (sqlite | postgres) │
│ └──────────────────────┘
▼
┌──────────────────────┐
│ Router │ ← chi v5
└──────────────────────┘
│
▼
HTTP request → ┌──────────────────────────────────────────────────┐
│ Pipeline: │
│ Auth → Deserialize → Validate → │
│ Service → DB → Response │
└──────────────────────────────────────────────────┘
│
▼
HTTP response ← APIResponse envelope
The framework has five primary moving parts:
| Piece | What it is | Where it lives |
|---|---|---|
| Registry | An in-memory map of every registered model’s *ModelMeta — fields, tags, relations, indices, scheduled specs. | built by MustRegister, consumed by the router and the adapter |
| Router | A chi v5 Router mounted with one sub-router per registered model. | router.go |
| Pipeline | Six ordered steps that every model-route request flows through, plus a parallel three-step pipeline for /openapi.json. | pipeline.go |
| ServerContext | The single per-request struct threaded through every step. | context.go |
| DBAdapter | The backend interface implemented by db/sqlite, db/postgres, and any custom backend. | db.go |
Every other concept in the docs sits on top of these five.
Reflection, not codegen
The framework derives everything from the registered structs at startup:
ScanModelwalks the struct withreflectonce per model, builds a*ModelMeta, and inserts it into the registry.- The adapter reads the registry to emit
CREATE TABLE/ALTER TABLEstatements duringAutoMigrate. - The router reads the registry to mount the five REST routes per model
plus
/openapi.json. - The Validate step reads each request’s model meta to enforce
mfx:tag rules. - The DB step reads each request’s model meta to assemble the SQL.
Reflection runs once at registration, never per request. The per-request
path is allocation-light: a map[string]any for the body, a few string keys,
and the slice of registered middleware filtered by ForModel /
ForOperation. A model with N fields produces O(N) work at boot and O(N)
work per request, both linear in the size of the model.
This is the architectural difference from codegen frameworks: there is no
generated file to keep in sync. Changing a mfx: tag changes the runtime
behaviour the next time the process starts.
The registry is the contract
Every other piece reads from the registry; nothing writes to it after
Start(). That single rule explains several constraints:
MustRegistermust run beforesqlite.Open/postgres.Open. The adapter reads the registry during its constructor to learn about tables and relations.- Models cannot be added or removed at runtime. New models require a process restart.
- Models can be inspected from middleware via
ctx.Model, which is the*ModelMetathe router selected for this request. - Cross-model operations work because middleware reaches into the
registry through
ctx.GetModel(name)or by name in scoped registration.
The framework’s startup sequence is deliberate:
1. maniflex.New(cfg) → empty registry
2. server.MustRegister(...) → populated registry
3. sqlite.Open(..., reg) → adapter built from registry
4. server.SetDB(db) → adapter wired into the DB step
5. middleware.Register(...) → pipeline customised
6. server.Start() → router built from registry, listener opens
Start() runs AutoMigrate (if enabled) before opening the listener, so a
schema mismatch fails fast instead of corrupting writes.
The pipeline is the unit of customisation
Every HTTP request to a model route is wrapped in a ServerContext and run
through six steps in this order:
| Step | Default behaviour |
|---|---|
| Auth | passthrough |
| Deserialize | parse query string + body |
| Validate | enforce mfx: tag rules |
| Service | passthrough — business logic goes here |
| DB | dispatch to the adapter |
| Response | write the JSON envelope |
Each step has its own StepRegistry on server.Pipeline. A registration
attaches a MiddlewareFunc at Before (the default), After, or Replace
position, scoped by ForModel / ForOperation. At request time the
registry returns the matching chain for the (model, operation) pair, the
chain runs, and any step can short-circuit by setting ctx.Response and
returning without calling next().
The pipeline is the answer to “where do I put X?”:
- Identity checks — Auth.
- Coerce types, strip unknown fields — Validate (Before).
- Hash passwords, set derived fields — Service.
- Bracket the DB call — DB (Before / After).
- Webhooks, events, audit log — DB (After).
- Headers, redactions, metrics — Response.
The same answer applies whether the code lives in a catalogue middleware,
a custom function, or a per-model ModelConfig.Middleware.
The adapter is one interface
maniflex.DBAdapter has fewer than a dozen methods — FindByID, FindMany,
Create, Update, Delete, BeginTx, Raw, Ping, plus the schema
operations called by AutoMigrate. Two implementations ship:
db/sqlite— pure-Go SQLite for development.db/postgres—lib/pqfor production.
Both implementations share db/sqlcore, a SQL adapter that knows about
filters, sorts, includes, soft delete, and relations. A custom backend — an
HTTP data service, a different SQL database — implements the same interface
and is injected with server.SetDB(myAdapter). No other code changes.
The same holds for FileStorage and KeyProvider: small interfaces with
shipped implementations and obvious extension points.
Satellite modules
maniflex is a multi-module repository. The core module imports only chi and
uuid. Everything heavier — a database driver, a JWT library, a Kafka client,
bcrypt — lives in its own satellite under the same root, so a consumer
pulls in only the dependencies it actually imports.
The split keeps the core small and stable. It also keeps the trust boundary
clear: the framework’s surface area is the maniflex package; everything in
middleware/*, events/*, jobs/*, db/* is application code that
happens to ship alongside the framework.
See Satellite Modules for the full layout and import rules.
Two pipelines, one router
The router actually mounts two pipelines.
The first one — the six-step pipeline described above — handles
/<table> and /<table>/{id} for every registered model.
The second is a three-step pipeline for GET /openapi.json:
OpenAPI.Auth → OpenAPI.Generate → OpenAPI.Response
Generate derives the spec from the registry every time the endpoint is
hit, then Response serialises it. After-position middleware on Generate
can mutate the spec — change titles, add servers, install security schemes,
or rewrite arbitrary fields. See OpenAPI Spec and the
OpenAPI Middleware catalogue.
A third, trimmed pipeline runs for custom actions:
Auth → [per-action middleware] → handler → Response
The Deserialize, Validate, Service, and DB steps are skipped — actions own their body parsing and database work.
Where each feature lives in the lifecycle
| Feature | Step(s) | Notes |
|---|---|---|
mfx: tag rules (required, enum, min, …) | Validate | per-field |
| Required-on-create, immutable-on-update, readonly-strip | Validate | tied to Operation |
mfx:"file" multipart parsing | Deserialize | populates ctx.Files |
| File storage write | Service (built-in) | writes the storage key |
| Soft-delete filter on reads | DB | adapter rewrites the SQL |
mfx:"encrypted" envelope + HMAC | DB (Before for writes, after for reads) | needs KeyProvider |
Versioned history row | DB (Before for pre-image, After for write) | sibling _history table |
mfx:"scheduled" sweep | outside the request — separate runner | see Scheduled Fields |
?filter=…&sort=…&include=… parsing | Deserialize | into ctx.Query |
?include= population | DB | secondary queries after the main SELECT |
Auto-tenant filter (db.Tenancy) | DB (Before) | appends to ctx.Query.Filters |
| Audit log | DB (Before) | needs the pre-image; writes outside the request |
maniflex.WithTransaction | Service (Before) or DB (Replace) | wraps the DB step |
LockForUpdate | inside the DB step’s transaction | SELECT ... FOR UPDATE on Postgres |
For a single concrete trace of every step running on one request, see the Request Lifecycle walkthrough.
What maniflex is not
To set expectations on the architecture choice:
- Not codegen. No generated files, no separate build step.
- Not a router framework. The HTTP layer is chi; maniflex uses it.
- Not opinionated about JSON shape. The envelope is the default, but
response.Envelopelets you replace it. Errors always use the error envelope. - Not a service mesh. One process, one binary. Multi-process concerns (events, jobs, distributed locks) live in the satellite modules.
- Not magic. Every behaviour is a function in the
maniflexpackage or one of the catalogue middlewares. Read the source when in doubt; the pipeline is small.
Next
- Request Lifecycle — a single
POST /api/orderstraced end-to-end through every step. - Glossary — every framework term in one place.
- Pipeline Overview — the per-step reference.
Request Lifecycle
This page traces a single request through every piece of the framework. The
example is a POST /api/orders on an authenticated user, with a Service
middleware that hashes a derived field, a transaction wrapping the DB step,
and an audit-log middleware on DB-After. It exercises the full pipeline
without being contrived.
Setup
server.MustRegister(models.Order{})
server.Pipeline.Auth.Register(auth.JWTAuth("secret"))
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
server.Pipeline.Service.Register(
service.SetField("customer_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.UserID
}),
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate),
)
server.Pipeline.DB.Register(
db.AuditLog(sink, db.WithChanges()),
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
server.Pipeline.Response.Register(response.CORSHeaders("https://app.example.com"))
The request
POST /api/orders HTTP/1.1
Authorization: Bearer eyJ...
Content-Type: application/json
Idempotency-Key: 2af9...
{"total": 42.50, "status": "pending"}
What happens, in order
1. The router selects the route
chi matches POST /api/orders to the sub-router mounted by mountModel
for the Order model. The matched handler calls handler.Create(meta),
which:
- Allocates a fresh
*ServerContext. - Sets
ctx.Request,ctx.Writer,ctx.Ctx. - Reads the
X-Request-Idchi added in the outer middleware and stores it onctx.RequestID. - Reads the
traceparentheader (if present) intoctx.TraceID. - Sets
ctx.Model = metaforOrder. - Sets
ctx.Operation = OpCreate. - Leaves
ctx.ResourceIDempty — create has no path id. - Calls
Pipeline.execute(ctx).
If Config.QueryTimeout is non-zero, ctx.Ctx is wrapped in a
context.WithTimeout here, so every downstream DB call inherits the
deadline.
2. The Auth step runs
Pipeline.Auth.build("Order", OpCreate) returns a chain consisting of
matching middleware in registration order, then the default Auth handler at
the end (which is a passthrough).
For our setup the chain is just [auth.JWTAuth, defaultAuth]. JWTAuth:
- Reads the
Authorizationheader. - Verifies the signature with the configured secret.
- Parses the claims and populates
ctx.Auth = &AuthInfo{UserID: …, Roles: …, TenantID: …}. - Calls
next()to continue.
A missing or invalid token would have produced ctx.Abort(401, "UNAUTHORIZED", …)
and returned without next(), ending the request right here.
3. The Deserialize step runs
The default handler parses two things:
- Query parameters →
ctx.Query(a*QueryParams). For a create, this is mostly empty — there are nofilterorsortto read. - Body. The
Content-Typeisapplication/json, so it reads up to 4 MB fromctx.Request.Body, setsctx.RawBodyto the raw bytes, and parses the JSON into the read-onlyctx.ParsedBody(a*RequestBody):
// ctx.ParsedBody now holds { "total": 42.5, "status": "pending" }
total, _ := ctx.Field("total") // 42.5
status, _ := ctx.Field("status") // "pending"
The same values are bound to the typed record ctx.Record; middleware mutate
either through ctx.SetField / ctx.DeleteField.
If the Content-Type had been multipart/form-data, the default handler
would route through parseMultipart instead, populating ctx.ParsedBody
with form fields and ctx.Files with the file parts.
After-position middleware on Deserialize (e.g. idempotency.Middleware,
which sees ctx.RawBody) runs next. With idempotency configured, the
middleware would compute a body hash and either replay a cached response or
fall through to step 4.
4. The Validate step runs
The default handler iterates ctx.Model.Fields and applies the mfx: tag
rules to ctx.ParsedBody:
idis stripped — the adapter assigns it.readonlyfields (created_at,updated_at) are stripped.immutablefields are stripped ifOpUpdate— not on create.requiredfields must be present.totalandstatusare required; the request supplies both, so no error.enummembership is checked onstatus—"pending"is in the allowed set, so no error.min/maxare checked on numeric fields when present.
If any rule had failed, the step would have called ctx.Abort(422, "VALIDATION_ERROR", …)
with details: [...] listing the bad fields.
Custom Validate middleware (none in this example) would run alongside the
default — Before middleware first, then the default, then After.
5. The Service step runs
Two middleware are scoped to this request:
maniflex.WithTransaction(nil) runs first:
- Sees
ctx.Tx == nil. - Calls
ctx.BeginTx(ctx.Ctx, nil), which delegates to the adapter. - Assigns the resulting
Txtoctx.Txand re-wrapsctx.Ctxwith the tx stored undertxContextKey{}. - Defers
tx.Rollback()— a no-op afterCommit. - Calls
next()to run the rest of the pipeline inside the transaction.
service.SetField("customer_id", ...) runs second:
- Resolves the callback against
ctx.Auth.UserID. - Calls
ctx.SetField("customer_id", "user-alice"), writing through to bothctx.ParsedBodyand the typedctx.Record. - Calls
next().
6. The DB step runs
The default DB step calls defaultSteps.db:
- Sees
ctx.Tx != niland constructs adbExec{adapter, tx: ctx.Tx}. - Builds the DB-column write set from the typed
ctx.Record(falling back totoDBMap(ctx.ParsedBody)for bodies the record can’t represent); here the column names match the JSON keys. - If the model had
mfx:"encrypted"fields, callsencryptFieldsto replace plaintexts withenc:<base64>envelopes and write{field}_hmaccompanions for unique ones. - Dispatches by
ctx.Operation:
result, err := exec.Create(ctx.Ctx, model, dbData)
exec.Create on a transactional dbExec calls tx.Create(...), which
runs INSERT INTO orders (...) RETURNING * (Postgres) or
INSERT ... ; SELECT ... (SQLite).
The adapter returns the inserted row as a map[string]any. The DB step
assigns it to ctx.DBResult.
If the adapter returned maniflex.ErrNotFound, the step would abort with
404 NOT_FOUND. *maniflex.ErrConstraint becomes 409 CONFLICT. A
context-cancelled error becomes 504 TIMEOUT when a server-side deadline
expired, or 499 when the client simply hung up. Any other adapter error
becomes 500 DB_ERROR.
6a. Audit-log Before middleware
We registered db.AuditLog at the default Before position (because
WithChanges() needs to read the pre-image). On OpCreate there is no
pre-image, so the middleware merely sets up to collect the post-image. It
calls next(), which runs the rest of the chain — the default DB handler
above.
After next() returns and ctx.Response is still nil (the create
succeeded), the middleware:
- Reads
ctx.DBResultfor the inserted row. - Builds an
AuditRecordwith model, operation, actor (ctx.Auth.UserID), tenant, request id, trace id, and a diff of every changed field. - Spawns a goroutine that calls
sink.Write(bgCtx, record). Audit writes are fire-and-forget — a sink error never fails the request.
7. WithTransaction commits
Control returns to WithTransaction (because we are inside its next()
call). It checks:
next()returned nil → no pipeline error.ctx.Response == nilor< 400→ no aborted step.- Calls
tx.Commit(). The deferredRollbackis now a no-op. - Runs any callbacks registered with
ctx.AfterCommit— the write is durable, so the side effects that were waiting on it may fire. - Clears
ctx.Txso any post-commit code uses the bare adapter.
If next() had returned an error, or if any step had set ctx.Response
to a status >= 400, Commit would have been skipped and the deferred
Rollback would have fired — and the AfterCommit callbacks would have
been dropped rather than run, since the write they announce did not happen.
8. The Response step runs
The default Response handler:
- Sees
ctx.Response == nil. - Sees
ctx.Operation == OpCreate. - Builds:
ctx.Response = &APIResponse{
StatusCode: http.StatusCreated,
Data: toJSONMap(ctx.DBResult.(map[string]any), model),
}
toJSONMap converts DB column names back to JSON field names and applies
hidden and writeonly filtering — any column tagged those is dropped
from the response shape.
After-position middleware on Response runs next. response.CORSHeaders
adds the appropriate Access-Control-* headers via ctx.Writer.Header().
9. The envelope is written to the wire
APIResponse.Write(ctx.Writer):
- Sets
Content-Type: application/json. - Writes the status code header (
201 Created). - Encodes
{"data": {...}}to the response body. - Returns.
chi’s RequestID middleware (registered at the router root, outside the maniflex
pipeline) wraps the whole exchange — it has already set X-Request-Id on the
response by the time we get here. chi’s RealIP is registered alongside it only
when Config.TrustProxyHeaders is set, so RemoteAddr reflects the forwarded
client IP just for servers that opted into trusting proxy headers.
The dispatch cleanup
After the response is written, the handler runs its cleanup phase:
- Closes any open multipart file readers in
ctx.Files. - Removes the multipart temporary directory.
- Lets the
*ServerContextgo out of scope; it is garbage-collected with the request.
The framework does not pool or reuse ServerContext values. The per-request
allocation is small; the simplicity is worth more than the saved
allocations.
What changes for other operations
Different operations exercise slightly different paths:
OpRead/OpList— Validate runs but is a no-op (it only fires for create/update). The DB step callsFindByIDorFindMany. Response for List wraps withmeta: {total, page, limit, pages}.OpUpdate— Like create, butctx.ResourceIDis set,immutablefields are stripped in Validate, and the DB step callsUpdate. The audit middleware fetches the pre-image beforenext()so theChangesdiff has both sides.OpDelete— No body to deserialize, no Validate work. The DB step callsDelete(which becomes a soft-deleteUPDATEfor models withWithDeletedAt). The default Response is204 No Content.OpAction— The trimmed pipeline runsAuth → action middleware → handler → Response. The handler is responsible for its own body parsing, validation, and database calls./openapi.json— A separate three-step pipeline (OpenAPI.Auth → Generate → Response) builds the spec from the registry every time, then writes the JSON document.
What happens on errors
Three error paths at every step:
| Trigger | Effect |
|---|---|
Middleware returns a non-nil error from next() | Bubbles up; later steps are skipped. The chain returns the error to the handler, which logs and writes a 500 INTERNAL envelope. |
Middleware calls ctx.Abort(...) and returns nil without next() | ctx.Response is set; subsequent steps are skipped (because next() was never called); the Response step’s default reads ctx.Response and writes it. |
| Panic anywhere in the chain | PanicRecoverer catches it, logs through Config.PanicLogger, writes a 500 PANIC envelope. |
A transaction in flight is rolled back by WithTransaction’s deferred
Rollback in all three cases — the same code path that handles success.
What this tour did not show
- Multi-tenant scoping with
db.Tenancy— would have appended toctx.Query.Filtersin step 5/6. mfx:"file"uploads — would have parsed multipart in step 3 and written bytes toFileStoragebetween steps 5 and 6.mfx:"scheduled"— fires outside the request, in a separate Scheduled Runner goroutine.mfx:"versioned"— would have written a sibling history row in step 6’s After phase, in the same transaction.- The OpenAPI pipeline — same shape, three steps, separate registrations.
Each of those is covered in its own page; the lifecycle in step-by-step form is the same.
Models & BaseModel
A model is a Go struct registered with the server. From it, maniflex derives a database table, the JSON request and response shapes, the set of REST routes, and the validation applied to every write. This page covers what a struct must contain to be a valid model, how it maps to a table, and the options available at registration. Field-level tags are documented in Field Tags Reference; relationships in Relations.
Definition
A model is an ordinary struct that embeds maniflex.BaseModel:
type Article struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
}
Registration validates the struct and adds it to the registry:
server.MustRegister(Article{})
Register returns an error; MustRegister panics on failure and is intended
for use in main or package initialisation. A struct is rejected at
registration if it is not a struct type or does not embed BaseModel.
Two embedding rules are enforced there as well:
- Embed by value, never by pointer.
*maniflex.BaseModel— or any other pointer embed — is left nil by the record scanner, so every field access through it panics on the first request. It is refused at registration instead. - Two fields cannot map to the same column. A field shadowing one of
BaseModel’s columns produced two entries with the same DB name, and the framework then disagreed with itself about which was meant: lookups took the first, writes took the last. Rename one, or give it an explicitdb:"...".
BaseModel
Every model must embed maniflex.BaseModel. It contributes three columns common to
all tables:
type BaseModel struct {
ID string `json:"id" db:"id"`
CreatedAt time.Time `json:"created_at" db:"created_at" mfx:"readonly,sortable"`
UpdatedAt time.Time `json:"updated_at" db:"updated_at" mfx:"readonly,sortable"`
}
ID— the primary key, a UUID assigned by the framework on create.CreatedAt— set once, when the row is created.UpdatedAt— refreshed on every update.
All three are managed by the framework. CreatedAt and UpdatedAt are
readonly: values supplied for them in a request body are ignored rather than
stored. Because they are part of BaseModel, they are never declared on
individual models.
A struct that does not embed BaseModel — or otherwise lacks an id column —
fails registration.
Field mapping
Each exported field of a model maps to a database column. Three struct tags control the mapping:
| Tag | Purpose |
|---|---|
json | the field’s name in request and response bodies |
db | the column name; defaults to the snake_case field name if omitted |
maniflex | field behaviour — validation, filterability, and so on |
A minimal field needs only a json tag; db is derived and mfx is optional.
The mfx tag is the largest of the three and has its own reference in
Field Tags Reference. Fields that name a related model — for example
a UserID foreign key — are interpreted as relations; see
Relations.
Nullability is the Go type
A pointer field gets a NULL column; everything else gets NOT NULL. No
tag controls this — the type is the whole declaration:
type Note struct {
maniflex.BaseModel
Title string `json:"title"` // NOT NULL — cannot be null
Body *string `json:"body"` // NULL — may be null
}
That decides what a request may send. {"body": null} stores SQL NULL;
{"title": null} is refused with 422 naming the field, because the column has
no null to store:
{
"error": {
"code": "VALIDATION_ERROR",
"details": [{
"field": "title",
"message": "field \"title\" cannot be null; its type has no null value — send a value, omit the field, or make it a pointer to allow null"
}]
}
}
Three things are distinct and stay distinct: null (refused unless the field is
a pointer), "" (a value, stored as written), and omitting the key (leaves
the stored value alone on a PATCH).
If you need to tell “empty” from “not set”, make the field a pointer. A non-pointer field genuinely cannot represent the difference — it reads back as the zero value either way.
Table names
By default the table name is the struct name converted to snake_case and pluralised:
| Struct | Table |
|---|---|
Article | articles |
BlogPost | blog_posts |
Category | categories |
Analysis | analyses |
Matrix | matrices |
A short list of Latin and Greek stems inflect classically — analysis, basis,
axis, diagnosis, thesis (and hypothesis), datum, medium, matrix,
vertex, criterion, phenomenon — including inside a compound name, so
BloodAnalysis becomes blood_analyses. Everything else takes the English
ending: album → albums, complex → complexes, and index → indexes,
since that is the plural the database world uses.
To use a different name, pass a ModelConfig with TableName set when
registering:
server.MustRegister(
Article{}, maniflex.ModelConfig{TableName: "articles"},
)
TableName is also how you pin an existing table whose generated name has
changed — see the v0.3.0 note in the changelog if you have a model named after
one of the stems above.
Registration options
ModelConfig carries per-model options. All fields are optional; an omitted
ModelConfig applies the defaults described above.
| Field | Purpose |
|---|---|
TableName | override the derived table name |
SoftDelete | opt the model into soft deletion — see Soft Delete |
Middleware | pipeline middleware scoped to this model, installed at registration — see Writing Middleware |
Versioned | record field-change history in a sibling {model}_history table |
VersionedDiffOnly | with Versioned, store only changed fields rather than full snapshots |
Indices | additional database indexes created during AutoMigrate |
ExportEnabled | mount GET /:model/export (CSV / XLSX) — see CSV / XLSX Export |
MaxExportRows | row cap for the export endpoint; default 100,000 |
AggregateEnabled | mount GET /:model/aggregate?aggregate=<url-encoded JSON> (grouped count/sum/avg/min/max) — see Aggregations |
OptimisticLock | enable If-Match / ETag concurrency control on PATCH and DELETE |
Adapter | route this model to a separate database adapter |
Singleton | expose the model as a single-row resource (GET / PATCH, no id) — global, or one row per tenant when scoped; see Singleton models |
Headless | register the model fully but mount no REST routes, freeing its path for a custom action — see Serving a model’s own path from an action |
Optimistic locking (OptimisticLock)
When OptimisticLock: true, every PATCH and DELETE request that includes an
If-Match header is checked against the current record’s ETag before the write
executes. A mismatch returns 412 Precondition Failed (PRECONDITION_FAILED).
Requests without If-Match are unaffected — the flag opts in to enforcement,
not mandatory locking.
The ETag format is identical to the one emitted by response.Cache (MD5 of the
JSON response body), so clients can use the header from a preceding GET directly:
server.MustRegister(Invoice{}, maniflex.ModelConfig{OptimisticLock: true})
server.Pipeline.Response.Register(
response.Cache(300),
maniflex.ForModel("Invoice"),
maniflex.ForOperation(maniflex.OpRead),
maniflex.AtPosition(maniflex.After),
)
GET /invoices/42 → 200 ETag: "d41d8cd9..."
PATCH /invoices/42 If-Match: "d41d8cd9..." → 200
PATCH /invoices/42 If-Match: "stale" → 412
PATCH /invoices/42 If-Match: * → 200
PATCH /invoices/99 If-Match: * → 404 (no such record)
If-Match: * is the RFC 9110 wildcard: it holds for any existing record, so it
means “overwrite whatever is there, but do not create it” rather than pinning a
particular version. It still takes the row lock, so it is safe to use on a
contended record — it just does not care which version it lands on.
The check and the write it guards run as a single transaction, with the record
held under a row lock (SELECT … FOR UPDATE on Postgres) from the ETag
comparison until the write commits. Two clients holding the same ETag therefore
cannot both succeed: the loser waits on the lock, then re-reads a record whose
ETag has moved on and gets its 412. When the request already runs inside a
transaction (maniflex.WithTransaction) the guard joins it and the lock is held
until that transaction commits; otherwise the DB step opens and commits one of
its own.
Singleton models (Singleton)
Some resources are inherently single-row: an application config record, a set of
feature flags, the banner an admin edits and every client reads at launch. With
Singleton: true the model drops its collection and item routes and exposes just
two endpoints on the bare table path — no id in the URL:
GET /:model → read the one row
PATCH /:model → update the one row
There is no POST, DELETE, or list endpoint; requesting them returns
405 Method Not Allowed, and there is no /:model/:id subtree.
The single backing row is provisioned lazily under the well-known
maniflex.SingletonID on first access, from each column’s default. So the first
GET returns defaults before anything has been written, and PATCH always
targets an existing row — it behaves like an upsert:
type AppConfig struct {
maniflex.BaseModel
MaintenanceMode bool `json:"maintenance_mode" mfx:"default:false"`
MinAppVersion string `json:"min_app_version" mfx:"default:1.0.0"`
Banner string `json:"banner"`
}
server.MustRegister(
AppConfig{}, maniflex.ModelConfig{Singleton: true, TableName: "config"},
)
GET /config → 200 {"data":{"id":"singleton","maintenance_mode":false,"min_app_version":"1.0.0","banner":""}}
PATCH /config {"maintenance_mode": true} → 200 {"data":{"id":"singleton","maintenance_mode":true, ...}}
GET /config → 200 (reflects the update)
POST /config → 405
Because the row is auto-provisioned from column defaults, a singleton model may
not declare mfx:"required" fields — there would be no value to satisfy them on
first access. Such a model is rejected at registration. Give fields sensible
mfx:"default:…" values (or make them pointers) instead.
One row per tenant
The example above is one row for the whole application. The other common shape is one row per tenant — a storefront, a profile, a per-org settings record — which is near-universal in B2B SaaS.
You get it by scoping the model the same way you scope any other: register a
db.Tenancy or db.ForceFilter for it on the DB step. The singleton then
resolves and provisions the caller’s row rather than a global one.
type StoreSite struct {
maniflex.BaseModel
OwnerID string `json:"owner_id" db:"owner_id" mfx:"filterable,unique,default:"`
Banner string `json:"banner" mfx:"default:untitled"`
}
server.MustRegister(StoreSite{}, maniflex.ModelConfig{Singleton: true})
server.Pipeline.DB.Register(
db.Tenancy("owner_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.Claims["owner_id"].(string)
}),
maniflex.ForModel("StoreSite"),
)
GET /store_sites (as owner A) → 200 A's storefront, created on first access
GET /store_sites (as owner B) → 200 B's storefront — a different row
PATCH /store_sites (as owner A) → 200 updates A's, never B's
There is no separate SingletonScope setting, because a bare column name cannot
say where the value comes from — whether it is ctx.Auth.UserID, a tenant
claim, or something else. The scoping middleware already answers that, so the
singleton reads its scope from the request’s forced filters and there is exactly
one place to configure it.
The route shape is unchanged: still the bare table path, still no id, still no
POST/DELETE. Only which row it addresses changes.
A few consequences worth knowing:
- A scoped row keeps an ordinary generated primary key.
maniflex.SingletonIDnames the global row only; one fixed id could not name one row per scope. - Give the scope column a unique index (
mfx:"unique", above). Two concurrent first accesses would otherwise both provision a row; with the index, the loser collides and re-reads the winner’s. - A request with no scope gets the global row. If your resolver returns
nilfor an unauthenticated caller, that caller reads theSingletonIDrow. Scope-or-refuse is whatdb.Tenancydoes (it answers403when it cannot determine the tenant);db.ForceFilterapplies no filter instead. - The scope has to be one the framework can write, not merely read. It is
stamped onto the provisioned row, so it must be a plain equality — which is
what
db.Tenancyanddb.ForceFilterbuild. A scope that names no single value (aninfilter, or adb.ForceFilterViawhose value lives on another table) cannot provision a row and is refused rather than creating one its own author could not then read.
This replaces the previous workaround — Headless plus a hand-written action —
which cost 40–60 lines and, because actions skip the Validate
step, silently gave up every
mfx tag rule (required, enum, min/max, immutable) and the generated
OpenAPI schema along with them.
Upgrading: a singleton that already has a global row and then gains a scope does not migrate that row — it has no scope column value, so it matches nobody and each tenant is provisioned a fresh one. Backfill or drop it deliberately.
ModelConfig registration order
A ModelConfig is positioned immediately after the model it configures:
server.MustRegister(
User{},
Article{}, maniflex.ModelConfig{Versioned: true},
Comment{},
)
Here User and Comment use defaults; only Article is versioned.
Two argument shapes are registration errors, because there is no valid reading of either and both used to silently discard the config you wrote:
- A
ModelConfigat position 0 (no preceding model to attach to). - Two
ModelConfigs in a row (the second has no fresh model to bind to).
See Strict mode for the full set of configuration problems caught at startup.
Optional embeds
Beyond BaseModel, the framework provides embeds that add columns and switch on
behaviour when present:
| Embed | Adds | Effect |
|---|---|---|
maniflex.WithDeletedAt | deleted_at (nullable timestamp) | timestamp-based soft delete |
maniflex.WithIsDeleted | is_deleted (boolean) | flag-based soft delete |
Embedding one of these is equivalent to setting SoftDelete in ModelConfig.
The two approaches and their query semantics are covered in
Soft Delete.
type Article struct {
maniflex.BaseModel
maniflex.WithDeletedAt // DELETE marks deleted_at instead of removing the row
Title string `json:"title" mfx:"required"`
}
Registration order
Models must be registered before the database adapter is opened. The adapter is constructed from the registry — it reads the registered models to run migrations and resolve relations — so the registry must be complete first:
server.MustRegister(User{}, Article{}, Comment{}) // 1. populate the registry
db, err := sqlite.Open("./app.db", server.Registry()) // 2. build the adapter from it
server.SetDB(db) // 3. inject the adapter
Registering a model after SetDB has no effect on an already-open adapter.
Next
- Field Tags Reference — every
mfx:tag and its meaning. - Relations — foreign keys and slice fields.
- Soft Delete —
WithDeletedAt,WithIsDeleted, and query behaviour.
Field Tags Reference
A model’s behaviour is declared with three struct tags on each field. This page
documents all three, and every directive accepted by the mfx tag.
The three tags
| Tag | Controls | Default if omitted |
|---|---|---|
json | field name in request and response bodies | snake_case of the Go field name |
db | database column name | the resolved json name |
mfx | field behaviour — validation, querying, and more | no directives |
Title string `json:"title" db:"title" mfx:"required,filterable,sortable"`
The mfx tag holds a comma-separated list of directives. Whitespace around each
directive is trimmed. Directives are either flags (a bare word) or
key-value directives (key:value).
An unrecognised directive is a registration error, and the message names the directive it thinks you meant:
maniflex: model "User" field "Role" has unknown mfx option
"read_only" (did you mean "readonly"?) — an unrecognised option is
not applied, so a protective directive that is misspelt leaves the
field unprotected
Directives used to be matched exactly and anything else discarded in silence.
For a descriptive directive that was merely puzzling — a misspelt sortable just
meant no sorting — but for a protective one it was a hole: mfx:"read_only"
left Readonly false, so the field stayed writable by any client, with nothing
in the logs or the OpenAPI spec to show it. Directives are case-sensitive and
lowercase; mfx:"Readonly" is rejected rather than quietly accepted.
Excluding a field
A field is dropped from the model entirely — no column, not in any payload — if
any of its three tags is set to -:
Internal string `mfx:"-"` // excluded
Cache string `json:"-"` // excluded
Scratch string `db:"-"` // excluded
Validation directives
These constrain the values a field accepts on write.
| Directive | Effect |
|---|---|
required | the field must be present in a create request |
enum:a|b|c | the value must be one of the pipe-separated options |
min:N | numeric minimum (N is a number) |
max:N | numeric maximum |
default:V | value applied when the field is absent; cast to the field’s type |
Status string `json:"status" mfx:"required,enum:draft|published|archived"`
Priority int `json:"priority" mfx:"min:1,max:5,default:3"`
Malformed values are refused at registration
A value these directives cannot use is a startup error, not a silent no-op:
| Written | Result |
|---|---|
mfx:"min:abc", mfx:"max:" | registration error — not a number |
mfx:"enum:", mfx:"enum:a||b" | registration error — empty option |
mfx:"readonly,required" | registration error — unsatisfiable |
The first two used to be dropped on the floor, so a tag that reads as a
constraint enforced nothing. readonly,required is the same unsatisfiable shape
as hidden,required: readonly strips the field from the request before the
required check runs, so the check never sees it and the field was quietly
optional. (hidden implies readonly, so that combination reports its own more
specific message.)
Separately, a value that a min/max bound cannot be measured against — a
string sent to a numeric field — now fails validation with “must be a number”
rather than skipping the check.
How default: is applied
default:V becomes a SQL DEFAULT clause on the column. Nothing applies it in
Go — it fires because the INSERT omits the column, so it takes effect whenever
a write does not name the field. That holds for both doors into a model:
| Create path | Column omitted when |
|---|---|
POST /:model | the request body has no such key |
maniflex.Create[T] | the struct field is at its Go zero value |
The Go zero is the only signal a struct can give, so a defaulted column cannot
be given an explicit zero through Create[T] — Priority: 0 against
default:3 stores 3. Make the field a pointer when that distinction matters:
Priority *int `json:"priority" mfx:"default:3"` // nil → 3, new(int) → 0
Only defaulted columns behave this way. A zero on a field with no default:
tag is written as that zero, on both paths.
Before v0.2.5,
Create[T]wrote every column at its Go value and adefault:never fired — the same model created over HTTP and in Go disagreed. In the same release,default:on a pointer field started reaching the schema at all; it was previously emitted only forNOT NULLcolumns and silently dropped.
Write-access directives
These govern whether a field can be set by a client, and when.
| Directive | Effect |
|---|---|
readonly | stripped from all write operations; values sent by a client are ignored |
immutable | accepted on create, rejected on update |
BaseModel’s created_at and updated_at are readonly. Use immutable for
values that are set once and must not change afterwards, such as an owner ID.
Both mean “not from a client”. A value the server stamps via ctx.SetField —
db.Tenancy writing a tenant column, an auth middleware writing an owner — is
kept, even on a readonly or immutable field. Only values parsed from the
request body are stripped.
Email string `json:"email" mfx:"required,immutable"`
ApiKey string `json:"api_key" mfx:"readonly"`
Response-visibility directives
Both directives drop the field from API responses. They differ in whether the client may write the field.
| Directive | Read in responses | Write on create / update |
|---|---|---|
writeonly | no | yes |
hidden | no | no |
writeonlyis for values the client must supply but should never see again — typically passwords. The field is included in the create and update request schemas; only the response is scrubbed.hiddenis for values clients have no business touching at all — server-managed internals, audit fields, derived data. The field is dropped from create and update schemas as well, so it cannot be set from the API. It is still stored, and code running inside the pipeline (a middleware on the Service step, for example) can populate it.
// Client sets it on create, never sees it back.
Password string `json:"password" mfx:"required,writeonly"`
// Server-managed; client can neither read nor write it.
InternalScore float64 `json:"internal_score" mfx:"hidden"`
hidden therefore implies readonly — writing it out as mfx:"hidden,readonly"
is allowed but redundant. If you want a field the client writes but never reads,
that is writeonly, and spelling both (mfx:"hidden,writeonly") leaves it
writable: the explicit directive wins over the implication.
mfx:"hidden,required" is rejected at registration. The two contradict —
hidden stops the client sending the field, required insists it does — so no
request could satisfy it. A value the client must supply but never reads back is
mfx:"writeonly,required".
A field that is both readonly and not hidden is the opposite case: visible
in responses, never accepted from the client.
json:"-" is treated as hidden and read-only: the field stays a real,
persisted column that the server owns, but it never appears in API responses and
is never accepted from the client. This matches the Go convention that json:"-"
affects serialization, not schema. To exclude a field from the database entirely
(no column), use db:"-" or mfx:"-" instead.
Record-locking directives
These freeze a whole record — not just a field — once its state matches a condition. Useful for terminal states in business workflows (posted invoices, closed pay periods, confirmed POs).
| Directive | Effect |
|---|---|
lock_when:field=value | when the existing record’s field equals value, updates and deletes return 422 RECORD_LOCKED |
Multiple lock_when directives accumulate; any matching condition locks
the record. The directive can be written on any field — the referenced
field is what matters. A typo in the referenced JSON name is caught at
registration so you never ship a rule that silently never matches.
type Invoice struct {
maniflex.BaseModel
Number string
Status string `mfx:"enum:draft|posted|void,lock_when:status=posted,lock_when:status=void"`
Amount int
}
The transition into a locked state is itself allowed — when the request arrives, the loaded record is still in its previous state. After that update commits, the record becomes frozen.
lock_when is checked before the default Validate step’s other rules on
update, and before the adapter’s Delete call on delete. Creates are
exempt: there is no prior state to check.
The guard fails closed. It reads the record through the request’s transaction
when one is active — so it sees state the same request has written but not yet
committed — and if that read fails for any reason other than “no such record”,
the request is rejected (500 DB_ERROR) rather than allowed through unchecked.
Pessimistic lock directive
| Directive | Effect |
|---|---|
lock_scope:ModelName | before a create, acquire a SELECT … FOR UPDATE lock on the row referenced by this field’s value |
Eliminates manual ctx.LockForUpdate calls in the most common case: a
create that must read-then-write a shared resource without a concurrent
write sneaking in between.
type Dispense struct {
maniflex.BaseModel
StockID string `json:"stock_id" db:"stock_id" mfx:"required,lock_scope:StockBalance"`
Quantity int `json:"quantity" db:"quantity" mfx:"required,min:1"`
}
Requirements:
- The model must run inside a transaction. Register
maniflex.WithTransaction(nil)on the Service step; otherwise the DB step aborts with500 LOCK_SCOPE_NO_TX. - The referenced model name must be registered. A typo is caught at startup
(in
Handler()), so it never reaches production silently. - If the referenced row does not exist, the create returns
404 NOT_FOUND.
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForModel("Dispense"),
maniflex.ForOperation(maniflex.OpCreate),
)
Comparison with ctx.LockForUpdate:
lock_scope tag | ctx.LockForUpdate | |
|---|---|---|
| Declaration | struct tag | custom Service middleware |
| Fields locked | one per tag directive | any ID at runtime |
| Requires transaction | yes (enforced at runtime) | yes (enforced at call time) |
| Use when | one fixed FK to lock | dynamic or multiple targets |
See Transactions for the underlying ctx.LockForUpdate
and BeginTx APIs.
Query directives
These opt a field into the query string. A field is not filterable or sortable unless explicitly tagged.
| Directive | Effect |
|---|---|
filterable | the field may be used in ?filter= |
sortable | the field may be used in ?sort= |
searchable | the field is indexed for native full-text search (?q=); text columns only |
cursor_field:<name> | opt the model into keyset (cursor) pagination; <name> is the column to walk by — it must be sortable and non-nullable (a pointer field is rejected at registration) |
See Querying for the filter, sort, and cursor-pagination grammar.
Schema directives
| Directive | Effect |
|---|---|
unique | a hint to the adapter to add a UNIQUE constraint on the column |
index | create a (non-unique) index on the column during AutoMigrate |
Slug string `json:"slug" mfx:"required,unique"`
Email string `json:"email" mfx:"index"`
index creates an index named idx_<table>_<column>. It is skipped when the
column is already covered by another index — a unique constraint on the same
field (databases index unique columns implicitly), a ModelConfig.Indices entry,
or a scheduled-column auto-index — so adding it is always safe. Indexing a
foreign-key column (e.g. mfx:"index" on UserID) is a common, valid use.
unique is enforced on both the create-table and the add-column (ALTER TABLE)
paths — AutoMigrate creates a UNIQUE INDEX named uidx_<table>_<column> in both
cases. Adding a unique column to a table that already holds rows with duplicate
values in that column fails migration (the index build errors, naming the
table and column) rather than silently dropping the constraint; resolve the
duplicates before deploying.
JSON and other custom columns
A bare map[string]any, map[string]string, or []string field has no SQL
column mapping and fails AutoMigrate with a clear error. Wrap it in a named
type that implements maniflex.SQLTyper (plus driver.Valuer + sql.Scanner)
so it controls its own column type — e.g. a JSONMap that maps to JSONB on
Postgres and TEXT on SQLite. maniflex.LocaleString is a built-in example. To
keep such a field out of the database entirely, tag it mfx:"-".
Relation directives
A field may declare a relationship to another model. Relations are opt-in —
an <Name>ID field is a plain column unless you tag it.
| Directive | Effect |
|---|---|
relation | marks an FK field as a BelongsTo; the target is inferred from the field name (AuthorID → Author) |
relation:Name | explicit relation; Name is the companion struct field carrying the target type |
relation:Name;onDelete:action | sets the referential action — cascade, setNull, or restrict |
through:Model | on a slice field, declares a many-to-many relation through the named junction model |
norelation | deprecated no-op — relations are no longer inferred from the ID suffix, so nothing to opt out of |
onDelete sub-options are joined to the relation: directive with a semicolon,
not a comma. Relationships are covered in full in Relations.
A field whose name ends in ID (e.g. UserID) is a plain scalar column unless
tagged mfx:"relation". So a value column that merely ends in ID — an external
reference, an opaque token — needs no special tag:
ExternalID string `json:"external_id"` // just a string, not a relation
UserID string `json:"user_id" mfx:"relation"` // → User (opt in)
File upload directives
file marks a field as a file-upload field. The column stores the storage key;
multipart form-data is then accepted for create and update on the model.
The field’s Go type must be string (one key) or maniflex.FileKeys (many).
Any other type is a registration error — every rule below is keyed on the column
being a storage key, so on another type they would all be silently skipped.
| Directive | Effect |
|---|---|
file | mark the field as a file upload |
max_size:N | maximum file size; accepts KB, MB, GB suffixes, or plain bytes. On FileKeys, per file |
max_count:N | FileKeys only — maximum number of keys (default 100) |
accept:p1|p2 | allowed MIME-type patterns, e.g. image/*|application/pdf |
auto_delete:false | keep the stored file when the record is hard-deleted or the field is replaced (default: delete it) |
upload:presigned | mount POST /{model}/{field}/upload-url so the client uploads straight to storage |
file_acl:private | (default) response carries the raw storage key |
file_acl:signed | response replaces the key with a pre-signed URL (TTL: Config.FilesConfig.SignedURLTTL, default 1h) |
file_acl:public | response replaces the key with a permanent / long-lived URL |
Avatar string `json:"avatar" mfx:"file,max_size:2MB,accept:image/*"`
Logo string `json:"logo" mfx:"file,file_acl:public,accept:image/*"`
Images maniflex.FileKeys `json:"images" mfx:"file,accept:image/*,max_count:10"`
See File Fields & Uploads for the upload workflow, and
Many files per field for FileKeys.
Encryption directives
| Directive | Effect |
|---|---|
encrypted | the field is encrypted at rest (AES-256-GCM) and decrypted on read |
key:name | the key name passed to the key provider; defaults to default |
Encrypted fields cannot be filtered or sorted, because the stored value is
ciphertext. If unique is also set, a companion {field}_hmac column enforces
uniqueness without exposing the plaintext.
SSN string `json:"ssn" mfx:"encrypted,key:patient-pii"`
Scheduled directives
The scheduled directive declares a time-driven transition on a timestamp
field — for example, soft-deleting a row once a timestamp passes. The directive
only marks the field; the transitions are applied by a background runner
documented in Events & Background Jobs.
It is an advanced feature with several sub-options joined by semicolons:
ExpiresAt time.Time `json:"expires_at" mfx:"scheduled;soft-delete"`
PublishAt time.Time `json:"publish_at" mfx:"scheduled;field=status;from=draft;to=published"`
| Sub-option | Effect |
|---|---|
soft-delete | soft-delete the row when the timestamp is reached |
hard-delete | permanently delete the row when the timestamp is reached |
field=F | the field to change |
from=V | apply only when field currently equals V |
to=V | the value to set field to |
Locale directives
These apply to fields declared as maniflex.LocaleString — a multilingual
string stored as a JSON object keyed by locale code (e.g. {"en":"Finance","ar":"مالية"}).
Mark a field as locale-aware with mfx:"locale". All other locale directives
require locale to be present.
| Directive | Effect |
|---|---|
locale | marks the field as a LocaleString; enables locale-aware response serialisation |
split | (default) response emits "name" = resolved string and "name_i18n" = full map |
resolve | response always emits "name" as a plain string; no companion field |
dynamic | response emits a string when ?locale= is set, the full map otherwise |
default_locale:code | field-level fallback locale (e.g. default_locale:ar) when the client did not request a specific locale |
type Department struct {
maniflex.BaseModel
Name maniflex.LocaleString `json:"name" mfx:"locale,filterable,sortable"`
Bio maniflex.LocaleString `json:"bio" mfx:"locale,resolve,default_locale:ar"`
}
The resolved locale for a request follows a precedence chain:
?locale= param → Accept-Language header → default_locale tag → model
DefaultLocale → app LocaleOptions.Default → "en".
See Localization for the full LocaleResolver setup and filtering/sorting behaviour.
Quick reference
| Directive | Category |
|---|---|
required | validation |
enum:… min: max: default: | validation |
readonly immutable | write access |
hidden writeonly | response visibility |
filterable sortable searchable cursor_field:… | querying |
unique index | schema |
relation relation:… through:… | relations |
file max_size: max_count: accept: auto_delete:false file_acl: upload:presigned | file upload |
encrypted key:… | encryption |
scheduled;… | scheduled transitions |
locale split resolve dynamic default_locale:… | localization |
- | exclude the field |
Relations
A relation connects two models through a foreign-key column or a junction
table. Relations are declared on the struct — by a field name, a tag, or a
slice — and are populated on demand via the ?include= query parameter.
maniflex recognises three kinds:
| Kind | Direction | Declared by |
|---|---|---|
| BelongsTo | this row holds the FK | an FK field tagged mfx:"relation" (or mfx:"relation:Name") |
| HasMany | the other table holds the FK | a slice field of the related type |
| ManyToMany | a junction table connects both sides | a slice field with mfx:"through:Junction" |
BelongsTo
Relations are opt-in: tag the foreign-key field mfx:"relation". The target
model is inferred from the field name with the trailing ID stripped
(UserID → User).
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required"`
UserID string `json:"user_id" mfx:"required,filterable,relation"` // → User
}
UserID is a foreign key to User, keyed user (snake-case of the trimmed
field name) — the value used in ?include=, in nested filters
(?filter=user.role:eq:admin), and in nested sorts (?sort=user.name:asc).
The FK field is also a regular column. Tag it filterable if clients need to
query by it, exactly like any scalar.
Note: an
<Name>IDfield is a plain scalar column unless you tag itmfx:"relation". A value column that merely ends inID—ExternalID,CloudEventID, a third-party token — stays a plain column with no relation and no?include=key. (The legacymfx:"norelation"opt-out is now a deprecated no-op, kept only so existing models compile.)
When the field name doesn’t match the target model, name it explicitly with
mfx:"relation:Target". A bare mfx:"relation" on a field that doesn’t end in
ID fails at startup: there’s no suffix to strip, so the target would be
inferred from the whole field name — a guess that is almost never a real model.
A relation whose target model is simply never registered is allowed by default (it may be a plain foreign id that wants no relation tag) and rejected under strict mode.
Including the related row
curl 'localhost:8080/api/posts?include=user'
Each post in the response gains a user object populated from the related
table. Multiple includes are comma-separated:
curl 'localhost:8080/api/posts/<id>?include=user,comments'
BelongsTo (explicit)
When the FK field name does not match the target model — for example, a
ManagerID pointing to User — declare the relation with mfx:"relation:Name"
and add a companion field of the target type:
type Team struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required"`
ManagerID string `json:"manager_id" mfx:"required,filterable,relation:Manager"`
Manager User `json:"manager,omitempty"`
}
ManagerIDis the column that stores the FK.Manageris the companion field: it carries the target type (User) so the framework can resolve the relation. It is not a column itself — its only role is type information for relation scanning.
The relation key here is manager (snake-case of Manager), so the include
becomes ?include=manager.
A relation:Name directive without a matching companion field fails
registration.
HasMany
The inverse side: a slice of the related struct, declared on the model that does not hold the FK.
type User struct {
maniflex.BaseModel
Name string `json:"name"`
Posts []Post `json:"posts,omitempty"` // populated when ?include=posts
}
There is no column on users for this — Posts is purely a relation
declaration. The FK is expected on the related table, named after the parent
model: Post is expected to carry user_id. (That column comes from UserID
on Post, as in the BelongsTo example above.)
The relation key for a slice is the snake-case of the field’s JSON name, here
posts.
curl 'localhost:8080/api/users/<id>?include=posts'
ManyToMany
A many-to-many relation uses a third junction model that carries the two FKs.
Both sides declare a slice with mfx:"through:JunctionModel":
type Product struct {
maniflex.BaseModel
Name string `json:"name"`
Tags []Tag `json:"tags,omitempty" mfx:"through:ProductTag"`
}
type Tag struct {
maniflex.BaseModel
Label string `json:"label"`
Products []Product `json:"products,omitempty" mfx:"through:ProductTag"`
}
// The junction model — register it just like any other.
type ProductTag struct {
maniflex.BaseModel
ProductID string `json:"product_id" mfx:"required,filterable,relation"`
TagID string `json:"tag_id" mfx:"required,filterable,relation"`
}
All three models must be registered. The junction’s FK columns are declared as
BelongsTo relations (mfx:"relation" on ProductID and TagID), so the
framework can resolve which side of the join goes where.
?include=tags on a product follows the junction and returns the related tags
directly.
Junction payload (_through)
A junction often carries data of its own — a role, a position, a date the link
was made. When it does, each included row gains a _through object holding
those columns:
{
"id": "tag-1",
"label": "blue",
"_through": { "position": 2, "linked_at": "2026-07-19T09:00:00Z" }
}
The two foreign keys and the junction’s id are excluded — they say nothing the
response does not already carry.
The junction is a model like any other, and _through honours its tags. A
column marked mfx:"hidden" or mfx:"writeonly" is absent, exactly as it would
be in a response from the junction’s own endpoint:
type ProductTag struct {
maniflex.BaseModel
ProductID string `json:"product_id" mfx:"required,filterable,relation"`
TagID string `json:"tag_id" mfx:"required,filterable,relation"`
Position int `json:"position"` // in _through
InvitedBy string `json:"invited_by" mfx:"hidden"` // not in _through
}
Two further exclusions are worth knowing:
mfx:"encrypted"columns are omitted, along with their_hmaccompanions. The junction payload has no decryption pass, so the alternative would be emitting ciphertext — no use to a client, and it discloses the encryption envelope. Put a value you need to read on the related model, not on the junction.- Columns the junction model does not declare are omitted. A column present in the table but absent from the model is schema drift, and dumping it is what used to make this a leak.
Before v0.2.5
_throughwas copied verbatim from the junction row, so hidden and write-only columns surfaced on every include.
Which models are join tables
A junction can be declared three ways. In order of precedence:
| How | When to use | |
|---|---|---|
| Explicit relation | mfx:"through:Junction" on a slice field | you want the relation named on the model |
| Explicit marker | embed maniflex.JunctionModel | the junction carries columns of its own |
| Auto-detected | two BelongsTo relations, no other columns | a plain link table |
Auto-detection accepts only the unambiguous shape — two foreign keys, an id,
timestamps, and nothing else:
type ProductTag struct { // auto-detected
maniflex.BaseModel
ProductID string `json:"product_id" mfx:"relation"`
TagID string `json:"tag_id" mfx:"relation"`
}
Add a column of its own and detection stops, because that shape is equally what an ordinary entity looks like:
type Order struct { // NOT a junction — nothing is inferred
maniflex.BaseModel
CustomerID string `json:"customer_id" mfx:"relation"`
AddressID string `json:"address_id" mfx:"relation"`
Total int `json:"total"`
}
Before v0.2.6
Orderwas treated as a join table, so Customer and Address silently gained a many-to-many through it. If you relied on detection for a junction that carries payload, embedJunctionModel— see below.
A junction that carries payload says so:
type Enrollment struct {
maniflex.BaseModel
maniflex.JunctionModel
StudentID string `json:"student_id" mfx:"relation"`
CourseID string `json:"course_id" mfx:"relation"`
Term string `json:"term"` // in _through
}
JunctionModel is embedded alongside BaseModel, not instead of it — a
junction is an ordinary model with an id. The model must have exactly two
BelongsTo relations to distinct models; anything else is a registration error.
ModelConfig.DisableAutoJunction opts a model out of detection entirely, for a
link-shaped model that should not gain the relation.
Unique links
Junction pairs are not unique by default. Declare it when they should be:
type ProductTag struct {
maniflex.BaseModel
maniflex.JunctionModel `mfx:"unique"`
ProductID string `json:"product_id" mfx:"relation"`
TagID string `json:"tag_id" mfx:"relation"`
}
This emits a UNIQUE index over the two key columns, so a repeated link is
refused with 409. It also lets an include collapse duplicate pairs: with the
declaration a repeat is corruption, without it a repeat is data.
Off by default because a junction carrying its own attributes may legitimately
repeat a pair — Enrollment{student_id, course_id, term} holds one row per
term, and each carries its own _through payload. A pure link table almost
always wants the tag.
Adding it to a table that already holds duplicates fails the migration until
they are cleaned up. That is why it is separate from the marker: embedding
JunctionModel changes nothing about the schema, so declaring what a model is
never risks a migration.
Junction deletes
A junction’s foreign keys default to ON DELETE CASCADE, so deleting either
endpoint takes its link rows with it — a link to a row that no longer exists
says nothing. An explicit mfx:"on_delete:..." on the column wins, and edges
where either side soft-deletes are handled in maniflex’s own delete path rather
than by a database constraint (see below).
Cascading deletes
A BelongsTo relation may declare what happens to this row when the parent it
points at is deleted, using the onDelete sub-option:
AuthorID string `json:"author_id" mfx:"relation:Author;onDelete:cascade"`
| Action | Effect on this row when the referenced row is deleted |
|---|---|
cascade | this row is deleted too |
setNull | the FK column is set to NULL (the field must be a pointer, so it is nullable — otherwise a registration error) |
restrict | the parent delete is refused with 409 DELETE_RESTRICTED while this row exists |
| omitted | nothing — the delete leaves this row untouched (its FK dangles) |
The action is validated at startup: it must target a registered model, and
setNull needs a nullable FK. cascade recurses — deleting a row deletes its
children, then their children — and reference cycles are handled.
How it is enforced (and soft delete)
The action is enforced one of two ways, chosen automatically per relation:
- Neither side soft-deletes → a real database
FOREIGN KEY … ON DELETE …constraint carries it, and the database enforces it natively (and enforces referential integrity on insert: a row naming a non-existent parent is refused with409). - Either side soft-deletes → the database cannot help (a soft delete is an
UPDATE, soON DELETEnever fires, and a DB cascade can only hard-delete), so maniflex enforces it in the delete request’s own transaction instead. A soft-delete child of a cascaded parent is soft-deleted identically — itsdeleted_atis set, the row is not removed.
Either way the whole deletion is atomic: a restrict that fires rolls back any
cascade that ran alongside it.
Existing SQLite tables. New FK constraints are declared when a table is created. SQLite cannot add a foreign key to a table that already exists, so adding
onDeleteto a model with a pre-existing SQLite table does not retroactively add the constraint — recreate the table, or rely on the soft-delete path, which is enforced in the application layer regardless. Postgres adds the constraint on the next migration.
Soft delete and relations
When the related model uses soft delete, rows whose deleted marker is set are
omitted from ?include= results — the same filter the framework applies to
list endpoints. See Soft Delete.
Scoping and ?include=
A request’s forced filters — the scope imposed by db.Tenancy or
db.ForceFilter — are applied to included rows as well as to the primary read,
for every relation kind, wherever the related model carries the filtered column.
This matters because the foreign key is the client’s to set. Without it, a
caller who can write an FK (or a many-to-many junction row) puts their own row
inside another tenant’s ?include=, and an attach pulls another tenant’s record
into their own response. The framework does not validate junction writes, so the
include is where the scope has to hold.
// Tenancy on both models; the include is scoped by the same filter.
server.Pipeline.DB.Register(
db.Tenancy("org_id", orgOf),
maniflex.ForModel("Order", "OrderLine"),
)
What is not covered. Be precise about this, because the gap is narrow but real:
- A related model with no such column is not scoped. That is the right answer for the case that produces it — a shared lookup table (currencies, categories, statuses) is not tenant-partitioned and has nothing to scope by — but if a model is partitioned by something the filter cannot name, its includes are unscoped and the FK write is yours to validate.
- Relation-path filters (
db.ForceFilterVia) are skipped: they are written against the primary model’s relations and mean nothing on the related table. - The scope applies to the rows returned, not to the junction. A row that should never have been attached is now invisible rather than absent; clean it up at the write.
Quick reference
| Goal | Declaration |
|---|---|
Belongs to User (FK column matches) | UserID string |
Belongs to User under a different name | ManagerID string with mfx:"relation:Manager" + Manager User companion |
Has many Post | Posts []Post (other side carries UserID) |
Many-to-many via ProductTag | Tags []Tag with mfx:"through:ProductTag", on both sides |
| Cascade on parent delete | mfx:"...,onDelete:cascade" |
| Populate in a response | ?include=user,comments |
Soft Delete
A soft-deleted row is left in the database but marked as deleted. DELETE
requests flip the marker; list, read, and include queries hide rows whose
marker is set. This page covers how a model opts in, the two storage styles,
and the query semantics that follow.
Opting in
There are two ways to enable soft delete on a model. Both produce the same behaviour; pick whichever fits the declaration style of the rest of the model.
By embed
Embed one of the framework’s marker types:
type Article struct {
maniflex.BaseModel
maniflex.WithDeletedAt // timestamp-based soft delete
Title string `json:"title"`
}
| Embed | Column added | Storage style |
|---|---|---|
maniflex.WithDeletedAt | deleted_at — nullable timestamp; NULL means not deleted | timestamp |
maniflex.WithIsDeleted | is_deleted — boolean; false means not deleted | flag |
Both columns are tagged readonly and filterable. They are not part of any
write request — the framework manages them.
By configuration
The same setup, expressed at registration:
server.MustRegister(
Article{}, maniflex.ModelConfig{
SoftDelete: maniflex.SoftDeleteConfig{
Enabled: true,
Field: "deleted_at",
FieldType: maniflex.SoftDeleteTimestamp, // or maniflex.SoftDeleteBool
},
},
)
If both an embed and a ModelConfig.SoftDelete are present, the explicit
config wins.
Choosing between timestamp and boolean
Both styles work; they differ in what you can tell from the column afterwards.
WithDeletedAtrecords when the row was deleted, which makes audit trails, “deleted in the last 30 days” queries, and undelete-with-context possible. It is the default choice.WithIsDeletedstores only the fact of deletion. Use it when the surrounding system already records deletion timestamps elsewhere, or when a boolean fits an existing schema better.
Delete semantics
For a soft-deletable model, DELETE /api/<table>/{id} updates the marker
instead of removing the row:
| Style | What DELETE does |
|---|---|
| Timestamp | sets deleted_at to the current UTC time |
| Boolean | sets is_deleted to true |
The endpoint, the response, and the status code are the same as for a hard-delete model; only the underlying SQL differs.
Query semantics
Once enabled, soft-deleted rows are filtered out everywhere the framework reads the table:
- List (
GET /<table>) — only un-deleted rows are returned. - Read (
GET /<table>/{id}) — a soft-deleted row returns404. - Includes — relations populated via
?include=skip soft-deleted children. - Update —
PATCHon a soft-deleted row returns404; the row is treated as absent. - Delete — a second
DELETEon the same row returns404and leaves the marker as it was, so the original deletion time survives. This holds inside a transaction too.
To surface the marker for clients that need it (e.g. an admin tool), filter on it explicitly:
# Only soft-deleted rows
curl 'localhost:8080/api/articles?filter=deleted_at:ne:null'
deleted_at and is_deleted are filterable, so the standard filter grammar
applies — see Querying.
Restoring a row
Opt a model in with ModelConfig.RestoreEnabled to mount a restore endpoint:
server.Register(Article{}, maniflex.ModelConfig{RestoreEnabled: true})
curl -X POST 'localhost:8080/api/articles/<id>/restore'
The request carries no body — it names the row in the URL and says only
“undo the delete”. A successful restore returns 200 with the restored record,
so the client need not re-read it.
| Status | When |
|---|---|
200 | restored; the record is returned |
404 | no such row, or the row exists and is not deleted (mirroring the re-delete guard) |
501 RESTORE_UNSUPPORTED | the configured adapter does not implement Restorer |
It is off by default, and deliberately so: un-deleting is a privileged operation, and an endpoint that appeared merely because a version was upgraded would not be covered by the authorisation an app had already written. The route is only mounted for models that actually soft-delete — on a hard-delete model there is nothing to restore.
It dispatches as an update
A restore runs as OpUpdate, not as an operation of its own. That is the point:
every middleware an app already registered for “who may modify this row” — auth,
tenancy, force filters, audit — governs un-deleting it, with nothing to rewrite
and no new operation constant to discover.
// This already covers restore. Nothing further to do.
server.Pipeline.Auth.Register(auth.RequireRole("editor"),
maniflex.ForOperation(maniflex.OpUpdate))
Where the two must be told apart — an audit sink recording “restored” rather
than “updated”, or a validation rule that only makes sense against a body — read
ctx.IsRestore(). Note a restore’s ctx.ParsedBody is empty, so body-driven
validation middleware sees nothing to check.
Scoping is enforced too. Because a soft-deleted row is invisible to every read path, the restore cannot read its target back to check scope the way an update does; instead the request’s forced filters are applied to the restore statement itself. A caller cannot un-delete a row outside their tenancy by knowing its id.
What it writes
Only the delete marker is cleared. updated_at is left untouched, so a restore
does not read as an edit to caches, sync clients, or anything else watching that
column — the audit or event record is where a restore is recorded.
Cascade is not undone. A restore brings back the row it names and nothing
else: nothing records which children an onDelete:cascade removed, so restore
each explicitly, or model the relationship so the children survive the parent’s
deletion. See Relations.
Custom adapters
The endpoint needs a database adapter implementing maniflex.Restorer. The
bundled SQLite and Postgres adapters do; a third-party adapter that does not is
unaffected and answers 501. It is a separate interface, not a DBAdapter
method, so adding it broke nothing:
type Restorer interface {
Restore(ctx context.Context, model *ModelMeta, id string, q *QueryParams) (any, error)
}
Apply q’s filters (the request’s forced scope, possibly nil) to the statement,
and return ErrNotFound when no row matches — including a row that exists but is
not deleted.
Interaction with hard delete
A model is either soft- or hard-delete; the choice is a property of the model, not the request. If you need a true hard delete on a soft-deletable model — for example, to honour an erasure request — perform it through a raw query or a custom action that bypasses the standard handler.
Quick reference
| Goal | Declaration |
|---|---|
| Timestamp soft delete | embed maniflex.WithDeletedAt |
| Boolean soft delete | embed maniflex.WithIsDeleted |
| Soft delete with a custom column | ModelConfig.SoftDelete |
| List only deleted rows | ?filter=deleted_at:ne:null (timestamp) or ?filter=is_deleted:eq:true (boolean) |
| Restore a deleted row | ModelConfig.RestoreEnabled → POST /:model/{id}/restore |
File Fields & Uploads
A field tagged mfx:"file" accepts an uploaded file alongside the model’s
JSON. The column stores an opaque storage key; the bytes live in a
configured FileStorage backend. Standalone upload, download, and delete
endpoints can also be mounted (see Standalone file endpoints).
Config changed. File settings now live under a single
Config.FilesConfigstruct (maniflex.FilesConfig). The old flatConfig.FileStorage,Config.FileSignedURLTTL, andConfig.FileMiddlewarefields have been removed. The mapping is:
Old ( Config.…)New ( Config.FilesConfig.…)FileStorageStorageFileSignedURLTTLSignedURLTTLFileMiddlewareBeforeMiddlewares(implied by FileStorage != nil)MountEndpoints— now explicit, see the footgunNew:
KeyGen(custom storage-key layout) andAfterMiddlewares(post-handler observation).
Declaring a file field
Add the file directive to a string field:
type Article struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required"`
Cover string `json:"cover" mfx:"file,max_size:2MB,accept:image/*"`
}
The column’s Go and DB types are string — what is stored is the storage key
returned after the upload. The on-disk bytes are managed by the storage
backend; the database row holds only the reference.
A file field’s Go type must be string (one key) or
maniflex.FileKeys (many). Anything else is a
registration error: every file rule is keyed on the column being a storage key,
so on another type they would all be skipped.
Tag sub-options:
| Sub-option | Effect |
|---|---|
file | mark the field as a file upload |
max_size:N | per-field size limit; suffixes KB, MB, GB or plain bytes. On a FileKeys field it bounds each file, not their total |
max_count:N | FileKeys only — maximum number of keys (default 100). See Many files per field |
accept:p1|p2 | allowed MIME-type patterns, e.g. image/*|application/pdf |
auto_delete:false | keep the stored file when the row is hard-deleted or the field is replaced (default: delete) |
upload:presigned | mount POST /{model}/{field}/upload-url so the client uploads straight to storage instead of through the app — see Direct-to-storage uploads. Requires a backend that can presign (storage/s3; not LocalStorage) |
upload:stream | multipart upload, but piped straight to storage as it arrives instead of buffered to the app server’s disk first — see Streaming uploads. Mutually exclusive with upload:presigned |
file_acl:private | (default) response carries the raw storage key; downloads go via /files/<key> or the per-model attachment route |
file_acl:signed | response replaces the key with a pre-signed URL valid for Config.FilesConfig.SignedURLTTL (default 1h). Requires FileStorage.URL() |
file_acl:public | response replaces the key with a permanent / long-lived URL (e.g. S3 7-day max). Pair with public-read ACL on the bucket for true permanence |
Those three are the whole set — anything else is a registration error naming the
field. A typo used to be read as private, which looks like the safe direction
and is not the safe outcome: the field was asked to serve signed or public URLs
and served raw storage keys instead, with nothing to say why.
accept matches the content type the client declared on the multipart part; when
the part declares nothing (or the generic application/octet-stream), the type is
sniffed from the first 512 bytes. A declared type is a client-supplied claim, so
treat accept as an input filter, not a security boundary — downloads are what
enforce safety, via X-Content-Type-Options: nosniff and forced attachment for
anything outside the inline allowlist (see
Standalone file endpoints).
Tag detail is in Field Tags Reference.
Many files per field (FileKeys)
A gallery, or any attachment set, is a maniflex.FileKeys column:
type Post struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required"`
Images maniflex.FileKeys `json:"images" mfx:"file,accept:image/*,max_size:5MB,max_count:10,file_acl:signed"`
}
It stores as a JSON array (JSONB on Postgres, TEXT on SQLite), so ordering is
preserved exactly as written — a gallery keeps its sequence. Every rule a
single-key field enforces applies per key: existence, max_size (per file,
not per array), accept, file_acl signing on read, auto_delete, and cleanup
on hard delete.
Write by key reference, not multipart. Upload each file first — via
POST /files or a
presigned upload — then send the
keys:
PATCH /posts/1
{"images": ["uploads/a1/one.jpg", "uploads/b2/two.jpg"]}
A multipart upload to a FileKeys field is refused (422). Multipart carries
one file per field, so it could only ever store one key and would silently drop
the rest; and routing many large files through the app process is what presigned
uploads exist to avoid.
A PATCH replaces the whole array, as it does any column. With auto_delete
(the default), keys present before the write and absent after it are deleted from
storage once the write commits. The diff is a set difference, so reordering
deletes nothing, and a key you keep is never touched:
PATCH /posts/1 {"images": ["uploads/a1/one.jpg", "uploads/c3/three.jpg"]}
// one.jpg kept — still referenced
// two.jpg deleted — dropped by this write
// three.jpg kept — newly referenced
max_count bounds the array (default DefaultMaxFileCount, 100). Every key
is Stat’d against storage to enforce the field’s rules, so an uncapped array
would be one request buying N storage round-trips. Over the cap is 422 TOO_MANY_FILES.
No attachment route is mounted for a FileKeys
field — that route streams one object’s bytes and a list names no single one.
Use file_acl:signed to have each key rewritten to a URL in the response.
file_acl modes
type Attachment struct {
maniflex.BaseModel
Logo string `mfx:"file,file_acl:public,max_size:1MB,accept:image/*"`
Resume string `mfx:"file,file_acl:signed,max_size:5MB,accept:application/pdf"`
Notes string `mfx:"file"` // implicit private — raw key in the response
}
The rewrite happens in the Response step on create, read, list, and update. A
null/empty value passes through unchanged — no fabricated URLs to nothing.
Configure the signed-URL lifetime once on Config:
maniflex.New(maniflex.Config{
FilesConfig: maniflex.FilesConfig{
SignedURLTTL: 15 * time.Minute, // default: 1h
},
})
LocalStorage.URL returns the server-relative /files/<key> for both signed
and public modes (no real signing — bring an HMAC layer if you need it).
S3Storage.URL uses awss3.NewPresignClient; ttl=0 (public mode) maps to
the AWS 7-day maximum.
Configuring storage
Uploads require a FileStorage implementation, set on
Config.FilesConfig.Storage. The framework ships one backend; bring your own
for cloud storage.
import "github.com/xaleel/maniflex/storage"
fs, err := storage.NewLocalStorage("./uploads")
if err != nil {
log.Fatal(err)
}
server := maniflex.New(maniflex.Config{
Port: 8080,
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MountEndpoints: true, // mount POST/GET/DELETE /files — see the footgun below
},
})
FileStorage is a small interface — Store, Retrieve, Delete, Exists,
URL — making S3, R2, GCS, or any other key-value store straightforward to
adapt.
When Storage is nil, model endpoints still accept JSON, but multipart
uploads and the standalone /files routes respond with 501 Not Implemented.
MountEndpoints is explicit
MountEndpoints gates only the standalone /files routes and defaults to
false. Setting Storage alone is not enough to mount them:
| You set | Multipart on mfx:"file" fields | Per-model attachment routes | Standalone /files |
|---|---|---|---|
Storage only | ✅ enabled | ✅ mounted | ❌ not mounted (404) |
Storage + MountEndpoints: true | ✅ enabled | ✅ mounted | ✅ mounted |
MountEndpoints: true, Storage nil | ❌ 501 | ❌ 501 | ✅ mounted, returns 501 |
Migration footgun. Before this release a non-nil
FileStorageauto-mounted/files. That is no longer implied — if you relied on the standalone endpoints, addMountEndpoints: true. Model file fields and per-model attachment routes remain gated onStoragealone and are unaffected.
Upload size limits
A multipart request is capped at 32 MB in total by default, across every
part. Anything larger is rejected with 413 BODY_TOO_LARGE while it streams —
before a byte is written to a temp file, let alone to storage. Raise or lower
the ceiling per app:
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MaxUploadBytes: 100 << 20, // total request size (default 32 MB)
MaxUploadMemory: 8 << 20, // buffered in RAM before spooling (default 32 MB)
}
MaxUploadBytes bounds the request; the per-field max_size tag bounds an
individual attachment within it. Both apply, and the request ceiling is
checked first — so max_size:2MB on a field does not stop a client from sending
a 50 GB body, but MaxUploadBytes does.
For a tighter limit on one model, register
body.MaxBodySize on the Deserialize step; it
overrides MaxUploadBytes for the models it is scoped to.
Custom storage keys with KeyGen
By default a POST /files upload is stored under
uploads/<uuid>/<sanitised-filename>. Override KeyGen to control the layout —
for example, to shard by tenant read from the request context:
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MountEndpoints: true,
KeyGen: func(ctx *maniflex.ServerContext, h *multipart.FileHeader) string {
tenant := ctx.Auth.Claims["tenant"] // populated by a BeforeMiddleware
return fmt.Sprintf("%s/%s", tenant, h.Filename)
},
},
The returned string is used verbatim as the storage key — sanitise any
user-supplied component yourself (the default uses sanitizeFilename). KeyGen
applies only to the standalone POST /files route; per-model attachment keys
are framework-generated.
KeyGen controls the layout of a key; it does not bind the key to a caller.
The framework adds a scope prefix on top of whatever KeyGen returns — see
Key ownership and FilesConfig.KeyScope.
S3, R2, MinIO, DigitalOcean Spaces
The satellite module maniflex/storage/s3 ships a FileStorage implementation
backed by the AWS SDK v2. It works against any S3-compatible service.
import "github.com/xaleel/maniflex/storage/s3"
store, err := s3.New(ctx, s3.Config{
Bucket: "my-app-uploads",
Region: "us-east-1",
// Endpoint, UsePathStyle, KeyPrefix, ACL are all optional.
})
if err != nil { log.Fatal(err) }
server.SetStorage(store)
Credentials follow the standard AWS resolution chain (env vars, shared
config, IAM instance role, IRSA, ECS task role). Override with
Config.AWSConfig when you need a custom credential provider, HTTP client,
or retry policy.
Per-service tips:
| Service | Endpoint | UsePathStyle |
|---|---|---|
| AWS S3 | leave empty | false |
| MinIO | http://localhost:9000 | true |
| Cloudflare R2 | https://<account>.r2.cloudflarestorage.com | false |
| DigitalOcean Spaces | https://<region>.digitaloceanspaces.com | false |
Use KeyPrefix to share one bucket across environments
(KeyPrefix: "staging/") — callers pass logical keys and never see the
prefix. File metadata (filename, size, content type) is stored as native
S3 object metadata so objects remain browsable via the AWS console and
aws s3 cp without the maniflex layer.
How uploads work
A model containing one or more file fields accepts multipart/form-data on
create and update, in addition to JSON:
- Form fields named the same as JSON fields populate the row’s scalar values.
- Form file parts named after a
filefield are streamed to storage; the resulting key is written to the column.
Conceptually:
POST /api/articles
Content-Type: multipart/form-data; boundary=...
--...
Content-Disposition: form-data; name="title"
The First Post
--...
Content-Disposition: form-data; name="cover"; filename="hero.png"
Content-Type: image/png
<bytes>
--...
The response is the usual JSON envelope; the cover field carries the storage
key the client uses to fetch the file later.
The framework rejects an upload before it reaches storage if it violates the
field’s max_size or accept constraints.
Sending a pre-uploaded key
A file field also accepts a plain string in JSON — the storage key of a file
already uploaded via the standalone endpoint. This is useful when the upload
is decoupled from the record creation (large files uploaded ahead of time,
re-using an existing file, and so on).
The key is checked against storage before the write: setting a file field to
a string key that does not exist in the configured FileStorage is rejected with
422 FILE_NOT_FOUND, so a record can never reference a dangling key. (Pass JSON
null to clear the field.) In production the key exists because the client
uploaded it first — via the multipart part, a prior POST /files, or a
presigned upload. In tests, seed the
key into the shared storage before referencing it.
The field’s max_size and accept rules apply here too, checked against the
object actually in storage — the same bytes get the same answer whichever way they
arrived. This is new in v0.2.3: before it, this path checked only that the key
existed, so uploading out of band and referencing the key was a way past both
rules. See the changelog if you relied on that.
Key ownership
A storage key is bound to the principal that mints it, so a key one caller
learns cannot be pinned onto another caller’s record. Keys are not secrets — they
travel in signed URLs and file_acl:private responses — and without this a caller
who obtained another record’s (or another tenant’s) key could reference it on their
own record, and an auto_delete field could then delete a blob it never owned.
Every minting path — POST /files, a presigned upload, and a multipart upload
through a model — prefixes the key with a hash of the caller’s scope, and a
reference by a different scope is refused with 403 FILE_FORBIDDEN. The scope
defaults to ctx.Auth.TenantID (so a tenant’s members share one), else
ctx.Auth.UserID. Override it when your principal lives elsewhere:
FilesConfig: maniflex.FilesConfig{
Storage: store,
// Bind each key to the user rather than the tenant. Guard the nil case:
// an anonymous request has no principal, and returns an unscoped key.
KeyScope: func(ctx *maniflex.ServerContext) string {
if ctx.Auth == nil {
return ""
}
return ctx.Auth.UserID
},
}
The scope must resolve consistently across the mint request and the later
reference — if your POST /files auth and your model auth populate ctx.Auth
differently, read whatever both share.
A key minted while no principal was present is unscoped and referenceable by
anyone; returning "" from KeyScope opts a request out. Keys minted before
v0.2.3 carry no scope marker and are likewise left to the existence check, so
upgrading breaks no reference to an already-stored key — the guarantee holds for
every key minted under a principal from v0.2.3 on.
Direct-to-storage uploads (upload:presigned)
By default an upload travels client → app → storage, and the app holds the whole body while it does: the multipart form is drained before the handler runs, and the in-memory buffer defaults to the same 32 MB as the body cap, so nothing spools to disk either. A 60 MB video therefore costs 60 MB of server memory and two hops of bandwidth to store one object.
Add upload:presigned and the bytes go straight to the bucket:
type Post struct {
maniflex.BaseModel
Title string `json:"title"`
Video string `json:"video" mfx:"file,upload:presigned,accept:video/mp4,max_size:60MB"`
}
That mounts one extra route:
POST /posts/video/upload-url
There is no record id in that path, deliberately: a create-time file field has no record yet, so a record-scoped route could not serve one. The same route works for create and update.
The flow is two phases, and the second one is just an ordinary write:
① POST /posts/video/upload-url
{"filename": "clip.mp4", "content_type": "video/mp4", "size": 41231234}
→ 200 {
"url": "https://bucket.s3.amazonaws.com/",
"method": "POST",
"fields": { "key": "...", "policy": "...", "x-amz-signature": "..." },
"key": "uploads/<uuid>/clip.mp4",
"max_size": 62914560,
"expires_at": "2026-07-17T13:05:00Z"
}
② the client POSTs the file straight to `url` as multipart/form-data,
sending every entry of `fields` first and the file last
③ POST /posts {"title": "...", "video": "uploads/<uuid>/clip.mp4"}
→ the ordinary create, which verifies the object and stores the key
Phase ③ is the completion step, and there is nothing else to call: the record
either names the key or it does not. That is why no pending-upload state exists to
reconcile — if a client uploads and never completes, no record references the
object and nothing is corrupt (the object itself is an orphan; see auto_delete).
The field’s rules bind at both ends. At ① the declared content_type and
size are checked against accept and max_size, so a URL is never minted for a
file the field would refuse. The limits are then pinned into the signature —
S3’s POST policy carries a content-length-range, so S3 itself rejects an
oversize body — and at ③ the stored object’s real size and type are checked again.
That last check is the one that matters: a signature can only bound what the
backend enforces, and the record is what makes an object real.
The client never chooses the key. It is minted server-side through
FilesConfig.KeyGen (so a per-tenant prefix scheme covers presigned uploads too)
and returned in the response. A client that could name the key could aim its
upload at another record’s object.
Auth applies. The mint route runs Auth → handler → Response, so whatever
gates the model gates the minting — granting the right to write an object is not
something to leave unauthenticated. Note the operation is
maniflex.OpPresignUpload, not OpCreate: the mint is not the create and can
precede one by minutes, so scope middleware with ForOperation(OpPresignUpload)
if you need it there.
Backend support. Presigning requires a backend that can mint one.
storage/s3can (AWS S3, R2, MinIO, Spaces, …).LocalStoragecannot, and says so: the route answers501 PRESIGN_UNSUPPORTEDrather than handing back an unsigned URL, which would be an open write endpoint rather than a degraded presigned one. Use the ordinary multipart upload withLocalStorage.
Streaming uploads (upload:stream)
upload:presigned takes the bytes off the app entirely, which is ideal when the
client can do the two-phase dance and the backend can presign. When it cannot —
LocalStorage, or a client that only knows how to POST one multipart form — the
bytes still have to pass through the app, and by default the whole request is
buffered first (in memory up to MaxUploadMemory, the rest to temp files) before
a single byte reaches storage. upload:stream removes that landing: the field’s
part is piped straight into FileStorage.Store as it arrives off the socket.
type Post struct {
maniflex.BaseModel
Title string `json:"title"`
Video string `json:"video" mfx:"file,upload:stream,accept:video/mp4,max_size:60MB"`
}
Nothing about the request shape changes — it is the same multipart POST /posts
as an unflagged file field. Only where the bytes go changes: to the backend
directly, not to the app’s disk first.
Two rules bind differently when the length is not known until the stream ends:
acceptis checked first, from the part’s declared type (or a sniff of the first 512 bytes when it declares none), so a disallowed type is refused with415before anything is stored.max_sizeis enforced mid-stream: the upload is cut off and answered413the moment it runs past the limit. By then a partial object exists in storage — the request ends non-2xx, so the same orphan cleanup that protects every failed upload deletes it (see Automatic cleanup). A backend that can reject early (S3 rejects an oversize multipart part) still saves the bandwidth; the framework catches the rest.
Because the length is unknown while streaming, FileStorage.Store receives a
FileMeta.Size of 0 and must store an unsized reader — read it to EOF rather
than trusting Size. storage/s3 does this transparently (the AWS SDK uploader
switches to a multipart upload); LocalStorage copies the reader as-is. A custom
backend that assumed a known length needs to handle the unsized case before
enabling upload:stream against it.
Which one?
upload:presignedwhen the bytes should never touch the app and the backend can presign — the best choice for the very largest files.upload:streamwhen they must pass through (to scan, transform, or because the backend cannot presign) but should not land on disk on the way. Plainfile(buffered) for everything else — it is the default for a reason, and streaming trades a pre-storage validation gate for a stored-then-cleaned one. The twoupload:strategies are mutually exclusive on one field; setting both is a registration error.
Standalone file endpoints
When MountEndpoints is true, three routes are mounted under PathPrefix:
| Method | Path | Action |
|---|---|---|
POST | /files | upload a single file (multipart, field name file) |
GET | /files/{key...} | stream the file with its original content type |
DELETE | /files/{key...} | remove the file from storage |
POST /files returns 201 with
{"data": {"key": "...", "content_type": "...", "size": ..., "filename": "..."}}.
The returned key is the value to store in a file-tagged column. It streams
the file part straight to storage as it arrives (see
Streaming uploads), so a large standalone
upload never lands on the app server’s disk; size in the response is the length
measured while streaming. A custom FilesConfig.KeyGen here sees a
*multipart.FileHeader whose Size is 0, since the length is not yet known —
key off Filename/Header instead.
GET /files/{key...} streams the body with Content-Type and Content-Length
from the stored metadata. For safety it always sends X-Content-Type-Options: nosniff and serves only an allowlist of content types (common images, PDF,
plain text) inline; everything else — including text/html and
image/svg+xml — is sent as a Content-Disposition: attachment download so a
stored file cannot execute script on the API origin. Missing keys return 404.
These endpoints are storage-key-addressed and have no built-in auth
when BeforeMiddlewares is empty. Set it to wrap the routes with the
same pipeline middleware (e.g. JWT, role checks) that protects your model
endpoints:
maniflex.New(maniflex.Config{
FilesConfig: maniflex.FilesConfig{
Storage: fs,
MountEndpoints: true,
BeforeMiddlewares: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret, auth.JWTOptions{}),
auth.RequireRole("admin"),
},
},
})
Each middleware sees a synthesised ServerContext (Request, Writer, Ctx,
RequestID, logger — no Model/Operation, since these routes are outside
the model pipeline). Aborting the context short-circuits the request
before the file handler runs. Leaving BeforeMiddlewares empty keeps the
pre-fix behaviour for backward compatibility, but production deployments
should populate it — anyone who guesses a key could otherwise delete
arbitrary files. The server logs a warning at startup when /files is
mounted without BeforeMiddlewares.
Before vs. after middleware
BeforeMiddlewares run before the handler and own request control: they
can authenticate, populate ctx.Auth, or short-circuit (abort / set
ctx.Response) to replace the response entirely.
AfterMiddlewares run after the handler has served the request. Because
the handler streams its response (status, headers, body) straight to the
client, the response is already committed by the time an after-middleware runs —
so they are for observation and side effects only (audit logging, metrics,
cleanup), never for altering the response. Read the outcome with:
AfterMiddlewares: []maniflex.MiddlewareFunc{
func(ctx *maniflex.ServerContext, next func() error) error {
status := ctx.Writer.(interface{ Status() int }).Status()
log.Printf("file request completed: %d", status)
return next()
},
},
Setting ctx.Response from an after-middleware is ignored (and logged) rather
than corrupting the already-sent body. To alter, replace, or block the
response, use BeforeMiddlewares.
Per-model attachment routes
For each mfx:"file" field on each model, the framework mounts a record-
scoped download path:
GET /:model/:id/:file_field
E.g. GET /api/patients/123/discharge_summary streams the file referenced
by Patient.DischargeSummary for record 123.
Unlike GET /files/{key...}, this route runs through the read pipeline
for the parent record — the same Auth, soft-delete, and tenancy
middleware that protect GET /api/patients/123 also protect the download.
Use this for any attachment whose access depends on the parent row.
Response codes:
| Status | Meaning |
|---|---|
200 | file streamed with Content-Type, Content-Disposition, Content-Length |
206 | a Range request was satisfied — see Resumable downloads |
404 NOT_FOUND | the record does not exist (or is soft-deleted) |
404 FILE_NOT_SET | the record exists but the field is null/empty |
404 FILE_NOT_FOUND | the field references a key that is missing from storage |
401 / 403 | whatever the Auth middleware decided |
The route is only mounted when Config.FilesConfig.Storage is configured; with
no storage backend, the route is absent and requests return 404 from the
router. (Unlike the standalone /files endpoints, per-model attachment routes
do not require MountEndpoints.)
Internally this is dispatched as its own operation, maniflex.OpReadAttachment.
Middleware filtered by ForOperation(OpRead) does apply to attachment
requests: an attachment is a read of one record, so whatever decides who may
read that record decides who may download its file. The implication runs one way
only — ForOperation(OpReadAttachment) means attachment requests alone.
Before v0.2.5 an
OpRead-scoped middleware did not run for attachments, so tenancy written that way left the attachment route unscoped.
Resumable downloads (HTTP Range)
Both download routes — GET /files/{key...} and the per-model attachment
route — honour the Range request header, so a client can resume an
interrupted download or seek into a large media file instead of refetching it
from the start.
# the last 500 bytes
curl -H 'Range: bytes=-500' localhost:8080/api/files/uploads/<uuid>/clip.mp4
# resume from where a failed download stopped
curl -H 'Range: bytes=1048576-' localhost:8080/api/patients/123/scan
A satisfiable range answers 206 Partial Content:
| Header | Value |
|---|---|
Content-Range | bytes 1048576-2097151/8388608 — the window, and the object’s full length |
Content-Length | the window’s length, not the object’s |
Accept-Ranges | bytes (also sent on full 200 responses, so clients know they may resume) |
The security headers a full download sends — X-Content-Type-Options: nosniff
and the Content-Disposition inline/attachment decision — are identical on a
206.
Response codes:
| Status | When |
|---|---|
206 | the range resolved to a window of the object |
200 | no Range header, or one the server declined (see below) |
416 RANGE_NOT_SATISFIABLE | the range is well-formed but starts past the end of the object; Content-Range: bytes */<size> tells the client the real length |
Ranges the server declines. A malformed header, an unrecognised unit, and a
multi-range request (bytes=0-99,200-299) are all answered with the whole
object and 200 rather than an error — the spec permits ignoring any Range,
and serving several windows would require a multipart/byteranges body.
A client that asks for several windows gets the object and slices it itself.
Backend support
How much a range actually saves depends on the storage backend:
| Backend | Behaviour |
|---|---|
LocalStorage | seeks the file; only the window is read |
storage/s3 (S3, R2, MinIO, Spaces) | pushes the range into GetObject, so only the window leaves the bucket — you are not billed egress for bytes the client did not ask for |
A custom backend implementing RangeRetriever | same as above |
| A custom backend that does not | still works: the Range header is ignored and the whole object is served with 200, and Accept-Ranges is not advertised |
To support ranges in your own backend, implement the optional
maniflex.RangeRetriever alongside FileStorage:
func (s *MyStorage) RetrieveRange(
ctx context.Context, key string, offset, length int64,
) (io.ReadCloser, maniflex.FileMeta, error) {
// Return exactly the bytes [offset, offset+length).
}
The framework resolves the Range header against the size your Stat reports
before calling, so offset and length are always absolute and in bounds —
you never have to parse a header or clamp a window. Adding the method is all
that is needed; there is nothing to register.
Automatic cleanup
By default, a file field’s stored bytes are removed when:
- the record is hard-deleted, or
- the field is overwritten by an update.
Setting auto_delete:false opts out, leaving the file in storage for
out-of-band lifecycle management. Soft-deleted rows never trigger cleanup —
the file is preserved until the row is hard-deleted.
Bring-your-own storage
Implement maniflex.FileStorage:
type FileStorage interface {
Store(ctx context.Context, key string, r io.Reader, meta FileMeta) error
Retrieve(ctx context.Context, key string) (io.ReadCloser, FileMeta, error)
Delete(ctx context.Context, key string) error
Exists(ctx context.Context, key string) (bool, error)
Stat(ctx context.Context, key string) (FileMeta, error)
PresignUpload(ctx context.Context, key string, opts PresignUploadOptions) (*PresignedUpload, error)
URL(ctx context.Context, key string, ttl time.Duration) (string, error)
}
StatandPresignUploadare new in v0.2.3 and break every third-party backend until it implements them.
Statreturns an object’s metadata without fetching its body. Returnmaniflex.ErrFileNotFoundfor a missing key. ItsSizeandContentTypeare what afilefield’smax_sizeandacceptare checked against when a record references a key, so they must describe what is really stored — a client that uploaded 5 GB to a 60 MB field is caught here or nowhere. If your backend has no cheap metadata call,Retrieveand measure; it is correct, merely slower.
PresignUploadmints a direct-to-storage authorisation. If your backend cannot, returnmaniflex.ErrPresignUnsupported— the framework turns that into a clean501. Do not return an unsigned URL instead: that is not a degraded presigned upload, it is an unauthenticated write endpoint. Pinopts.MaxSizeinto the signature if you can (S3’s POST policy does, viacontent-length-range; a presigned PUT cannot), since the framework’s own check can only run once the bytes are already stored and paid for.
Retrieve returns maniflex.ErrFileNotFound when the key does not exist.
Delete should also return maniflex.ErrFileNotFound for missing keys so
the standalone DELETE /files/* handler can surface a 404 without an extra
Exists round-trip; backends that cannot detect the case atomically (e.g. S3
DeleteObject succeeds for missing keys) may return nil instead — both are
treated as “delete succeeded”. Store is given a framework-generated key of
the form uploads/<uuid>/<sanitised-filename>; create any intermediate
directories or object prefixes as needed.
Optional: RangeRetriever
Beyond FileStorage, a backend may implement maniflex.RangeRetriever to serve
byte windows for resumable downloads:
type RangeRetriever interface {
RetrieveRange(ctx context.Context, key string, offset, length int64) (io.ReadCloser, FileMeta, error)
}
It is optional and additive — a backend that omits it keeps working, serving the
whole object with 200 for a Range request. Implement it if your backend can
fetch a window natively (S3’s GetObject Range, a file Seek), since that is
where the saving is; the bounds are pre-resolved against your Stat size, so
they are always absolute and in range.
Storage backends are also expected to:
- honour
ctxcancellation inStore— long uploads must abort when the request deadline elapses or the server is shutting down, - reject keys ending in
.meta.jsonin bothStoreandRetrieveif the backend uses sibling JSON files as a metadata layout (LocalStorage’s case), so the framework’s internal layout is never reachable through the file handler.
Filenames flowing through the framework-generated key are sanitised to the
charset [A-Za-z0-9._-] (other runes become _), leading dots are stripped,
and the result is truncated to 120 characters. CR / LF / NUL bytes never
survive into the storage key.
File fields vs. static files
File fields handle user-supplied content. They are unrelated to Static Files, which serves a fixed directory of assets you ship with the app.
| File fields | Static files | |
|---|---|---|
| Source | uploaded at runtime | committed to the repo |
| Storage | FileStorage backend | local disk |
| URL | /files/<key> | /static/<path> |
| Configured by | Config.FilesConfig.Storage | a static/ directory |
Static Files
Alongside the generated model API, maniflex can serve a directory of plain static files — HTML, CSS, JavaScript, images, downloads — straight off disk. This is useful for a small admin page, a landing page, or assets referenced by an OpenAPI viewer, without standing up a separate web server.
How it works
Static serving is opt-in: set StaticDir to the directory you want served,
and every file inside it is served under the /static URL path.
server := maniflex.New(maniflex.Config{
StaticDir: "static", // serve ./static — nothing is served unless you name a dir
})
myapp/
├── main.go
└── static/
├── index.html → GET /static/index.html
├── css/app.css → GET /static/css/app.css
└── logo.png → GET /static/logo.png
Files are served verbatim, so a single-page app is served in full — its
index.html at the directory root and every nested asset at its own path:
static/
├── report.json → GET /static/report.json
└── admin/ → GET /static/admin/ (serves index.html)
├── index.html
└── scripts/app.js → GET /static/admin/scripts/app.js
If StaticDir is left empty, no static route is mounted and maniflex serves only
the API. If it is set but the directory does not exist, the server logs a warning
and skips the mount; the rest of the API is unaffected.
Changed in v0.2.1. Static serving used to default to
<cwd>/static— anystatic/directory in the working directory was published at/static/automatically. It is now opt-in: setStaticDirexplicitly. If you relied on the old default, addStaticDir: "static".
Customising the directory and prefix
Three maniflex.Config fields control static serving:
server := maniflex.New(maniflex.Config{
StaticDir: "public", // serve ./public
StaticPrefix: "/assets", // under /assets instead of /static
})
| Field | Default | Effect |
|---|---|---|
StaticDir | "" | filesystem directory served; empty serves nothing. A relative path resolves against cwd |
StaticPrefix | /static | URL prefix the directory is mounted under (at the router root) |
StaticDisabled | false | set true to turn serving off even when StaticDir is set |
StaticDisabled exists so an app that sets StaticDir unconditionally can still
flip serving off from an env var or flag without clearing the field.
The /static route
A few details follow from how the route is mounted (the buildRouter block in
router.go):
- Resolved from the working directory. A relative
StaticDirresolves against<cwd>, wherever the process was started — not the location of the binary. Run the server from the project root, orcdthere first, so a relative path like"static"is found. - Mounted outside
PathPrefix. Static files live at/static/...(or yourStaticPrefix), not/api/static/.... ThePathPrefixfrommaniflex.Configscopes only the model API and/openapi.json; the static mount sits at the router root. - Trailing-slash redirect. A request to
/static(no trailing slash) is301-redirected to/static/. Requests below it are served directly. - Directory listing. Because it is backed by Go’s
http.FileServer, a request for a directory with noindex.htmlreturns a file listing. Add anindex.htmlto each directory you do not want browsable — and only pointStaticDirat a directory whose whole contents are safe to publish.
Static files vs. file uploads
Static serving is for assets you ship with the app. It is unrelated to the
file-upload feature, which stores user-submitted files and is wired up
separately through Config.FilesConfig.Storage and the /files endpoints. For
user uploads see File Fields & Uploads.
| Static files | File uploads | |
|---|---|---|
| URL | /static/* | /files/* |
| Source | a directory you commit and name in StaticDir | user POSTs at runtime |
| Configured by | Config.Static* | Config.FilesConfig.Storage |
| Use for | app assets, admin pages | avatars, attachments |
Localization
maniflex has first-class support for multilingual string fields. A single
maniflex.LocaleString field stores all translations in one JSON column and
the framework resolves the right one for each request automatically.
The LocaleString type
maniflex.LocaleString is a map[string]string where each key is a locale
code and each value is the translation for that locale:
{ "en": "Finance", "ar": "مالية", "fr": "Finance" }
On SQLite it is stored as TEXT (JSON-encoded). On Postgres it is stored as
JSONB, which allows GIN-indexed key lookups.
Declare a field as locale-aware with the locale directive:
type Department struct {
maniflex.BaseModel
Name maniflex.LocaleString `json:"name" mfx:"locale,filterable,sortable"`
Code string `json:"code" mfx:"required,unique"`
}
On create and update the client sends the full map:
{ "name": { "en": "Finance", "ar": "مالية" }, "code": "FIN" }
On read the framework emits a locale-resolved view (see Response modes).
Response modes
Every LocaleString field has a response mode that controls the shape of
the field in API responses. The mode is resolved in this order:
- The field’s own
mfxtag (split,resolve, ordynamic) - The model’s
ModelConfig.DefaultLocaleMode - The app’s
LocaleOptions.DefaultLocaleMode - The framework default:
split
split (default)
Two keys are emitted in the response:
name— the resolved string for the effective localename_i18n— the fullmap[string]string(always present)
{
"name": "Finance",
"name_i18n": { "en": "Finance", "ar": "مالية" }
}
The resolved string gives display code a stable string type; the companion
_i18n map gives the editor everything it needs to build a translation form.
The companion suffix defaults to "_i18n" and is configurable via
LocaleOptions.SplitSuffix.
Writing a split-mode field back
Both keys are understood on write, so a client can PATCH the object it just GETed without special-casing localized fields:
| Body | Stored |
|---|---|
"name_i18n": {"en":"Finance","ar":"مالية"} | that map, verbatim — the companion wins |
"name": "Finance" (a bare string) | {"<effective locale>": "Finance"} |
"name": {"en":"Finance"} (a map) | that map |
The companion takes precedence because it is the complete value, while name
is one locale’s rendering of it. That is what makes an echoed response lossless:
the translations the response did not show still come back in _i18n.
A bare string replaces the column rather than merging into the stored map —
the same as a map write, which has never been a per-key patch. So sending only
"name": "Finance" with ?locale=en leaves the field as {"en": "Finance"}
and drops any other translations. Send the _i18n map when you mean to keep
them.
Before v0.2.5 the
_i18nkey was ignored on write and a bare string was stored as a JSON scalar, which the next read could not parse — a GET→PATCH round-trip through a generic edit form left the record unreadable, returning 500 for that record and for the whole collection, since one bad row fails the list scan. A column still holding a scalar from that era now resolves to its value and logs a warning instead of erroring, so the row can be repaired with an ordinary PATCH.
resolve
The field is always a plain string — the resolved value for the effective locale. No companion field is emitted.
{ "name": "Finance" }
Use resolve when clients only ever need one language and the extra _i18n
key adds no value.
dynamic
Replicates legacy behaviour:
- When
?locale=is present: emits a string (resolved for that locale) - When
?locale=is absent: emits the full map
The field type is non-deterministic. Not recommended for new models.
Setting up the LocaleResolver
Install the LocaleResolver middleware on the Deserialize step before
registration, so it runs before the framework’s built-in Deserialize:
server.Pipeline.Deserialize.Register(maniflex.LocaleResolver(maniflex.LocaleOptions{
Supported: []string{"en", "ar", "fr"},
Default: "en",
FromHeader: true,
RTL: []string{"ar", "he", "fa", "ur"},
}))
LocaleOptions fields:
| Field | Type | Default | Purpose |
|---|---|---|---|
Supported | []string | all locales | Whitelist of accepted locale codes; locales not in this list fall back to Default |
Default | string | "en" | App-wide fallback locale used when the request carries no recognisable preference |
FromHeader | bool | false | Also parse Accept-Language; first match in Supported wins (quality values are ignored) |
RTL | []string | — | Locale codes with right-to-left script; matching requests get "_dir":"rtl" in response meta |
DefaultLocaleMode | LocaleMode | split | App-wide default mode for all LocaleString fields |
SplitSuffix | string | "_i18n" | Companion-field suffix used in split mode |
Locale resolution chain
When resolving which string to return, the framework walks this chain (most to least specific) and returns the first non-empty match:
- Explicit
?locale=query parameter Accept-Languageheader (first match inSupported), whenFromHeader: true- Field’s
default_locale:codetag - Model’s
ModelConfig.DefaultLocale - App’s
LocaleOptions.Default(default"en") - Any non-empty value in the map (last resort)
// Field-level default: Arabic is preferred for this field even when the
// request does not specify a locale.
Bio maniflex.LocaleString `json:"bio" mfx:"locale,default_locale:ar"`
// Model-level default: all locale fields on this model use French by default,
// unless overridden by a field's `default_locale` tag.
server.MustRegister(Article{}, maniflex.ModelConfig{DefaultLocale: "fr"})
Requiring a locale key
Use validate.RequireLocale to enforce that specific locale keys are present
and non-empty on create (or update) requests:
server.Pipeline.Validate.Register(
validate.RequireLocale("name", "en"),
maniflex.ForModel("Department"),
maniflex.ForOperation(maniflex.OpCreate),
)
A request that omits the "en" key, or supplies an empty string for it, is
rejected with HTTP 422 MISSING_LOCALE. Pass multiple keys to require several
locales at once:
validate.RequireLocale("name", "en", "ar")
Filtering and sorting locale fields
LocaleString fields tagged filterable and sortable work with the
standard query string grammar. In split and resolve mode the framework
automatically targets the effective locale’s JSON key in the database query.
GET /departments?filter=name:ilike:%25fin%25&sort=name:asc
In the example above, when the effective locale is "en", the adapter runs:
- SQLite:
json_extract("departments"."name", '$.en') LIKE '%fin%' - Postgres:
"departments"."name"->>'en' ILIKE '%fin%'
You can also filter a specific locale key explicitly:
GET /departments?filter=name.ar:contains:مال
In dynamic mode without an explicit ?locale= the filter hits the raw JSON
column, which typically returns no results for plain-string comparisons — this
is intentional: in dynamic mode the field’s meaning depends on request context.
Searching localized content
The mfx:"searchable" full-text directive (the ?q= endpoint) indexes plain
string columns only — tagging a LocaleString field searchable fails at
model registration, because full-text search has no single text column to index:
maniflex: model "Department" field "Name" is mfx:"searchable" but its type is
maniflex.LocaleString; full-text search only indexes text (string) columns
Two ways to make localized content searchable:
- Substring filter (simplest). Tag the
LocaleStringfieldfilterableand use?filter=name:ilike:%25term%25. Because the value is stored as one JSON object, anilikeagainst the raw column matches across every locale’s text at once — no extra schema. - Denormalized plaintext column. Add a plain
stringcolumn (e.g.SearchText) taggedsearchable, and populate it on create/update from a Service-step middleware that flattens the localized strings. Use this when you need true?q=full-text ranking rather than substring matching.
RTL meta
When the resolved locale is in LocaleOptions.RTL, every response envelope
gains a meta object with "_dir": "rtl" — for both list responses
(which already carry pagination in meta) and single-record responses
(read / create / update):
{
"data": {
"name": "مالية",
"name_i18n": { "en": "Finance", "ar": "مالية" }
},
"meta": { "_dir": "rtl" }
}
List responses include pagination alongside the direction flag:
{ "data": [...],
"meta": { "total": 5, "page": 1, "limit": 20, "pages": 1, "_dir": "rtl" } }
Clients can use meta._dir to switch text direction without needing to know
which locale is active.
Model-level mode override
Set a uniform mode for all LocaleString fields on a model without tagging
each one individually:
server.MustRegister(LegacyArticle{}, maniflex.ModelConfig{
DefaultLocaleMode: maniflex.LocaleModeDynamic,
})
Field-level tags take precedence over the model setting, which in turn takes
precedence over the app-level LocaleOptions.DefaultLocaleMode.
Full example
type Product struct {
maniflex.BaseModel
Name maniflex.LocaleString `json:"name" mfx:"locale,filterable,sortable"`
Description maniflex.LocaleString `json:"description" mfx:"locale,resolve"`
SKU string `json:"sku" mfx:"required,unique"`
}
func main() {
server := maniflex.New(maniflex.Config{...})
server.Pipeline.Deserialize.Register(maniflex.LocaleResolver(maniflex.LocaleOptions{
Supported: []string{"en", "ar"},
Default: "en",
FromHeader: true,
RTL: []string{"ar"},
}))
server.MustRegister(Product{})
db, _ := sqlite.Open("store.db", server.Registry())
server.SetDB(db)
server.Start()
}
With this setup:
GET /productsreturnsnameas the resolved English string plusname_i18nwith all translations;descriptionis always a resolved English string.GET /products?locale=arresolves both fields to Arabic.POST /productswith{"name":{"en":"Laptop"},"sku":"LAP-01"}succeeds.
Pipeline Overview
Every HTTP request handled by a generated model route flows through the same six-step pipeline. Each step has a default behaviour supplied by the framework and a registry of user middleware that can run before, after, or in place of it. This page describes what each step is responsible for; later pages cover how to register middleware on them and how state flows between them.
The six steps
Auth → Deserialize → Validate → Service → DB → Response
| Step | Default behaviour |
|---|---|
| Auth | Pass-through. Populates nothing by default. User middleware sets ctx.Auth here. |
| Deserialize | Parses URL query parameters (page, limit, filter, sort, include) into ctx.Query. On POST/PATCH, reads the JSON body into ctx.ParsedBody (limit: 4 MB), or parses multipart/form-data into ctx.ParsedBody and ctx.Files. |
| Validate | For create and update, enforces the mfx: tag rules: strips readonly and id, strips immutable on update, checks required, enum, min, max. |
| Service | Pass-through. Reserved for business logic supplied by user middleware. |
| DB | Dispatches to the configured adapter for the current operation — FindMany, FindByID, Create, Update, or Delete. Routes through ctx.Tx when a transaction is active. |
| Response | Builds the JSON envelope from ctx.DBResult and writes it to the http.ResponseWriter. |
The OpenAPI endpoint (GET /openapi.json) has its own three-step pipeline —
Auth → Generate → Response — accessible via server.Pipeline.OpenAPI. The
model-route pipeline described here is the one used for everything else.
Operations
The CRUD operation a request performs is identified by an Operation value
that is stable across all six steps:
| Operation | Triggered by |
|---|---|
OpList | GET /<table> |
OpRead | GET /<table>/{id} |
OpCreate | POST /<table> |
OpUpdate | PATCH /<table>/{id} |
OpDelete | DELETE /<table>/{id} |
OpOptions | OPTIONS /<table> or OPTIONS /<table>/{id} |
OpReadAttachment | GET /<table>/{id}/<file_field> — per-model attachment download (see Files) |
OpAction | a custom action endpoint registered with server.Action() |
A HEAD request has no operation of its own: it is GET with the body
suppressed, so it runs as OpList (collection) or OpRead (item) — same
middleware, same status code, no body. The OpHead constant still exists but is
never set on a request; scope HEAD-aware middleware with OpRead / OpList.
ctx.Operation is the value middleware uses to branch behaviour. OpAction
requests follow a trimmed pipeline (Auth → action handler → Response); the
Deserialize, Validate, Service, and DB steps are skipped for them.
Per-step responsibilities
Auth
The Auth step is the place to verify a token, look up a user, and set
ctx.Auth. The default handler does nothing; an unauthenticated request reaches
the DB layer with ctx.Auth == nil. Add a middleware here to reject anonymous
callers, populate identity, or check scopes.
Deserialize
The Deserialize step assembles request input from three sources:
- The URL query string becomes a
*QueryParamsonctx.Query. Filter and sort references are validated against the model’s tag-derived field lists. - A JSON body becomes
ctx.ParsedBody(a read-only*RequestBody, JSON-keyed) and is bound to the typed recordctx.Record. Bodies over 4 MB are rejected as413 BODY_TOO_LARGE; a genuine I/O failure reading the body is400 BODY_READ_ERROR. - A multipart body populates both
ctx.ParsedBody(the form fields) andctx.Files(the file parts). The form-field-to-file-field mapping is by name.
Reads carry no body, so only ctx.Query is populated for OpList / OpRead.
Validate
The Validate step runs only on OpCreate and OpUpdate. It applies the rules
declared by mfx: tags to ctx.ParsedBody:
readonlyfields and theidcolumn are silently stripped.immutablefields are stripped on update.requiredfields must be present on create.enum,min,maxare checked when the value is present.
Validation failures abort the pipeline with 422 Unprocessable Entity and a
details payload listing every offending field.
Service
The Service step has no default behaviour — it exists for application logic.
Hashing a password before persistence, charging a payment, recomputing a
derived total, calling an external API: all of these belong here. A Service
middleware that needs to short-circuit the request calls ctx.Abort(...) and
returns without invoking next().
DB
The DB step is the only step with side effects on the database. It selects the
operation matching ctx.Operation, builds the column-keyed write set from
ctx.Record (falling back to ctx.ParsedBody), calls the adapter, and writes
the result into ctx.DBResult — a *ListResult for lists, otherwise the record
(a typed *T on reads). When ctx.Tx is set, the call is routed through the
transaction; otherwise the bare adapter is used.
Two error classes are normalised at this step:
maniflex.ErrNotFoundbecomes404 NOT_FOUND.*maniflex.ErrConstraintis split by kind: a unique or foreign-key violation becomes409 CONFLICT, while a NOT NULL violation becomes422 VALIDATION_ERROR(a missing-required-value problem, not a conflict).
A cancelled context becomes 504 TIMEOUT — unless the cancellation came from the
client hanging up, which is 499 with no body (see
Error Handling). All other adapter errors surface as
500 DB_ERROR.
Response
The Response step serialises ctx.DBResult into an APIResponse and writes it
to the wire. List responses include a meta block with total, page,
limit, and pages; single-record responses do not. The standard envelope is
{"data": ...} for success and {"error": {...}} for failure.
Short-circuiting
Any middleware can stop the pipeline by setting ctx.Response (typically via
ctx.Abort(status, code, message)) and returning without calling next().
Subsequent steps are skipped and the Response step writes the prepared error
envelope. This is the standard mechanism for unauthorised requests, validation
failures inside Service middleware, and any other refusal that should not
reach the database.
Per-step middleware
Each step exposes a *StepRegistry on server.Pipeline:
server.Pipeline.Auth.Register(jwtAuth)
server.Pipeline.Service.Register(hashPassword,
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate))
Registered middleware can run Before the default (the default position), After it, or Replace it entirely. Scoping by model and operation is covered in Writing Middleware.
Next
- ServerContext — the object threaded through every step.
- Writing Middleware —
Register, options, positions. - Transactions — wrapping the DB step in a transaction.
- Error Handling — the error envelope and sentinel errors.
ServerContext
*maniflex.ServerContext is the object threaded through every pipeline step for one
HTTP request. Steps read from it, write to it, and call next() to proceed.
Middleware does the same. This page documents the fields and methods middleware
will commonly touch.
Lifecycle
A new ServerContext is constructed by the handler for every request, populated
incrementally by the pipeline, and discarded once the response is written. It
is not safe to share across requests or goroutines.
The fields populated by each step are:
| Step | Sets |
|---|---|
| handler (before Auth) | Request, Writer, Ctx, Model, Operation, ResourceID, RequestID, TraceID |
| Auth | Auth (when user middleware populates it) |
| Deserialize | RawBody, ParsedBody, Query, Files |
| Service | (whatever user middleware sets) |
| DB | DBResult, possibly Tx |
| Response | Response, then writes it to Writer |
Routing context
Set by the handler before Auth runs; safe to read in any step.
| Field | Meaning |
|---|---|
Request | the original *http.Request |
Writer | the underlying http.ResponseWriter |
Ctx | the request context.Context; cancellation propagates from here |
Model | the *ModelMeta for the resource — name, table, fields, relations |
Operation | the Operation being performed (OpCreate, OpList, …) |
ResourceID | the {id} path parameter, empty for list and create |
RequestID | chi’s request ID, echoed in X-Request-Id |
TraceID | the W3C traceparent header, when present |
A Singleton has no {id} to read, so ResourceID is pinned to the
SingletonID placeholder. For an unscoped singleton that placeholder is the
row’s real id. For a scoped one it is not — there is one row per scope, and
the DB step swaps in the caller’s real id once the forced filters are known.
Middleware running before that swap — anything at the DB pipeline’s Before
position — therefore sees a value that addresses no row. Call
ctx.ResolveResourceID() instead of reading the field directly when you need an
id you can query with:
id := ctx.ResolveResourceID() // "" if the scoped row is not provisioned yet
It returns ctx.ResourceID unchanged for every non-singleton request, and never
provisions a row.
For it to resolve anything, the scope must already be established — register the
scope middleware with maniflex.ProvidesScope():
server.Pipeline.DB.Register(
db.Tenancy("org_id", tenantFromAuth),
maniflex.ProvidesScope(),
)
That hoists it to run right after Deserialize instead of at the DB step, so Validate, Service and the DB step all see the same scope. Without it the scope still applies to the query, but nothing before the DB step can ask what it is.
Step outputs
Populated in order by the pipeline.
| Field | Populated by | Type |
|---|---|---|
RawBody | Deserialize | []byte — the raw request bytes |
ParsedBody | Deserialize | *RequestBody — read-only JSON-keyed body (mutate via SetField) |
Record | Deserialize | the typed record carrier (*T for ctx.Model) bound from the body |
Query | Deserialize | *QueryParams — pagination, filters, sorts, includes |
Files | Deserialize (multipart only) | map[string]*UploadedFile |
DBResult | DB | *ListResult for lists; the record otherwise (a typed *T on reads) |
Response | Response | *APIResponse — the envelope written to the wire |
Setting Response from any step causes the remaining steps to skip and the
prepared envelope to be written. See Abort below.
Auth
Auth *AuthInfo is populated by Auth middleware. When nil, the request is
anonymous.
type AuthInfo struct {
UserID string
Roles []string
Claims map[string]any
TenantID string
IdentityType AuthIdentityType // human, service_account, anonymous
Scopes []string
SessionID string
AuthMethod string // "jwt", "api_key", "session", …
}
ctx.HasRole(role string) bool is a convenience wrapper that returns false
when Auth is nil.
Transactions
Tx Tx carries the active transaction, if any. When set, the default DB step
routes through it. ctx.BeginTx(ctx.Ctx, opts) returns a Tx and is the
standard way for middleware to start one. See Transactions
for the full pattern.
Aborting the pipeline
ctx.Abort(status int, code, message string) populates ctx.Response with an
error envelope. The current middleware must then return nil without calling
next(). Subsequent steps are skipped; the Response step writes the prepared
error.
if header == "" {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing token")
return nil
}
Calling next() after Abort
Abort does not stop the pipeline — it only populates ctx.Response. If the
middleware calls next() afterwards, the chain continues exactly as if the
abort had not happened:
- The remaining steps still execute, with all their side effects. The DB step will still issue its query and possibly modify the database; a Service middleware will still call out to external services.
- Any of those steps may overwrite
ctx.Response— for example, the DB step replaces it with a404 NOT_FOUNDif the record is missing, or the default Response step builds a200 OKenvelope fromctx.DBResult. Whichever step writes last wins. - If nothing downstream touches
ctx.Response, the original abort envelope is preserved and sent to the client — but the side effects have already happened.
The result is almost always a bug: either the client sees a misleading status
(a write succeeded but the response claims it was rejected), or the database
is mutated by a request that was meant to be refused. Always return without
next() after Abort.
Reading input
Three helpers wrap common request reads:
| Method | Purpose |
|---|---|
BindJSON(v any) error | decode the body into v, enforcing the 4 MB limit |
EnsureRawBody() ([]byte, error) | return RawBody, reading/buffering/restoring the body if an earlier step hasn’t (for middleware that needs the raw bytes in a trimmed action/search pipeline) |
URLParam(name string) string | read a chi URL parameter |
QueryParam(name string) string | read a URL query parameter |
BindJSON calls Abort internally on error and returns a non-nil error so the
caller can return nil immediately.
The request body
ctx.ParsedBody holds the deserialized JSON (or multipart form) body as a
*RequestBody. It is read-only: there is no exported way to index or assign
it, so a stray ctx.ParsedBody["x"] = y is a compile error. This is deliberate.
The body is mirrored onto a typed record (ctx.Record), and the only mutators —
ctx.SetField / ctx.DeleteField — keep both in sync; writing the map directly
would update one and not the other, and the change could be silently dropped at
the DB step.
Reading
| Call | Returns |
|---|---|
ctx.Field(name string) (any, bool) | one field by its JSON name |
ctx.ParsedBody.Has(name) bool | whether a key is present (an explicit null counts) |
ctx.ParsedBody.Keys() []string / .Len() int | the top-level key set |
ctx.ParsedBody.Map() map[string]any | a copy of the body, for read-only consumers |
All are nil-safe: ctx.ParsedBody is nil for body-less requests (GET, DELETE)
and the readers return zero values rather than panicking.
For typed access, read the whole body as the concrete model struct:
u, ok := maniflex.For[User](ctx) // (*User, bool) — false if no User body is bound
u, err := maniflex.Bind[User](ctx) // (*User, error) — errors when absent
// or adapt a typed handler straight into middleware:
server.Pipeline.Service.Register(
maniflex.Handle(func(ctx *maniflex.ServerContext, u *User) error {
if u.Age < 18 {
ctx.Abort(http.StatusUnprocessableEntity, "TOO_YOUNG", "must be 18+")
}
return nil
}),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
Writing
Middleware that injects or rewrites a field must go through these setters so the value reaches both the body and the typed record (and so the DB step persists it):
| Call | Effect |
|---|---|
ctx.SetField(name string, value any) | set a field by its JSON name |
ctx.DeleteField(name string) | remove a field (e.g. strip an input-only field) |
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
ctx.SetField("owner_id", ctx.Auth.UserID) // force the owner server-side
return next()
}, maniflex.ForOperation(maniflex.OpCreate))
Cross-step storage
For state that one middleware needs to pass to another:
ctx.Set("invoiceID", inv.ID)
// later, in another middleware:
id, ok := ctx.Get("invoiceID")
The store is per-request and discarded with the context.
Direct database access
Middleware that needs to reach beyond ctx.Model — to read another model, run
a raw query, or take a row lock — has four entry points, all routed through
ctx.Tx when one is active:
| Method | Purpose |
|---|---|
GetModel(name string) *ModelAccessor | CRUD on any registered model (.List / .Read / .Create / .Update / .Delete) |
RawQuery(sql string, args ...any) ([]map[string]any, error) | parameterised SELECT, CTE-SELECT, or a data-modifying statement with RETURNING (e.g. UPDATE … RETURNING id) |
RawExec(sql string, args ...any) (int64, error) | parameterised non-SELECT (returns rows affected) |
LockForUpdate(modelName, id string) (map[string]any, error) | pessimistic row lock; requires ctx.Tx |
GetModel returns an accessor whose methods route through ctx.Tx when set,
so middleware in a transaction does not have to thread the Tx manually.
Typed cross-model helpers
GetModel(name) is dynamic — string-named, exchanging map[string]any. For
compile-time types use the generic free functions, which resolve the model from
the type parameter and route through ctx.Tx the same way (so they also
participate in maniflex.Batch):
u, err := maniflex.Read[User](ctx, id) // *User
all, err := maniflex.List[User](ctx, nil) // []*User
created, err := maniflex.Create(ctx, &User{Name: "Jane"})
maniflex.Update(ctx, id, &User{ /* full record */ })
maniflex.Delete[User](ctx, id)
Results that are not a registered model — raw SQL, aggregates, recursive
queries — use maniflex.Row (an alias for map[string]any); RawQuery,
Aggregate, and RecursiveQuery return []maniflex.Row.
Placeholders in raw SQL are rebound to the adapter’s dialect, so ? works on
both SQLite and Postgres ($N). Always pass values as args — never interpolate
them into the query string.
Logging
ctx.Logger() *slog.Logger returns a slog logger pre-seeded with
request_id, trace_id, and service attributes, so log lines emitted from
middleware are correlated automatically.
ctx.Logger().Info("payment captured",
slog.String("invoice_id", inv.ID),
slog.Float64("amount", inv.Total),
)
Service name
ctx.ServiceName() returns the Config.ServiceName configured on the server.
Middleware uses this to enrich audit records or outgoing requests without
holding a reference to the framework Config.
Next
- Writing Middleware — composing middleware on these fields.
- Transactions —
ctx.Tx,BeginTx,LockForUpdate. - Error Handling —
Abortand the response envelope.
Writing Middleware
A middleware is a function that runs as part of one of the six pipeline steps. It can inspect and modify the request context, call into the database, decide whether to proceed, and inject behaviour before or after the step’s default handler.
Signature
type MiddlewareFunc func(ctx *maniflex.ServerContext, next func() error) error
A middleware does one of two things:
- Continue the pipeline — perform its work, then call
next()and return the result. The chain executes the remaining middleware in the step and then the steps after it. - Short-circuit — set
ctx.Response(typically viactx.Abort(...)) and returnnilwithout callingnext(). The remaining steps are skipped and the prepared response is written to the wire.
func bearerToken(ctx *maniflex.ServerContext, next func() error) error {
header := ctx.Request.Header.Get("Authorization")
if !strings.HasPrefix(header, "Bearer ") {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing bearer token")
return nil
}
ctx.Auth = &maniflex.AuthInfo{UserID: parseSubject(header)}
return next()
}
Typed middleware
For middleware that works with the request body, maniflex.Handle[T] adapts a
typed handler into a MiddlewareFunc. It hands you the bound record as a
concrete *T (the same value ctx.Record holds) instead of a map, runs only
when a *T body is bound, and is skipped on body-less operations:
server.Pipeline.Service.Register(
maniflex.Handle(func(ctx *maniflex.ServerContext, u *User) error {
if u.Age < 18 {
ctx.Abort(http.StatusUnprocessableEntity, "TOO_YOUNG", "must be 18+")
}
return nil
}),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
To change a field, call ctx.SetField(name, value) rather than mutating the
struct, so the write reaches both the body and the record (see
ServerContext › The request body). For an ad-hoc typed read inside
a plain middleware, use maniflex.For[T](ctx) / maniflex.Bind[T](ctx).
Registration
Each pipeline step exposes a *StepRegistry on server.Pipeline. Register a
middleware on the step where its work belongs:
server.Pipeline.Auth.Register(bearerToken)
Without options, the middleware applies to every model and every operation.
Register must be called before Start() or Handler(). Building the router
closes the registration window: each step’s chain is composed once per
(model, operation) and cached from then on, rather than rebuilt on every request,
so the middleware set has to stop changing. A Register after that point panics —
it could otherwise only apply to some requests and not others, and it would be
mutating a slice live requests are reading.
server.Pipeline.Auth.Register(bearerToken) // fine
server.Start() // window closes
server.Pipeline.DB.Register(audit) // panics
Scoping
Two functional options narrow the scope. They are independent and may be combined.
ForModel(names ...string)
Restrict to one or more models, by struct name:
server.Pipeline.Service.Register(hashPassword, maniflex.ForModel("User"))
ForOperation(ops ...Operation)
Restrict to specific operations:
server.Pipeline.Auth.Register(requireToken,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Operation values: OpList, OpRead, OpCreate, OpUpdate, OpDelete,
OpOptions, OpAction. Registering on Validate/Service/DB with OpAction has
no effect — those steps are skipped for action endpoints. A HEAD request runs
as the GET it mirrors (OpRead / OpList), so scope it with those.
Combining
server.Pipeline.Service.Register(chargePayment,
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpCreate),
)
Position
By default, a middleware runs before the step’s default handler. Use
AtPosition to change that.
| Position | When the middleware runs |
|---|---|
maniflex.Before (default) | before the default handler |
maniflex.After | after the default handler |
maniflex.Replace | instead of the default handler — the step’s built-in behaviour is skipped |
// Run after the DB step succeeds — useful for audit logs.
server.Pipeline.DB.Register(auditLog,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
// Replace the default DB step for one model entirely.
server.Pipeline.DB.Register(customDispatch,
maniflex.ForModel("LegacyOrder"),
maniflex.AtPosition(maniflex.Replace),
)
Within a step, all matching Before middlewares run in registration order,
then the core handler (default or Replace), then all matching After
middlewares in registration order. If multiple Replace middlewares match,
the last one registered wins.
A Replace on the DB step takes over feeding the Response step, so it must
leave ctx.DBResult in the shape that step expects: a *maniflex.ListResult for
a list, and a record (map[string]any or a *T) for a read, create, or update.
Anything else is rejected with 500 INVALID_DB_RESULT naming the type it got. On
a ListResult you need only set Items and Total — a missing or partial
Query is filled in with the default page and limit.
Naming for traces
maniflex.WithName("name") attaches a human label to a middleware for use in
pipeline trace logs (enabled via Config.Trace). It does not change runtime
behaviour:
server.Pipeline.Auth.Register(rateLimit, maniflex.WithName("rate-limiter"))
Step-specific guidance
| Step | What middleware here typically does |
|---|---|
| Auth | Verify a token, populate ctx.Auth, reject unauthenticated requests. |
| Deserialize | Rarely customised. After middleware can rewrite the body via ctx.SetField / ctx.DeleteField. |
| Validate | Custom validation that goes beyond mfx: tags. Abort with 422 on failure. |
| Service | Business logic — derive fields, call external services, start transactions (maniflex.WithTransaction). |
| DB | Hooks around the database call. After middleware sees ctx.DBResult; Replace substitutes a different backend. |
| Response | After middleware can add headers; Replace lets you write a non-envelope response. |
After-middleware error handling
An After middleware sees ctx.Response when the default step has populated
it. Inspect it to decide whether to act:
func auditLog(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil {
return err
}
// don't audit failed writes
if ctx.Response != nil && ctx.Response.StatusCode < 400 {
record(ctx)
}
return nil
}
Per-model middleware at registration
For middleware that belongs to exactly one model, ModelConfig.Middleware
attaches hooks scoped to that model at registration time, avoiding the separate
Register call:
server.MustRegister(
Order{}, maniflex.ModelConfig{
Middleware: &maniflex.ModelMiddleware{
Validate: []maniflex.MiddlewareFunc{checkStock},
Service: []maniflex.MiddlewareFunc{chargePayment},
},
},
)
Both forms are equivalent; choose whichever keeps the declaration close to the code that depends on it.
Built-in middleware
Several middleware functions ship with the framework or its satellite modules — JWT auth, password hashing, audit logging, CORS, and more. They are documented in Middleware Catalogue.
Next
- ServerContext — the fields a middleware reads and writes.
- Transactions —
maniflex.WithTransactionas a Service-step middleware. - Error Handling — what
Abortproduces and how it propagates.
Transactions
A transaction wraps one or more database operations so they either all commit or all roll back. In maniflex the unit of transactional work is normally a single request: every database call performed by its pipeline runs in the same transaction, and the transaction commits if and only if the request produces a successful response.
Enabling transactions
The shipped middleware maniflex.WithTransaction wraps the DB step in a
transaction. Register it on the Service step, scoped to the operations that
should be transactional:
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil), // nil = default isolation level
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Once registered, every matching request executes the DB step (and any subsequent After-DB middleware) inside a transaction. The middleware:
- Begins a transaction before calling
next(). - Stores it on
ctx.Txand the underlyingctx.Ctx, so all downstream code can join it. - Commits if
next()returns nil andctx.Responseis a 2xx. - Rolls back if
next()returns an error, ifctx.Response.StatusCode >= 400, or if anything panics.
The same middleware can be registered on the DB step at Replace position if
you want to substitute the default DB step entirely.
Customising isolation
maniflex.TxOptions is an alias for sql.TxOptions:
server.Pipeline.Service.Register(
maniflex.WithTransaction(&maniflex.TxOptions{
Isolation: sql.LevelSerializable,
ReadOnly: false,
}),
maniflex.ForModel("Invoice"),
)
SQLite ignores most isolation levels. Write-locking does not depend on them:
sqlite.Open gives its write connections _txlock=immediate, so every
transaction takes the write lock at BEGIN rather than on its first write.
Joining the transaction from middleware
When ctx.Tx is set, every CRUD call made through ctx.GetModel(...),
ctx.RawQuery, ctx.RawExec, and the default DB step routes through it
automatically:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
// This Update participates in the same transaction as the DB step that follows.
if _, err := ctx.GetModel("Inventory").Update(itemID, map[string]any{
"reserved": true,
}); err != nil {
return err // triggers rollback
}
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
There is no separate “transactional” API — calling ctx.GetModel is enough.
Starting a transaction manually
When WithTransaction does not fit — for example, when only part of a request
should be transactional, or when the transaction must span an action endpoint
— begin one yourself:
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
ctx.Tx = tx
defer tx.Rollback() // no-op after Commit
// ... transactional work via ctx.GetModel / ctx.RawExec ...
if err := tx.Commit(); err != nil {
return err
}
ctx.Tx = nil // clear so post-commit code uses the bare adapter
tx.Rollback() after a successful commit is safe — it returns
sql.ErrTxDone, which the framework swallows. Always defer it.
Pessimistic locking
Inside a transaction, ctx.LockForUpdate(modelName, id) acquires a row-level
write lock and returns the current row:
row, err := ctx.LockForUpdate("StockBalance", stockID)
if err != nil {
return err
}
if row["quantity"].(int64) < 1 {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "no inventory remaining")
return nil
}
On Postgres this appends FOR UPDATE to the SELECT. On SQLite the lock is at
the transaction level — the row is protected because the transaction itself is
write-locked.
LockForUpdate returns an error if ctx.Tx is nil; the lock is meaningless
outside a transaction.
Joining from outside a ServerContext
Code without access to *ServerContext — for example, a job-queue helper, or an
outbox writer — can retrieve the active transaction from ctx.Ctx using
maniflex.TxFromContext:
func enqueue(ctx context.Context, job Job) error {
if tx := maniflex.TxFromContext(ctx); tx != nil {
return enqueueInTx(tx, job)
}
return enqueueDirect(job)
}
WithTransaction stores the active transaction on the context.Context for
exactly this purpose.
The transaction is reachable only while it is open. Once WithTransaction
commits or rolls back, both ctx.Tx and TxFromContext go back to nil, so a
middleware that resumes after its next() — an audit hook, an outbox enqueue —
takes the tx == nil branch and writes through the bare adapter, rather than
into a finished transaction.
Adapter scope
A transaction lives on a single DBAdapter. ctx.BeginTx opens the
transaction on the request’s model adapter (its ModelConfig.Adapter if
set, otherwise Config.DB). All operations on that ctx.Tx must target
models routed to the same adapter.
maniflex.Batch enforces this at runtime: calling b.Create("X", ...) for a
model that routes to a different adapter than the batch transaction
returns an error suggesting pkg/saga for cross-adapter coordination.
See Per-model adapter routing.
Nesting
WithTransaction is idempotent: if ctx.Tx is already set when it runs, it
simply calls next() without starting a new transaction. The outer transaction
remains the unit of commit.
SQLite does not support nested transactions; calling ctx.BeginTx while one
is already active returns an error.
Failure semantics
A transaction is rolled back when:
- the chain returns a non-nil error from any step;
ctx.Responseis set to a status>= 400(e.g. viactx.Abort);- a panic occurs (the framework’s panic recoverer ensures rollback).
WithTransaction is committed when:
- the chain completes without error and
ctx.Responseis a 2xx (or unset).
A commit failure is reported as 500 TX_COMMIT_ERROR; a begin failure as
500 TX_BEGIN_ERROR.
Next
- Writing Middleware — how to register
WithTransactionor write your own transactional middleware. - Error Handling — how a rollback surfaces to the client.
Error Handling
Every error returned by the pipeline is delivered to the client as the same JSON envelope. This page describes that envelope, the sentinel errors the framework recognises, and how to produce errors from middleware.
The error envelope
A failing request writes:
{
"error": {
"code": "VALIDATION_ERROR",
"message": "field \"email\" is required",
"details": { /* optional, per-error */ }
}
}
with the HTTP status code from the underlying failure. The code is a short
machine-readable identifier; message is a human-readable summary; details
is optional and may carry per-field errors or other structured context.
A successful request uses the {"data": ...} envelope instead; the two are
mutually exclusive.
Built-in error responses
The default pipeline produces the following errors without any user code:
| Status | Code | Source |
|---|---|---|
400 | INVALID_JSON | malformed JSON body |
400 | EMPTY_BODY | empty body on POST / PATCH |
400 | BODY_READ_ERROR | an I/O failure while reading the request body |
400 | INVALID_QUERY | unknown filter/sort field, malformed ?include, etc. |
400 | MULTIPART_ERROR | malformed multipart/form-data |
404 | NOT_FOUND | record does not exist (or is soft-deleted) |
409 | CONFLICT | unique or check constraint violated |
413 | BODY_TOO_LARGE | request body exceeded the 4 MB read limit |
422 | VALIDATION_ERROR | one or more mfx: tag rules failed |
500 | DB_ERROR | unclassified adapter error |
500 | TX_BEGIN_ERROR / TX_COMMIT_ERROR | transaction lifecycle failure |
499 | (no body) | the client disconnected before the response was written |
501 | NO_STORAGE | file endpoint hit with no FileStorage configured |
504 | TIMEOUT | Config.QueryTimeout (or another server-side deadline) expired |
A panic anywhere in the pipeline is caught and reported as 500 PANIC by the
framework’s recoverer — with one deliberate exception. A handler that panics with
http.ErrAbortHandler is abandoning its response on purpose (this is what
httputil.ReverseProxy does when an upstream dies mid-stream), so the recoverer
passes it on and net/http closes the connection silently. It is not logged as a
panic, and no error envelope is appended to whatever was already written.
499 is nginx’s non-standard “Client Closed Request” (maniflex.StatusClientClosedRequest).
Nothing is written to the client — the connection is already gone — but the status
is what your access log, metrics, and any After middleware reading the response
status will see, so a caller who hangs up is not counted as a server timeout.
Only a genuine server-side deadline produces 504 TIMEOUT. The disconnect itself
is logged at DEBUG, not as an error.
Aborting from middleware
The standard way to produce an error from middleware is ctx.Abort:
func ctx.Abort(statusCode int, code, message string)
It populates ctx.Response with an error envelope. The caller must then return
nil (or an error) without calling next():
if header == "" {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing token")
return nil
}
Subsequent steps are skipped; the Response step writes the prepared envelope.
Calling next() after Abort allows downstream steps to overwrite the
response — usually not what you want.
Returning structured details
For per-field errors and similar payloads, set ctx.Response directly:
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusUnprocessableEntity,
Error: &maniflex.APIError{
Code: "VALIDATION_ERROR",
Message: "one or more fields failed validation",
Details: []map[string]string{
{"field": "email", "message": "must be a valid email"},
{"field": "password", "message": "must be at least 8 characters"},
},
},
}
return nil
This is the shape used by the default Validate step.
Sentinel errors from the adapter
Adapter methods return errors that the DB step maps to HTTP responses. The two that user code most often interacts with are exported as sentinels.
maniflex.ErrNotFound
var ErrNotFound = errors.New("record not found")
Returned by FindByID, Update, and Delete when the row does not exist (or
is soft-deleted). Detect it with errors.Is:
row, err := ctx.GetModel("Invoice").Read(id)
if errors.Is(err, maniflex.ErrNotFound) {
ctx.Abort(http.StatusNotFound, "INVOICE_NOT_FOUND",
fmt.Sprintf("invoice %s does not exist", id))
return nil
}
The default DB step does this conversion automatically; you only need it when you are calling the adapter yourself from a Service middleware.
*maniflex.ErrConstraint
type ErrConstraint struct {
Kind ConstraintKind // unique, foreign_key, or not_null
Table string
Column string // may be empty when the driver does not expose it
Detail string // raw driver message
}
Returned by Create and Update on unique or check constraint violations.
Both SQLite and Postgres errors are normalised into this type, so middleware
need not inspect driver-specific codes.
row, err := ctx.GetModel("User").Create(data)
var ec *maniflex.ErrConstraint
if errors.As(err, &ec) {
ctx.Abort(http.StatusConflict, "DUPLICATE",
fmt.Sprintf("%s already exists", ec.Column))
return nil
}
Errors and transactions
When a request runs inside a transaction (see Transactions)
and any step returns an error or sets ctx.Response to status >= 400, the
transaction is rolled back before the response is written. The client sees the
error envelope; the database sees no change.
Logging errors
ctx.Logger() returns a *slog.Logger pre-seeded with request_id and
trace_id, so a single log line correlates with the request that produced it:
ctx.Logger().Error("payment provider rejected charge",
slog.String("provider", "stripe"),
slog.String("error_code", resp.Code),
)
ctx.Abort(http.StatusBadGateway, "PAYMENT_DECLINED", resp.Message)
return nil
Log first, then abort — the log line is what you’ll need when debugging.
Next
- ServerContext — the full set of fields available to error-producing middleware.
- Transactions — rollback semantics.
- Writing Middleware — where to attach error-producing logic.
Example 2: B2B SaaS API
This example builds a small multi-tenant SaaS backend. Compared to Example 1, it exercises every concept introduced in the “Defining Your API” and “The Request Pipeline” sections:
- Multiple related models with
BelongsToandHasManyrelations. - Soft delete on the records that matter for an audit trail.
- A bearer-token Auth middleware that populates
ctx.Auth. - A Service-step middleware that scopes every query to the caller’s tenant.
- A Service-step middleware that runs inside a transaction.
- Custom error envelopes with
ctx.Abort.
The goal is to show how the pieces compose; the auth and tenancy code is deliberately minimal so the example fits on one page.
Domain
A SaaS platform with three resources:
- Organization — the tenant. Every other record belongs to one.
- Member — a user belonging to an organization, with a role.
- Project — a unit of work owned by a member.
type Organization struct {
maniflex.BaseModel
maniflex.WithDeletedAt
Name string `json:"name" mfx:"required,filterable,sortable"`
Plan string `json:"plan" mfx:"required,enum:free|pro|enterprise,default:free"`
Members []Member `json:"members,omitempty"`
Projects []Project `json:"projects,omitempty"`
}
type Member struct {
maniflex.BaseModel
maniflex.WithDeletedAt
OrganizationID string `json:"organization_id" mfx:"required,filterable,immutable"`
Email string `json:"email" mfx:"required,filterable,unique"`
Role string `json:"role" mfx:"required,enum:owner|admin|editor|viewer,default:viewer,filterable"`
Projects []Project `json:"projects,omitempty"`
}
type Project struct {
maniflex.BaseModel
maniflex.WithDeletedAt
OrganizationID string `json:"organization_id" mfx:"required,filterable,immutable"`
OwnerID string `json:"owner_id" mfx:"required,filterable,relation:Owner"`
Owner Member `json:"owner,omitempty"`
Name string `json:"name" mfx:"required,filterable,sortable"`
Status string `json:"status" mfx:"required,enum:active|paused|archived,default:active,filterable,sortable"`
}
Owner is a companion field; the explicit relation:Owner tag is required
because the FK name (OwnerID) does not match the model name (Member).
OrganizationID follows the convention, so no companion is needed there.
Wiring
A single main.go registers the models, installs three middlewares, and
starts the server.
func main() {
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
server.MustRegister(Organization{}, Member{}, Project{})
db, err := sqlite.Open("./saas.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
registerMiddleware(server)
log.Fatal(server.Start())
}
Auth — populating ctx.Auth
A real deployment would verify a JWT; this example resolves a bearer token
against an in-memory map to keep the focus on ctx.Auth:
var tokens = map[string]maniflex.AuthInfo{
"alice-token": {UserID: "user-alice", TenantID: "org-acme", Roles: []string{"owner"}},
"bob-token": {UserID: "user-bob", TenantID: "org-acme", Roles: []string{"editor"}},
"carol-token": {UserID: "user-carol", TenantID: "org-globex", Roles: []string{"admin"}},
}
func bearerAuth(ctx *maniflex.ServerContext, next func() error) error {
header := ctx.Request.Header.Get("Authorization")
token := strings.TrimPrefix(header, "Bearer ")
info, ok := tokens[token]
if !ok {
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "invalid or missing token")
return nil
}
ctx.Auth = &info
return next()
}
Tenant scoping — filtering by ctx.Auth.TenantID
A B2B API must never leak data across tenants. A Service-step middleware inspects every list and read, and rejects writes that would assign records to a foreign organization:
func enforceTenant(ctx *maniflex.ServerContext, next func() error) error {
tenant := ctx.Auth.TenantID
switch ctx.Operation {
case maniflex.OpList:
// Inject a filter so users only see their organization's rows.
ctx.Query.Filters = append(ctx.Query.Filters, maniflex.Filter{
Field: "organization_id", Op: "eq", Value: tenant,
})
case maniflex.OpCreate, maniflex.OpUpdate:
if v, ok := ctx.Field("organization_id"); ok && v != tenant {
ctx.Abort(http.StatusForbidden, "TENANT_MISMATCH",
"organization_id does not match the authenticated tenant")
return nil
}
ctx.SetField("organization_id", tenant)
}
return next()
}
The middleware applies to the two tenanted models — Member and Project —
and to every operation. Organization itself is not scoped, because the auth
middleware already binds each token to exactly one organization.
Transactions — creating a project atomically
A new project should fail entirely if the member quota check fails. A Service
middleware wraps the operation in a transaction and uses
ctx.LockForUpdate to read the organization with a write lock:
func enforceProjectQuota(ctx *maniflex.ServerContext, next func() error) error {
org, err := ctx.LockForUpdate("Organization", ctx.Auth.TenantID)
if err != nil {
return err
}
rows, err := ctx.RawQuery(
`SELECT COUNT(*) AS n FROM projects WHERE organization_id = ? AND deleted_at IS NULL`,
ctx.Auth.TenantID,
)
if err != nil {
return err
}
count := rows[0]["n"].(int64)
limit := planLimit(org["plan"].(string))
if count >= limit {
ctx.Abort(http.StatusPaymentRequired, "PROJECT_LIMIT",
fmt.Sprintf("plan %q allows %d projects; upgrade to add more", org["plan"], limit))
return nil
}
return next()
}
func planLimit(plan string) int64 {
switch plan {
case "enterprise":
return 1000
case "pro":
return 25
default:
return 3
}
}
Registering the middleware
All three middlewares are registered in one place:
func registerMiddleware(s *maniflex.Server) {
// Auth on every write — reads are public within the tenant once they
// pass enforceTenant below; tighten or relax to taste.
s.Pipeline.Auth.Register(bearerAuth,
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete, maniflex.OpList, maniflex.OpRead),
)
// Tenant scoping for the two tenanted models.
s.Pipeline.Service.Register(enforceTenant,
maniflex.ForModel("Member", "Project"),
)
// The DB step is wrapped in a transaction for project creation, and the
// quota check runs inside it before next() reaches DB.
s.Pipeline.Service.Register(maniflex.WithTransaction(nil),
maniflex.ForModel("Project"), maniflex.ForOperation(maniflex.OpCreate),
)
s.Pipeline.Service.Register(enforceProjectQuota,
maniflex.ForModel("Project"), maniflex.ForOperation(maniflex.OpCreate),
)
}
Order matters: WithTransaction is registered before enforceProjectQuota, so
the transaction is open by the time the quota check calls ctx.LockForUpdate.
A request, end to end
# Alice (org-acme, owner) creates a project.
curl -X POST localhost:8080/api/projects \
-H 'Authorization: Bearer alice-token' \
-H 'Content-Type: application/json' \
-d '{"name":"Atlas","owner_id":"user-alice"}'
What happens:
- Auth —
bearerAuthresolvesalice-tokenand setsctx.Auth. - Deserialize — JSON body parsed into
ctx.ParsedBody. - Validate —
mfx:tag rules pass;organization_idis missing but the next step injects it. - Service:
enforceTenantwritesorganization_id = "org-acme"into the body.WithTransactionbegins a transaction.enforceProjectQuotalocks the organization row, counts existing projects, and either aborts with402 PROJECT_LIMITor proceeds.
- DB —
Createruns through the transaction;ctx.DBResultholds the inserted row. - Response — the envelope is written; the transaction commits.
Carol’s carol-token belongs to org-globex, so the same payload from her is
rejected before it reaches the DB step:
curl 'localhost:8080/api/projects?filter=organization_id:eq:org-acme' \
-H 'Authorization: Bearer carol-token'
# → list filtered to org-globex only; the requested filter is ignored
What this example showed
- Relations declared with both the convention (
OrganizationID) and the explicitrelation:Ownerform. WithDeletedAton every audited model.ctx.Authpopulated by an Auth middleware, then read by Service middleware to scope queries.ctx.Query.Filtersmodified to enforce a tenant invariant.maniflex.WithTransactionplusctx.LockForUpdatefor a check-and-act write.- Custom error codes (
TENANT_MISMATCH,PROJECT_LIMIT) emitted withctx.Abort.
Where to go next
The next section covers the ready-made middleware that ships with maniflex — the production-quality versions of the auth and validation helpers sketched here.
- Middleware Catalogue — JWT auth, password hashing, unique validation, audit logging, and more.
- Querying — the full filter, sort, and
includegrammar used in this example.
Middleware Catalogue
The catalogue is a set of ready-made middleware packages that cover the common
needs of every production API — authentication, validation, password hashing,
audit logging, caching, CORS, and so on. Each one is an ordinary
maniflex.MiddlewareFunc you register on the appropriate pipeline step, with the
same scoping options as any other middleware.
The packages live under maniflex/middleware/. Most are part of the root module,
so they need no extra require to use. Only the two with heavy third-party
dependencies are broken out into their own Go modules — middleware/db/redis
(the Redis rate-limit/cache backend) and middleware/service/bcrypt (password
hashing) — so a project pulls those dependencies in only when it uses them:
| Package | Step | What it ships |
|---|---|---|
middleware/auth | Auth | JWT, API key, role gates, public-read helpers |
middleware/body | Deserialize / Validate | body size limits, unknown-field stripping, type coercion |
middleware/idempotency | Deserialize | idempotency-key replay for safely retried POSTs |
middleware/validate | Validate | uniqueness, regex, cross-field rules, numeric precision, date ranges, conditional required |
middleware/workflow | Validate | state-machine transitions with role-gated guards |
middleware/service | Service / DB-After | password hashing, slugify, derived fields |
middleware/db | DB | tenancy, forced filters, rate limiting, audit log, cache invalidation |
middleware/response | Response | CORS, caching, transforms, redaction, envelopes, metrics |
middleware/openapi | OpenAPI.Generate | security schemes, servers, titles, custom extensions |
How to use the catalogue
Import the package you need and register the returned middleware on the matching pipeline step:
import (
"github.com/xaleel/maniflex/middleware/auth"
"github.com/xaleel/maniflex/middleware/service"
)
server.Pipeline.Auth.Register(
auth.JWTAuth("my-signing-secret", auth.JWTOptions{Issuer: "my-app"}),
)
server.Pipeline.Service.Register(
service.HashField("password", svcbcrypt.Hasher()), // svcbcrypt = middleware/service/bcrypt
maniflex.ForModel("User"),
)
Each middleware factory returns a maniflex.MiddlewareFunc, so the standard
options — ForModel, ForOperation, AtPosition, WithName — apply
verbatim.
Composition
Catalogue middleware is designed to be composable. The expected stack for a typical REST API is roughly:
- Auth —
JWTAuthorAPIKeyAuthpopulatesctx.Auth;RequireRolegates sensitive operations. - Body —
MaxBodySizeandStripUnknownFieldsshape input early. - Validate — built-in tag rules plus
UniqueFieldand friends. - Service —
HashField,SetField,SlugifyField, then any custom business logic, then themaniflex/eventshelpers (events.Emit/events.Webhook/events.SendEmail) on the After side. - DB —
TenancyorForceFilterenforces row-level scoping;AuditLogandInvalidaterun After. - Response —
CORSHeaders,Cache,RedactField, thenLogging/Metricson the After side.
Mix and match freely; nothing in the catalogue is required.
Writing your own
The catalogue is just an applied form of Writing Middleware.
If a built-in does not match your needs, write your own — the contract is the
same func(ctx *maniflex.ServerContext, next func() error) error signature.
Auth Middleware
The maniflex/middleware/auth package supplies authentication and authorisation
middleware. Each function returns a maniflex.MiddlewareFunc that either populates
ctx.Auth on success or aborts with 401/403 on failure. Most register on the
Auth step; the exceptions are called out where they occur — Enforce
(attribute policies) runs on the DB step so the record is available, and
ReadAudit runs on the Response step after a 2xx.
JWTAuth
Verifies a bearer JWT and populates ctx.Auth from its claims.
import "github.com/xaleel/maniflex/middleware/auth"
server.Pipeline.Auth.Register(
auth.JWTAuth("my-signing-secret", auth.JWTOptions{
Issuer: "my-app",
Audience: "api",
TenantClaim: "org_id", // copied into AuthInfo.TenantID
ScopesClaim: "scope", // copied into AuthInfo.Scopes
}),
)
Supports HMAC (HS256/384/512) when the secret is a string and asymmetric
algorithms (RS256/384/512) when JWTOptions.PublicKey is set — useful with
external identity providers (Auth0, Okta, Cognito, etc.). AuthMethod on
ctx.Auth is set to "jwt".
Where the token is read from
By default the token comes from Authorization, which must carry the
Bearer scheme — a bare token there is malformed, not merely unadorned. Point
JWTOptions.Header at another header to read it from there instead:
auth.JWTAuth(secret, auth.JWTOptions{Header: "X-Auth-Token"})
A custom header accepts the token with or without the Bearer prefix, so a
client that sends it out of habit is not punished for it. The scheme name is
matched case-insensitively on both, per RFC 7235 — bearer <token> is valid.
Tokens must carry an exp claim: one with no expiry is rejected
(401 TOKEN_MISSING_EXPIRY), since it would otherwise be valid forever. Set
JWTOptions.AllowNoExpiry to accept non-expiring tokens from issuers that
deliberately mint them. On the HMAC path the signing secret must be non-empty (an
empty secret panics at startup) and should be at least 32 bytes — a shorter
secret is allowed but logs a warning.
JWKSAuth
Verifies asymmetric JWTs against a rotating JWK Set published by an identity
provider (e.g. an issuer’s /.well-known/jwks.json), instead of pinning a single
static key. Signing keys are fetched, cached, and selected by the token header’s
kid; an unknown kid triggers a rate-limited refetch, so a rotated key is
picked up without a redeploy. RSA (RS256/384/512) and EC (ES256/384/512) are
supported.
server.Pipeline.Auth.Register(auth.JWKSAuth(
"https://issuer.example.com/.well-known/jwks.json",
auth.JWTOptions{Issuer: "https://issuer.example.com", Audience: "api"},
))
All JWTOptions (Issuer, Audience, claim mappings, ClockSkew) apply exactly
as with JWTAuth — reach for the static-key JWTAuth only when the key is fixed
or for offline tests. See
Auth & Security Hardening for the
production checklist.
APIKeyAuth
Validates a static API key from a request header. Each entry maps one key to
the AuthInfo it grants.
server.Pipeline.Auth.Register(auth.APIKeyAuth("X-API-Key",
auth.APIKeyEntry{Key: "svc-abc", Auth: maniflex.AuthInfo{
UserID: "svc-1", Roles: []string{"admin"},
}},
auth.APIKeyEntry{Key: "svc-xyz", Auth: maniflex.AuthInfo{
UserID: "svc-2", Roles: []string{"reader"},
}},
))
AuthMethod on ctx.Auth is set to "api_key". Combine with JWTAuth on
the same step to accept either credential — the first match wins.
RequireRole
Rejects the request unless ctx.Auth.Roles contains the named role. Typically
registered with ForModel / ForOperation to scope where the check applies.
server.Pipeline.Auth.Register(
auth.RequireRole("admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpDelete),
)
Both failure cases return 403 FORBIDDEN: an anonymous request (ctx.Auth == nil) and an authenticated request lacking the role are treated alike. This
differs from RequireOwner, which answers 401 to an anonymous caller.
RequireOwner
Enforces that the authenticated user owns the record being read or written. On
create it stamps ownerField = ctx.Auth.UserID automatically; on read, update,
and delete it fetches the target and compares its ownerField to the caller —
answering 404 (not 403, so the endpoint never reveals that a record it doesn’t
own exists). ownerField may be given as the JSON or the DB column name. Callers
holding any role in adminRoles bypass the check.
server.Pipeline.Auth.Register(auth.RequireOwner("user_id", "admin"))
RequireOwner scopes single-resource operations only — a collection GET
still returns every row. Constrain list reads with db.ForceFilter or
db.Tenancy on the DB step.
Enforce — attribute-based policies
RequireRole and RequireOwner answer who is calling; Enforce answers may
this principal touch this record, evaluating a Policy against the affected
row’s fields. A Policy is a plain function:
type Policy func(ctx *maniflex.ServerContext, resource map[string]any) (allow bool, err error)
Return (false, nil) for a 403 FORBIDDEN; a non-nil error becomes a 500.
Register on the DB step (not Auth), so the row is available:
sameTenant := func(ctx *maniflex.ServerContext, r map[string]any) (bool, error) {
return r["tenant_id"] == ctx.Auth.TenantID, nil
}
server.Pipeline.DB.Register(auth.Enforce(sameTenant), maniflex.ForModel("Patient"))
Which record the policy sees depends on the operation:
| Operation | Record checked | When |
|---|---|---|
OpCreate | the proposed body (ctx.ParsedBody) | before the insert |
OpUpdate / OpDelete | the current stored record | fetched before the write |
OpRead | the fetched record | after the DB step |
OpList | each row in turn | after the DB step; denied rows are dropped |
Compose policies with AllOf, AnyOf, and Not:
auth.Enforce(auth.AllOf(sameTenant, auth.AnyOf(isOwner, auth.Not(isArchived))))
For lists the policy runs per row after the query, so pagination totals
reflect the pre-filter count. When the rule can be expressed as a WHERE clause,
prefer db.ForceFilter — it scopes in SQL, keeps totals accurate, and
never fetches rows the caller can’t see.
AllowPublicRead
A passthrough that exempts read operations from upstream auth requirements.
Register it before JWTAuth / APIKeyAuth to let unauthenticated callers
hit GET routes while keeping writes locked down.
server.Pipeline.Auth.Register(auth.AllowPublicRead())
server.Pipeline.Auth.Register(auth.JWTAuth("..."),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
BlockOperation
Refuses the listed operations outright, regardless of identity. Useful for making a model effectively read-only at the HTTP layer.
server.Pipeline.Auth.Register(
auth.BlockOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.ForModel("AuditLog"),
)
The model’s routes remain mounted but always return 405 METHOD_NOT_ALLOWED.
Token revocation and logout
A JWT is valid until its exp and nothing the server does can take it back.
That is fine until a user logs out, changes their password, or has their account
compromised — at which point “valid until it expires” is exactly wrong. Set
JWTOptions.Revoker to give the server its say back, at the cost of a lookup
per request.
rev := auth.NewMemoryRevoker()
server.Pipeline.Auth.Register(auth.JWTAuth(secret, auth.JWTOptions{Revoker: rev}))
server.Action(auth.Logout(rev, "")) // POST /logout
server.Action(auth.LogoutAll(rev, "", 24*time.Hour)) // POST /logout-all
Logout revokes the token the caller authenticated with — logout on this device.
LogoutAll revokes every token belonging to the caller, including sessions whose
jti the server has never seen; it is the one to call after a password change.
Both respond 204, and both must be mounted behind the same auth middleware as
everything else (they read ctx.Auth).
The two granularities
| Call | Effect |
|---|---|
RevokeToken(ctx, jti, exp) | blocks one token; the entry may be dropped once exp passes, since the token is refused for being expired anyway |
RevokeUser(ctx, userID, cutoff, retainUntil) | blocks every token for the user issued before cutoff; keep the record until retainUntil, which must be past the exp of the longest-lived token you mint |
The per-user cutoff is what makes “log out everywhere” possible: the outstanding
jti values are unknown and usually unknowable, so a per-token blocklist alone
cannot express it. Logging in again works immediately — the cutoff kills tokens
issued before it, not the account.
What changes when it is on
- A
jtibecomes mandatory. A token without one cannot be revoked, so it is refused with401 TOKEN_NOT_REVOCABLErather than handed a permanent exemption from the blocklist. Mint a uniquejtiper token. - An
iatbecomes required for users who have a cutoff. A token that cannot be placed relative to the cutoff is refused. Tokens withoutiatkeep working for every user who has never calledLogoutAll. - A store outage refuses requests — see below.
Error codes:
| Code | Status | Meaning |
|---|---|---|
TOKEN_REVOKED | 401 | explicitly revoked; the client should discard it and log in again |
TOKEN_NOT_REVOCABLE | 401 | no jti, so it cannot be revoked — a minting bug, not a client error |
REVOCATION_UNAVAILABLE | 503 | the blocklist could not be consulted |
Failing closed
Every Revoker method returns an error, and the middleware refuses the request
when a lookup fails rather than reading “I could not check” as “not revoked”. A
blocklist that fails open silently un-revokes every logged-out token for the
duration of an outage — precisely when it matters most. This is why Revoker is
its own interface rather than a reuse of maniflex.CacheStore, whose Get
reports a miss and an outage identically.
The refusal is 503, not 401: the credential is fine, the server is not, and
answering 401 would push every healthy client into a re-login storm during an
incident.
Backends
NewMemoryRevoker() keeps the blocklist in-process. Its limitation is
structural: a second replica does not see a logout performed by the first, and a
restart loses every entry — which un-revokes every still-unexpired token. It
suits single-replica deployments, development, and tests.
Behind more than one replica, use the Redis backend:
import authredis "github.com/xaleel/maniflex/middleware/auth/redis"
rev := authredis.NewRevoker(redisClient, "myapp:revoked")
Or implement auth.Revoker yourself over any store — four methods, and the
interface is in terms of context.Context and time.Time only.
VerifyToken hook
JWTOptions.VerifyToken is the final say on a token that has passed every other
check — signature, registered claims, and revocation. Return an error to refuse
the request with 401 INVALID_TOKEN carrying that message.
auth.JWTAuth(secret, auth.JWTOptions{
VerifyToken: func(ctx *maniflex.ServerContext, claims map[string]any, info *maniflex.AuthInfo) error {
if claims["tier"] != "paid" {
return errors.New("subscription required")
}
info.Roles = append(info.Roles, "subscriber") // may enrich the principal
return nil
},
})
Use it for issuer-specific rules the framework does not model: a required custom
claim, a tenant allowlist, a per-user check against your own store. For
revocation specifically, prefer Revoker — it is the same hook point with the
blocklist and its failure semantics already written.
CSRF
Protects unsafe HTTP methods (POST/PUT/PATCH/DELETE) against cross-site
request forgery. Two strategies, selected by CSRFOptions.Mode:
CSRFDoubleSubmit(default) — issues a random token in a non-HttpOnly cookie on safe requests and requires the client to echo it in a header on unsafe ones.CSRFSignedToken— derives the expected token asHMAC(SessionID, Secret); stateless, no cookie issued. Register it after the JWT step soctx.Auth.SessionIDis populated (e.g. from the token’sjticlaim).
server.Pipeline.Auth.Register(auth.CSRF(auth.CSRFOptions{
AllowedOrigins: []string{"https://admin.example.com", "*.example.com"},
Secure: true,
}))
Bearer-authenticated requests are exempt by default — a bearer token read
from JS is not an ambient credential, so it isn’t CSRF-vulnerable. Set
EnforceBearer: true only if your bearer flow also rides browser-managed cookies.
The optional AllowedOrigins allowlist (exact origins or *.host wildcards) is
checked first on unsafe methods. Failures abort with a 403 carrying one of
CSRF_ORIGIN_REJECTED, CSRF_TOKEN_MISSING, CSRF_COOKIE_MISSING,
CSRF_NO_SESSION, or CSRF_TOKEN_MISMATCH.
From a login Action, hand the token to the SPA with auth.IssueCSRFCookie(w, opts) (double-submit) or auth.SignedCSRFToken(sessionID, secret) (signed mode).
ReadAudit
Writes a structured audit record after every successful read or list — the
read-side counterpart to db.AuditLog, for data where who looked is itself the
compliance control (clinical, financial). Implement a ReadAuditSink and register
ReadAudit at the After position on the Response step, so it fires only on
a 2xx:
server.Pipeline.Response.Register(
auth.ReadAudit(mySink),
maniflex.ForModel("Patient", "LabResult"),
maniflex.ForOperation(maniflex.OpRead, maniflex.OpList),
maniflex.AtPosition(maniflex.After),
)
Each ReadAuditRecord carries the actor, roles, tenant, session, request/trace
IDs, client IP, and either the accessed RecordID (read) or the RecordCount
(list). Writes are fire-and-forget — a goroutine with a 5 s timeout, so a slow
sink never delays the response — and sink errors are discarded, so back a lossless
requirement with a durable queue.
Scoping patterns
The middleware in this package combines with ForModel / ForOperation to
build per-route policy without writing custom Auth code:
// Public reads, JWT writes, admin-only deletes
server.Pipeline.Auth.Register(auth.AllowPublicRead())
server.Pipeline.Auth.Register(auth.JWTAuth("..."),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete))
server.Pipeline.Auth.Register(auth.RequireRole("admin"),
maniflex.ForOperation(maniflex.OpDelete))
Body Middleware
The maniflex/middleware/body package shapes the request body during the
Deserialize and Validate steps — before the framework’s tag rules run.
MaxBodySize
Overrides the default 4 MB body limit for the current request. Register on the Deserialize step, scoped to the model that needs the larger limit:
import "github.com/xaleel/maniflex/middleware/body"
server.Pipeline.Deserialize.Register(
body.MaxBodySize(16 << 20), // 16 MB
maniflex.ForModel("Article"),
)
Requests over the limit are aborted with 413 BODY_TOO_LARGE before the JSON
parser runs — as is any request over the 4 MB default when this middleware is
not registered. An oversized body is never truncated to fit.
StripUnknownFields
Removes keys from ctx.ParsedBody that do not correspond to a model field.
Register on the Validate step (or Deserialize After-position) so the cleanup
happens before tag validation and the DB step:
server.Pipeline.Validate.Register(body.StripUnknownFields())
The default behaviour is to accept and silently ignore unknown fields. Use this middleware to enforce a stricter contract when desired.
CoerceTypes
Coerces string values in ctx.ParsedBody into the Go type declared on the
model — "42" → 42 (int), "3.14" → 3.14 (float64), "true" → true
(bool). Helps when the client sends form-encoded or query-string-shaped payloads.
Only string→int/float64/bool is performed; other types are left as-is.
server.Pipeline.Validate.Register(body.CoerceTypes())
Coercion happens before the framework’s min / max / enum checks, so
numeric ranges and enums work against the coerced values.
Validate Middleware
The maniflex/middleware/validate package supplies validators that go beyond what
mfx: tags can express. Each one runs on the Validate step alongside the
built-in tag enforcement, and most abort with 422 VALIDATION_ERROR on rejection.
The exceptions:
RestrictField/FieldRoleanswer403 FIELD_FORBIDDEN, because they gate on who is asking, not on whether the value is valid.RequireLocaleanswers422 MISSING_LOCALE— still a 422, but with its own code so a missing translation is distinguishable from a plain validation miss.UniqueFieldanswers409 CONFLICT, not a422— a duplicate is a conflict with existing state rather than malformed input, and this is the same answer the database’s own constraint gives for the identical error. If the underlying count query itself fails it answers500 UNIQUE_CHECK_FAILEDrather than letting a duplicate slip through.
UniqueField
Rejects a create or update whose value collides with an existing row.
import "github.com/xaleel/maniflex/middleware/validate"
server.Pipeline.Validate.Register(
validate.UniqueField(sqlDB, maniflex.Postgres, "email"),
maniflex.ForModel("User"),
)
The middleware runs a count query against the underlying database before the DB
step. It answers the same 409 CONFLICT the mfx:"unique" constraint produces —
identical status, code, message and details — so a client handles one response
whichever mechanism caught the duplicate. What it buys is when: the collision is
reported before the write is attempted, and named by JSON field.
It is not a substitute for mfx:"unique". Count-then-write is not atomic, so two
concurrent requests can both pass the count; the database constraint remains the
actual guarantee.
The driver argument selects the placeholder dialect (maniflex.Postgres →
$N, maniflex.SQLite → ?) and must match the driver used to open the
adapter. The third argument is the JSON field name; it is resolved to the
underlying DB column via ctx.Model.FieldByJSONName.
RegexField
Validates that a string field matches a regular expression:
server.Pipeline.Validate.Register(
validate.RegexField("phone", `^\+?[0-9]{7,15}$`),
maniflex.ForModel("Contact"),
)
A non-matching value aborts with 422 and the field name in details.
ForbiddenValues
Rejects writes that contain any of the listed values for a field. Use it for defence-in-depth on enum-like fields where the mfx enum tag would still allow a privileged value:
server.Pipeline.Validate.Register(
validate.ForbiddenValues("role", "superadmin", "root"),
maniflex.ForModel("User"),
)
RequireAtLeastOne
Ensures at least one of the named fields is present in the request body. Most
useful on OpUpdate, where every field is otherwise optional:
server.Pipeline.Validate.Register(
validate.RequireAtLeastOne("name", "email"),
maniflex.ForOperation(maniflex.OpUpdate),
)
RequireLocale
Ensures a localised (LocaleString) field carries a non-empty value for each of
the required locale keys. Use it for translatable fields where certain languages
are mandatory:
server.Pipeline.Validate.Register(
validate.RequireLocale("name", "en", "ar"),
maniflex.ForModel("Department"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
The field value may arrive as a JSON object (map[string]any) or a Go-native
map[string]string. A locale key that is absent, null, or empty fails the
check. When any required locale is missing the request is aborted with
422 MISSING_LOCALE, and details lists every offending locale keyed by field.
If the field is absent from the body, null, or not a locale map at all, the
rule passes silently — pair it with required when presence itself is mandatory.
NumericPrecision
Enforces decimal precision and scale on a numeric field. The check is a
string-parse, so it works regardless of how the column is stored
(INTEGER, NUMERIC(p,s), TEXT, custom SQLTyper):
server.Pipeline.Validate.Register(
validate.NumericPrecision("amount", 19, 4), // up to 19 total digits, max 4 after the point
maniflex.ForModel("Invoice"),
)
precision— maximum total significant digits (integer + fractional). Pass0to disable.scale— maximum digits after the decimal point. Pass0to disable.
Sign (+/-), leading zeros, and trailing fractional zeros do not count
toward either limit. Scientific notation (1e3) is rejected because its
implied precision is ambiguous — supply financial values in plain form.
Absent and null values are skipped; pair with required when presence
matters.
CrossFieldValidate
A general-purpose escape hatch for rules that span multiple fields:
server.Pipeline.Validate.Register(
validate.CrossFieldValidate(func(body map[string]any) error {
if body["status"] == "scheduled" && body["scheduled_at"] == nil {
return fmt.Errorf("scheduled_at is required when status is scheduled")
}
return nil
}),
maniflex.ForModel("Post"),
)
The returned error becomes the message of a 422 VALIDATION_ERROR response.
DateRange
Ensures an end field is not before a start field. Accepts RFC3339 timestamps
("2026-05-01T08:00:00Z") and YYYY-MM-DD date strings. If either field is
absent, null, or unparseable, the rule passes silently — pair with required
or another rule when presence matters.
server.Pipeline.Validate.Register(
validate.DateRange("start_date", "end_date"),
maniflex.ForModel("Booking"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
Equal dates are accepted (start == end). A 422 VALIDATION_ERROR is returned
with the end field named in details when the end precedes the start.
RequireWhen
Makes a field required only when other fields satisfy all listed conditions.
Each condition is a "field:op:value" string; op is one of eq, ne, gt,
gte, lt, lte. Multiple conditions are ANDed. Invalid syntax panics at
startup so misconfiguration is caught before the first request.
// Require rejection_reason whenever status is "rejected"
server.Pipeline.Validate.Register(
validate.RequireWhen("rejection_reason", "status:eq:rejected"),
maniflex.ForModel("Claim"),
)
// Require shipping_address only for physical orders with priority >= 3
server.Pipeline.Validate.Register(
validate.RequireWhen("shipping_address", "order_type:eq:physical", "priority:gte:3"),
maniflex.ForModel("Order"),
)
When the conditions are met but the target field is absent, null, or empty
string, the request is rejected with 422 VALIDATION_ERROR. When the
conditions are not all met, the rule passes regardless of the target field’s
value — it does not prevent the field from being supplied.
Numeric comparisons (gt, gte, lt, lte) coerce both the body value and
the condition value to float64. Non-numeric body values cause the condition to
evaluate as false (the target field stays optional).
FieldRole / RestrictField
Gate a field on who is writing it. These are the write-side twin of
response.RedactField — the same predicate shape, on
the opposite step.
// Only a superuser may write status; everyone else may write the rest of the row
server.Pipeline.Validate.Register(
validate.FieldRole("subscription_expires_at", "superuser"),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
RestrictField takes a predicate instead of a role list, for gates roles cannot
express (ownership, tenant, plan tier):
server.Pipeline.Validate.Register(
validate.RestrictField("document_quota_bytes",
func(ctx *maniflex.ServerContext) bool {
return ctx.HasRole("superuser") || isBillingAdmin(ctx)
}),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
FieldRole(field, roles...) is exactly RestrictField with a ctx.HasRole
predicate (OR-semantics). With no roles passed it rejects every write of the
field, so an accidentally empty list fails closed — matching auth.RequireRole.
Declare the field so a typo can’t fail open
Misspell the field name and the gate is inert: it watches for a body key nothing
sends, the real field keeps its name, and nothing gates it. Add
maniflex.RequiresField and that becomes a startup failure:
server.Pipeline.Validate.Register(
validate.FieldRole("subscription_expires_at", "superuser"),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
maniflex.RequiresField("subscription_expires_at"), // ← checked at startup
)
The declared name is checked against the model, so writing it twice catches a misspelling rather than duplicating one. Do this on every gate that protects a field you care about.
Without it, a mismatch is only a warning on the first request that reaches the model — no help for an endpoint nobody exercises, and indistinguishable from a gate deliberately registered across models where only some carry the field. See Startup Validation.
Why this isn’t readonly
The mfx: write controls are static: readonly, immutable, hidden apply to
every caller identically. They cover “no client ever writes this”. They cannot
express “only a superuser may set subscription_expires_at, while the owner
writes the rest of their own row” — which, without this, costs a separate
endpoint per privileged field.
Refused, not stripped
A caller without permission gets 403 FIELD_FORBIDDEN naming the field.
This deliberately differs from readonly, which silently drops the field:
readonly | FieldRole | |
|---|---|---|
| meaning | nobody writes this, ever | someone writes this — not you |
| client sending it | confused about the schema | making a privilege error |
| on violation | field dropped, 200 | whole write refused, 403 |
Answering 200 to a write that did not happen, echoing the old value back, is
indistinguishable from success. The write is refused whole — a mixed
PATCH {"title": …, "status": …} from a non-holder changes neither field.
Details
- Only a field present in the body is gated. A PATCH that does not mention it
passes untouched. An explicit
nullis a write, and is gated. - Create and update both, when you scope it with
ForOperation. fieldis the JSON name.- Scope it with
maniflex.ForModel. Registered without one it applies to every model, and a gate naming a field the model does not have can never fire. It warns once per model in that case, since a typo would otherwise leave the real field ungated in silence.
Workflow Middleware
maniflex/middleware/workflow enforces a state-machine on a model’s status
field. Declare the permitted transitions once; the middleware rejects writes
that would move a record between states the workflow does not allow, or that
fail a guard (e.g. role check).
import "github.com/xaleel/maniflex/middleware/workflow"
sm := workflow.New("status",
workflow.Allow("draft", "submitted"),
workflow.Allow("submitted", "approved", workflow.RequireRole("manager")),
workflow.Allow("submitted", "rejected", workflow.RequireRole("manager")),
workflow.Allow("approved", "paid", workflow.RequireRole("finance")),
workflow.AllowAny(workflow.RequireRole("admin")), // admin escape hatch
workflow.AllowInitial("draft", "submitted"), // legal seed states on Create
)
server.Pipeline.Validate.Register(
sm.Middleware(),
maniflex.ForModel("Invoice"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
How it runs
The middleware lives on the Validate step. (A machine that declares
OnTransition hooks registers on the DB step instead — see
Transition hooks.)
- On
OpCreate— ifAllowInitialis declared, the value of the chosen field must be in the set; otherwise any initial value passes. No guards apply on Create (the Create itself is its own authorisation surface). - On
OpUpdate— if the body does not include the status field, the middleware is a no-op. Otherwise it reads the current record viactx.GetModel(modelName).Read(id)(so reads participate inctx.Txwhen active), extractsfrom, compares to the body’sto, and:- same-state writes (
from == to) pass silently; - the first matching rule wins; its guards run in order;
- the first guard error rejects the transition with
422 INVALID_TRANSITION.
- same-state writes (
A PATCH that triggers the read also costs one round-trip. Stash the loaded
record on ctx.Set if you need it again later in the request.
Rules
| Option | Effect |
|---|---|
Allow(from, to, guards...) | permit from → to; both sides may be "*" for “any” |
AllowAny(guards...) | shorthand for Allow("*", "*", guards...) |
AllowInitial(states...) | restrict Create to the listed initial states |
OnTransition(from, to, fn) | run fn inside the write’s transaction when from → to is taken |
Rule matching is a linear scan in declaration order — write narrow rules before broad ones if you want them to take precedence.
A machine with no Allow rules permits nothing — every transition is
rejected. If you want hooks without restricting transitions, say so explicitly
with AllowAny().
Guards
type Guard interface {
Check(ctx *maniflex.ServerContext, from, to string) error
}
RequireRole(roles ...string)— pass if the caller holds any one of the listed roles. OR-semantics; passing zero roles always rejects (a defensive choice against accidentally unguarded “require” rules).GuardFunc— adapt anyfunc(ctx, from, to) errorto the interface.
A non-nil guard error becomes the response message:
{
"error": {
"code": "INVALID_TRANSITION",
"message": "role [manager] required for transition \"submitted\" → \"approved\"",
"details": [{"field": "status", "from": "submitted", "to": "approved"}]
},
"status": 422
}
Transition hooks (OnTransition)
OnTransition(from, to, fn) attaches a side effect to a transition. The hook
runs inside the write’s transaction, so it commits with the transition or
not at all:
sm := workflow.New("status",
workflow.Allow("pending", "confirmed", workflow.RequireRole("store_owner")),
workflow.Allow("confirmed", "pending", workflow.RequireRole("store_owner")),
workflow.AllowAny(workflow.RequireRole("admin")),
workflow.OnTransition("pending", "confirmed", applyStoreCredit),
workflow.OnTransition("confirmed", "pending", reverseStoreCredit),
workflow.OnTransition("*", "delivered", emitReviewRequested),
)
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpUpdate),
)
server.Pipeline.DB.Register(
sm.Hooks(),
maniflex.ForModel("Order"),
maniflex.ForOperation(maniflex.OpUpdate),
)
type Hook func(ctx *maniflex.ServerContext, from, to string) error
Hooks() replaces Middleware() — it does not supplement it
A machine with hooks registers sm.Hooks() on the DB step, and that is the
only registration it needs: Hooks() enforces the same rules and guards
Middleware() does, and more strictly. Declaring a hook makes Middleware()
panic rather than let it be registered on Validate where the hooks could never
fire.
| Machine | Register | Transaction |
|---|---|---|
| guards only | sm.Middleware() on Validate | not required |
any OnTransition | sm.Hooks() on DB | required — WithTransaction on Service |
The split exists because the two cannot run at the same point. A guard must run
before anything is written; a hook must run inside the write’s transaction — and
that transaction does not exist until WithTransaction opens it on the Service
step, after Validate.
That is also why Hooks() re-reads from rather than trusting the Validate
step’s verdict. The Validate-step read takes no lock, so two concurrent PATCHes
can both observe from="pending", both pass, and both apply the store
credit. Hooks() re-reads from through ctx.LockForUpdate inside the
transaction: the loser waits, then sees confirmed and is either a no-op
(same-state) or re-checked as the transition it has really become. Because the
re-read can name a different rule than the one that passed on the stale value,
the matching rule’s guards re-run too.
With no active transaction, Hooks() refuses the request with
500 WORKFLOW_NO_TX rather than take no lock and silently reintroduce the race
— the same fail-loudly stance as mfx:"lock_scope".
Semantics
- Fire-all-matching, in declaration order — unlike
Allow, which is first-match-wins.OnTransition("pending", "confirmed", …)andOnTransition("*", "delivered", …)are independent side effects, and a rule ordering that silenced one of them would be a bug rather than a policy. OpUpdateonly. A Create seeds an initial state (AllowInitialgoverns it); it is not a transition. Use a Service-step middleware for create-time side effects.- Same-state writes fire nothing — a no-op is not a transition.
- A returned error rolls the whole request back, transition included, and
answers
500 WORKFLOW_HOOK_ERROR. To choose your own status, callctx.Abort(…)and returnnil; the write still rolls back and your response is what the client sees. - The hook runs after the write lands, so a read through
ctx.Txsees the new state.
Status field type
Values are compared as strings via fmt.Sprintf("%v", v), matching
validate.ForbiddenValues. This covers string, int, and custom enum
types.
Service Middleware
The maniflex/middleware/service package supplies business-logic helpers for the
Service step — field transforms, derived values, and owner-scoping.
Side-effect helpers (events, webhooks, email) live in the separate
maniflex/events package, not here — see
Side effects below.
Field transforms
HashField
Replaces a plaintext field with its hash before the DB step, using a Hasher
you supply. The bcrypt hasher lives in the satellite package
maniflex/middleware/service/bcrypt (kept separate so the core has no bcrypt
dependency):
import (
"github.com/xaleel/maniflex/middleware/service"
svcbcrypt "github.com/xaleel/maniflex/middleware/service/bcrypt"
)
server.Pipeline.Service.Register(
service.HashField("password", svcbcrypt.Hasher()),
maniflex.ForModel("User"),
)
svcbcrypt.Hasher() takes an optional cost (svcbcrypt.Hasher(12)); the default
is suitable for production. A Hasher is just func(plaintext string) (string, error), so you can supply argon2 or any other implementation.
SlugifyField
Derives a slug field from a source field on create:
server.Pipeline.Service.Register(
service.SlugifyField("title", "slug"),
maniflex.ForModel("Post"), maniflex.ForOperation(maniflex.OpCreate),
)
Punctuation is stripped, spaces become hyphens, and the result is lowercased.
SetField
Sets a field on every create or update based on context — typically pulling
identity from ctx.Auth:
server.Pipeline.Service.Register(
service.SetField("user_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.UserID
}),
maniflex.ForOperation(maniflex.OpCreate),
)
StripField
Removes a field from the request body (and the typed record) before the DB step. Useful for input-only confirmation fields that should never reach the database:
server.Pipeline.Service.Register(service.StripField("password_confirm"))
TimestampWhen
Sets a timestamp column when another field transitions to a specific value —
for example, recording published_at the moment status becomes
"published":
server.Pipeline.Service.Register(
service.TimestampWhen("published_at", "status", "published"),
maniflex.ForModel("Post"),
)
Timestamp
Unconditionally sets a timestamp column to the current time on every write it
runs for — use ForOperation to scope it (e.g. a last_seen_at touched on
update):
server.Pipeline.Service.Register(
service.Timestamp("last_seen_at"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpUpdate),
)
CopyField
Copies one field’s value into another before the DB step — for denormalising a value the client shouldn’t set directly:
server.Pipeline.Service.Register(
service.CopyField("email", "billing_email"),
maniflex.ForModel("Account"), maniflex.ForOperation(maniflex.OpCreate),
)
Authorisation
OwnerScope
Forces a user-id field to the authenticated caller on create. It is exactly
SetField("user_id", ctx.Auth.UserID) — an unconditional overwrite, so any value
the client sent for that field is replaced (not rejected):
server.Pipeline.Service.Register(
service.OwnerScope("user_id"),
maniflex.ForOperation(maniflex.OpCreate),
)
Side effects
Side effects — events, webhooks, email — are not in this package. They live
in maniflex/events, and only events.Emit
is pipeline middleware; events.Webhook and events.SendEmail are event-bus
subscribers, not middleware, so they are wired with bus.Subscribe, not
Pipeline.Register.
events.Emit publishes a domain event on the DB step at maniflex.After,
after the write succeeds:
import "github.com/xaleel/maniflex/events"
server.Pipeline.DB.Register(
events.Emit(bus),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
Webhooks and email then react to those events as subscribers:
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: events.Webhook(events.WebhookConfig{URL: "https://hooks.example.com/orders", Secret: "whsec_…"}),
})
bus is any events.Bus — events/inproc for a single binary, or the Kafka,
NATS, RabbitMQ, and Redis adapters. See
Events & Background Jobs for the full API,
including the transactional outbox that makes events.Emit commit atomically
with the write (without it, a rolled-back transaction leaves an already-sent
side effect — outbound emails do not unsend themselves).
DB Middleware
The maniflex/middleware/db package wraps the DB step with row-level
scoping, request budgeting, post-write hooks, and result caching.
Row-level scoping
ForceFilter
Injects a filter on every list, read, update, and delete, regardless of what the client requested. Used to enforce invariants the client cannot override:
import "github.com/xaleel/maniflex/middleware/db"
server.Pipeline.DB.Register(
db.ForceFilter("org_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.Claims["org_id"]
}),
maniflex.ProvidesScope(),
)
On a list or a read the filter goes into the query. On an update or a delete
there is nowhere to put it — the adapter’s Update and Delete are keyed by id
alone — so the DB step reads the record back through the filter first and
answers 404 if it does not match, indistinguishably from a record that is
genuinely absent. The check and the write share one transaction, so the row
cannot leave scope in between.
On a create or an update the scope value is also stamped onto the row. An
equality on a plain column says two things at once — “only rows where field =
value” and “a row you create has field = value” — and both are honoured.
Without the stamp a create stored whatever the client sent in the scope column,
so the row landed outside its own author’s scope: invisible to them on their next
read, and, since that column is the client’s to send, plantable straight into
someone else’s scope. The stamp is skipped on update when the column is
immutable, which cannot change anyway.
That read is the only cost, and only a request carrying a forced filter pays it:
a write with nothing scoped goes straight to the adapter as before. A client’s
own ?filter= never constrains a write — only filters the server imposed do.
If you build a maniflex.FilterExpr by hand and it expresses who may touch the
row rather than which rows were asked for, set Forced: true on it; that is
what carries it onto updates and deletes. ForceFilter and Tenancy set it for
you.
Forced: true also lets a scope be imposed before the DB step. The
Deserialize step rebuilds ctx.Query from the request, which discards a plain
filter an earlier step appended — but a Forced filter survives that rebuild.
So an Auth-step middleware may append a Forced filter to ctx.Query.Filters
and have it reach the query; a non-forced one set that early is dropped. The DB
step remains the idiomatic home for scoping (ForceFilter, Tenancy), and is
required when the scope must also cover writes end-to-end within one transaction.
ProvidesScope — running the scope before Validate
Registering a scoper on the DB step puts it after Validate. The scope reaches
the query, but nothing earlier in the chain can ask what it is. Declare
maniflex.ProvidesScope() to hoist it to run right after Deserialize instead:
server.Pipeline.DB.Register(
db.Tenancy("org_id", tenantFromAuth),
maniflex.ProvidesScope(), // ← hoists it ahead of Validate
)
Add it to every ForceFilter, ForceFilterVia and Tenancy registration unless
you have a reason not to. It changes when the middleware runs, never what it
does: the same filter is appended, the same rows are scoped, and an operation
that skips the DB step still skips it. Hoisted middleware runs exactly once — it
is removed from its own step, not duplicated.
It is opt-in, and forgetting it fails quietly. The framework cannot tell a
scope provider from any other middleware, so it will not infer this, and there is
no startup error for a registration that omits it. What you get instead is a
scope that works for the query and is invisible to everything before the write.
The sharpest symptom is on a scoped Singleton, whose row id is not known until
its scope is: validate.UniqueField then excludes the record under edit by a
placeholder that matches nothing, the row collides with itself, and every PATCH
is refused with 422 "<field> is already taken" (audit 13.12).
Anything that needs the record a request addresses should ask
ctx.ResolveResourceID() rather than reading ctx.ResourceID directly — see
the context reference.
Use the maniflex.Op* constants for Operator. It’s a bare string type, and a
hand-built filter is never parsed — only filters arriving over HTTP are — so
Operator: "equals" compiles and boots, and until v0.2.3 it produced a scope that
silently matched every row. An operator no adapter implements is now refused with
an error naming it, and the adapters render it as a false predicate rather than a
true one, so a filter reaching one by some other path matches nothing instead of
everything. maniflex.FilterOperator.Valid() reports whether an operator is one
the query builder implements.
ForceFilterVia
ForceFilter maps a field to a value, which needs the model to carry that field.
A child table often doesn’t: a DamagedItem has an item_id and nothing else,
and whether it’s yours is a fact about its Item. ForceFilterVia scopes such a
model through the column its parent carries:
// A DamagedItem is the caller's if its Item is
server.Pipeline.DB.Register(
db.ForceFilterVia("item", "owner_id", func(ctx *maniflex.ServerContext) any {
return ctx.Auth.Claims["owner_id"]
}),
maniflex.ForModel("DamagedItem"),
maniflex.ProvidesScope(),
)
The first argument names the relation, in the same vocabulary a nested ?filter=
uses (?filter=author.status:neq:banned → author); the second names a column on
the parent, by JSON or DB name. The relation must be a BelongsTo — the join needs
a foreign key on this row pointing at the one row that owns it, which is what a
HasMany doesn’t have.
The alternative is to denormalise owner_id onto every child — a schema change
plus a standing obligation to keep it in step, on exactly the tables whose scoping
is easiest to get wrong — or to hand-write the predicate, which is what
declarative scoping exists to replace.
Reads join the parent and apply the predicate. Updates and deletes read the row
back through it and answer 404 on a miss, exactly as they do for ForceFilter.
Creates are scoped too, and they have to be. The foreign key is what the whole
scope hangs from, and it’s the one part of it the client supplies. On a create
there’s no row to read back, so without a check nothing looks at item_id at all
and POST {"item_id": "<another tenant's>"} lands a row under their parent — where
they can see it and you can’t. On an update the row read back is the one the old
item_id points at, so a PATCH that rewrites the key passes a check of where the
row used to be and then moves it somewhere else. So the parent a write names is
read through the scope’s own predicate first, and a miss is the same 404 a scoped
read of that parent gives. A create that names no parent at all is refused with
422: the join would find nothing, so the row would be invisible to whoever
created it.
That parent read is the only added cost, and only a scope that runs through a parent pays it.
There’s no TenancyVia, because Tenancy is ForceFilter plus stamping the
tenant column onto writes — and the whole premise here is a model with no such
column to stamp. Checking the parent is what takes its place.
Actions:
ForceFilterViaregisters on the DB step, so like every other DB-step middleware it doesn’t run for a customAction— see Scoping Actions. There’s noForceFilterViaAction: anActionScope’s filters apply to whatever model the handler touches, while aViascope is resolved against one specific model’s relations, and an action runs on a synthetic model with none. For a single-model action, build the nestedFilterExprby hand (IsNested,RelationKey,RelationTable,RelationFK,NestedField,Forced) and pass it toctx.SetActionScope.ctx.ViaFilter— the resolverForceFilterViais built on — reports this rather than guessing if you call it from an action.
Tenancy
A specialised ForceFilter for the common multi-tenant case. Reads the tenant
id from ctx.Auth and pins every query to it:
server.Pipeline.DB.Register(
db.Tenancy("org_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.Claims["org_id"].(string)
}),
maniflex.ProvidesScope(),
)
Tenancy also rewrites the org_id field on creates and updates so a tenant
cannot place rows into another tenant’s bucket. That rewrite is why the scoping
on updates matters twice over: it stamps the caller’s tenant onto whatever row
the update reaches, so an update that reached the wrong row would not merely
overwrite it — it would move it into the caller’s tenant and leave the owner
unable to see it at all.
That stamp is written with ctx.SetField, which marks the field as set by the
server. A tenant column is normally readonly — a client must not choose its own
tenant — and readonly means not from a client, so the Validate step keeps a
value the server stamped and strips only one that arrived in the request body.
Without that distinction a hoisted Tenancy would have its stamp discarded and
write the row with an empty tenant: invisible to the tenant that created it, and
visible to every caller whose scope is also empty.
Scoping Actions: TenancyAction / ForceFilterAction
Tenancy and ForceFilter register on the DB step, and a custom Action
does not run it — its pipeline is Auth → middleware → handler → Response. Their
only output is a filter on ctx.Query, which nothing in that chain reads, so
registering them on the DB step does nothing at all for an Action, silently.
Use the Action variants instead, in the action’s own Middleware list, where
they run after Auth and can read ctx.Auth:
server.Action(maniflex.ActionConfig{
Method: "POST", Path: "/orders/{id}/refund",
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
db.TenancyAction("org_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.Claims["org_id"].(string)
}),
},
Handler: refund,
})
Inside that handler every DB path either applies the scope or refuses to run:
| Path | Under a scope |
|---|---|
ctx.GetModel(name) — List/Read/Create/Update/Delete | scoped |
maniflex.List/Read/Create/Update/Delete[T] | scoped |
ctx.Aggregate | scoped (AND-ed into WHERE) |
ctx.LockForUpdate | scoped |
ctx.BeginTx — and the Tx it returns | scoped |
ctx.RawQuery, ctx.RawExec | refuses |
ctx.Search, ctx.RecursiveQuery | refuses |
Reads see only matching rows. A create is stamped with the scope’s values,
overwriting whatever the caller supplied — a row created outside the scope would
be invisible to the caller that created it. An update or delete of a record
outside the scope returns ErrNotFound, the same answer the scoped read gives.
Transactions work normally. Tx mirrors DBAdapter — its FindByID/FindMany
take a *QueryParams, its Update/Delete are keyed by id — so the Tx that
ctx.BeginTx returns is scoped the same way the accessor is, and that matters
because ctx.Tx is a public field: anything downstream that picks the
transaction up is scoped too. maniflex.WithTransaction and maniflex.Batch
both call ctx.BeginTx, so both work on a scoped action:
tx, err := ctx.BeginTx(ctx.Ctx, nil) // scoped
defer tx.Rollback()
// tx.FindByID / tx.Update / tx.Delete all honour the scope
return tx.Commit()
The refusals are the point. Raw SQL is opaque to the framework: a SELECT
string cannot be scoped without rewriting it. Scoping the convenient paths and
letting those leak in silence would put the guarantee in this page and not in the
code — worse than no guarantee, because it would be trusted. Refusing means an
action either honours the scope or fails at the first request that exercises it.
When a path genuinely must step outside the scope — an audit query across tenants, a migration — say so:
rows, err := ctx.Unscoped().RawQuery("SELECT ... FROM orders")
ctx.Unscoped() exposes RawQuery, RawExec, BeginTx, GetModel and
Search with no scope applied. It is a distinct call rather than a flag so the
bypass shows up at the call site, in the diff, and in a grep — which only works
while it stays rare, so it is deliberately not needed for ordinary work.
The scope applies only to Actions. A generated CRUD route gets its scoping from
the DB step, where ctx.RawQuery from an After-DB middleware is a normal thing
to do and is not refused.
Action variants: RateLimitAction / AuditLogAction
RateLimit and AuditLog register on the DB step, which a custom Action
never runs, so they never fire for one. Their action-list counterparts run in the
action’s own Middleware list — after the global Auth step, so ctx.Auth is
available:
server.Action(maniflex.ActionConfig{
Method: "POST", Path: "/reports/{id}/evidence",
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
db.RateLimitAction(db.RateLimitConfig{RequestsPerMinute: 10}),
db.AuditLogAction(mySink),
},
Handler: uploadEvidence,
})
RateLimitAction takes the same RateLimitConfig as RateLimit but keys on the
caller (its KeyFunc, else the authenticated user id, else the remote IP) plus
the request method and path, since an action has no model/operation to key on. It
rejects over-limit requests with 429 RATE_LIMITED and a Retry-After header.
AuditLogAction writes one audit record after a successful action (a >= 400
response is skipped), sourcing the actor and tenant from ctx.Auth, the resource
id from the {id} URL param, the result from ctx.Response.Data, and
Operation = OpAction. Writes are fire-and-forget, so a sink error never fails
the request; change diffing (WithChanges) does not apply to actions.
Request budgeting
Paginate
Caps the maximum ?limit= accepted on list responses. Per-model overrides are
common for tables whose rows are expensive to render:
server.Pipeline.DB.Register(db.Paginate(50), maniflex.ForModel("AuditLog"))
RateLimit
A token-bucket rate limiter scoped by IP or by authenticated user:
server.Pipeline.DB.Register(
db.RateLimit(db.RateLimitConfig{
RequestsPerMinute: 10,
KeyFunc: func(ctx *maniflex.ServerContext) string {
if ctx.Auth != nil {
return ctx.Auth.UserID
}
return ctx.Request.RemoteAddr
},
}),
maniflex.ForModel("PasswordReset"),
)
Rejected requests receive 429 RATE_LIMITED. The counter is in-process by
default; for a limit shared across replicas set RateLimitConfig.Backend (see
middleware/db/redis
for the Redis implementation).
When the key falls back to the client IP (ctx.Request.RemoteAddr), that address
is the direct TCP peer unless Config.TrustProxyHeaders is enabled — set it
(only behind a trusted proxy) so per-IP limits see the real client instead of the
load balancer. See Security.
RateLimitField
Rate-limits by the value of a request-body field rather than by caller identity — cap password-reset or OTP requests per email address, for instance, so one address cannot be flooded regardless of which client sends the requests:
server.Pipeline.DB.Register(
db.RateLimitField("email", 3, time.Hour),
maniflex.ForModel("PasswordReset"),
maniflex.ForOperation(maniflex.OpCreate),
)
The arguments are the field name (its parsed-body value becomes the bucket key),
the request cap, and the window. If the field is absent from the parsed body the
limit is not applied, so a legitimate request is never blocked by a missing
field. Rejected requests receive 429 RATE_LIMITED. Pass
db.WithRateLimitBackend to share the window across replicas, or
db.WithRateLimitErrorMessage to override the 429 message.
Post-write hooks
These run at maniflex.After position so they only fire when the database write
succeeded.
AuditLog
Writes one audit record per mutating operation to a configured sink:
server.Pipeline.DB.Register(
db.AuditLog(mySink),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
mySink is anything implementing the audit interface — a logger, a database
table, an external SIEM. The record carries the operation, model name, actor
(from ctx.Auth), and a JSON diff of the affected row.
Invalidate
Invalidates cache keys when a row changes. The key list is computed per request:
server.Pipeline.DB.Register(
db.Invalidate(redisCache, func(ctx *maniflex.ServerContext) []string {
return []string{
"posts:list",
fmt.Sprintf("post:%s", ctx.ResourceID),
}
}),
maniflex.ForModel("Post"),
maniflex.AtPosition(maniflex.After),
)
CacheQuery
Memoises read results (OpRead, OpList) in a CacheStore — the read-side
complement to Invalidate. On a cache hit it sets ctx.DBResult and the
adapter read is skipped; the Response step renders the cached result. On a miss
it runs the read and stores the result for TTL. Pair it with Invalidate on
writes to evict stale entries.
cache := maniflex.NewMemoryCache() // or a Redis-backed CacheStore
server.Pipeline.DB.Register(
db.CacheQuery(cache, db.CacheConfig{
TTL: 5 * time.Minute,
KeyFunc: func(ctx *maniflex.ServerContext) string {
// Only cache the common, high-traffic query shapes. Requests that
// filter on `name` or carry a ?q= search are typically long-tail,
// ad-hoc lookups — caching them floods the store with low-value
// entries that are rarely read back, so skip them by returning "".
if q := ctx.Query; q != nil {
if q.Search != "" {
return ""
}
for _, f := range q.Filters {
if f.Field == "name" {
return ""
}
}
}
return "products:list:" + ctx.Request.URL.RawQuery
},
}),
maniflex.ForModel("Product"),
maniflex.ForOperation(maniflex.OpList, maniflex.OpRead),
)
server.Pipeline.DB.Register(
db.Invalidate(cache, func(*maniflex.ServerContext) []string {
return []string{"products:list:..."}
}),
maniflex.ForModel("Product"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
KeyFunc must capture every input that changes the result (model, tenant,
filters, sort, pagination, includes); returning "" skips the cache for that
request. The value stored is ctx.DBResult, so a distributed CacheStore must
round-trip a *maniflex.ListResult for lists — a store that decodes list
entries into a bare map is treated as a miss rather than panicking. Avoid
caching mfx:"encrypted" models, since the decrypted result would live in the
cache.
Ordering
Row-level scopers (ForceFilter, ForceFilterVia, Tenancy) must run Before
the default DB step — the default position — so the filter is in place when the
SELECT or UPDATE runs. Post-write hooks must run After, so they observe the result.
The package’s defaults follow this; only change them if you know why.
Before is the latest a scoper can run, not the earliest it should. Pair it with
maniflex.ProvidesScope()
to hoist the scoper ahead of the Validate step, so the steps between Deserialize
and the write see the same scope the query does.
Response Middleware
The maniflex/middleware/response package shapes the outgoing response —
headers, body transforms, redactions, and observability hooks — on the
Response step.
Cross-cutting headers
CORSHeaders
Adds CORS headers to every response. At least one origin is required — pass
explicit origins (recommended) or "*" to allow any origin. Calling it with no
origins panics at startup, so a permissive wildcard is never applied by accident.
import "github.com/xaleel/maniflex/middleware/response"
// Explicit origins (recommended)
server.Pipeline.Response.Register(response.CORSHeaders("https://app.example.com"))
// Public API: opt in to any origin explicitly
server.Pipeline.Response.Register(response.CORSHeaders("*"))
For credentials or custom allowed headers/methods/max-age, use
CORSHeadersWithConfig. AllowCredentials cannot be combined with a "*"
origin (browsers reject that combination) and panics if you try:
server.Pipeline.Response.Register(response.CORSHeadersWithConfig(response.CORSConfig{
AllowOrigins: []string{"https://app.example.com"},
AllowCredentials: true,
}))
AddHeader
Sets one static header on every response:
server.Pipeline.Response.Register(
response.AddHeader("Strict-Transport-Security", "max-age=63072000"),
)
Caching
Cache
Sets Cache-Control: public, max-age=N on successful reads. Register at
maniflex.After so the framework’s own headers do not override yours:
server.Pipeline.Response.Register(
response.Cache(300), // 5 minutes
maniflex.ForOperation(maniflex.OpRead, maniflex.OpList),
maniflex.AtPosition(maniflex.After),
)
Body transforms
TransformField
Rewrites a single field value before serialisation. Common use: rebasing a stored relative path onto a CDN host.
server.Pipeline.Response.Register(
response.TransformField("avatar_url", func(v any) any {
return cdnBase + v.(string)
}),
)
RedactField
Hides a field from the response conditionally. The predicate decides per
request, often based on ctx.Auth:
server.Pipeline.Response.Register(
response.RedactField("phone", func(ctx *maniflex.ServerContext) bool {
return !ctx.HasRole("support")
}),
)
RedactField is the right tool for view-time access control on individual
columns. For all-or-nothing exclusion across an entire model, the hidden or
writeonly field tag is simpler.
For the write side — “only a superuser may set this field” — use
validate.FieldRole / validate.RestrictField,
which takes the same predicate on the Validate step. Note the two differ on
purpose when the predicate fails: a redacted read returns the record without the
field, while a refused write returns 403 rather than quietly dropping it.
It covers exports too: GET /:model/export masks the same field for the
same callers, and drops it from the CSV/XLSX header rather than emitting an
empty column.
Writing your own masking middleware
A Response-step middleware normally masks by editing ctx.Response after
next() returns. That is not enough on its own, because an export has no
ctx.Response — it streams its bytes during next() — so a middleware that
only edits one masks the JSON and leaves the export in full.
Declare the field instead, before calling next():
func maskSalary(ctx *maniflex.ServerContext, next func() error) error {
if !ctx.HasRole("admin") {
ctx.RedactResponseField("salary") // before next(), so the export sees it
}
return next()
}
The declaration is honoured by every read path — list, single read, create and
update echoes, and both export formats. response.RedactField does this for
you.
Before v0.2.5 a masking middleware applied to JSON responses only. An app that hid a column from non-admins served it in full at
/:model/export.
Envelope
Replaces the default {"data": ...} envelope with one of your own:
server.Pipeline.Response.Register(
response.Envelope(func(ctx *maniflex.ServerContext, data any, meta *maniflex.ResponseMeta) any {
return map[string]any{
"result": data,
"paging": meta,
"trace_id": ctx.TraceID,
}
}),
)
Useful when integrating with a frontend or API gateway that expects a different shape. Error responses are unaffected; only success responses are re-enveloped.
Observability
Logging
Writes a structured access log line per request at maniflex.After:
server.Pipeline.Response.Register(
response.Logging(slog.Default()),
maniflex.AtPosition(maniflex.After),
)
The line carries request ID, trace ID, method, path, status, duration, and the authenticated user when set.
Metrics
Records per-request metrics — count, latency, status class — into a configured collector:
server.Pipeline.Response.Register(
response.Metrics(myCollector),
maniflex.AtPosition(maniflex.After),
)
A reference Prometheus collector is provided; any sink with the same interface works.
OpenAPI Middleware
The maniflex/middleware/openapi package customises the auto-generated OpenAPI
3.1 specification served at GET /openapi.json. Every middleware here is
registered on the Pipeline.OpenAPI.Generate step at maniflex.After
position, so it sees the framework’s generated spec and can mutate it before
the Response step serialises it.
SetTitle, SetDescription
Override the default title and description, which are derived from
Config.ServiceName:
import "github.com/xaleel/maniflex/middleware/openapi"
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetTitle("Orders API"),
maniflex.After,
)
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetDescription("# Orders API\nProduction endpoints for the orders service."),
maniflex.After,
)
SetVersion
Overrides the spec’s info.version, which is otherwise the framework default.
Use it to surface your service’s own release version in the published spec:
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetVersion("2.1.0"),
maniflex.After,
)
AddServer
Declares a server URL in servers[]. Repeat for multiple environments:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddServer("https://api.example.com", "Production"), maniflex.After)
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddServer("https://staging.example.com", "Staging"), maniflex.After)
Without AddServer the spec carries no servers array, leaving clients to
resolve URLs against the host that served the spec.
AddSecurityScheme
Adds a security scheme to components.securitySchemes and applies it to
every operation generated by the framework:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddSecurityScheme("bearerAuth", maniflex.OASSecurityScheme{
Type: "http",
Scheme: "bearer",
BearerFormat: "JWT",
}),
maniflex.After,
)
Pair with auth.JWTAuth on the runtime side. For API keys,
use Type: "apiKey" and set In and Name.
AddTag
Appends a Tag Object to the spec. Tags render as collapsible groups in Swagger
UI; maniflex already emits one tag per model, so use AddTag for cross-cutting
or documentation-only groupings. A tag whose name already exists is not
duplicated.
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddTag("Authentication", "Endpoints related to user authentication"),
maniflex.After,
)
InjectRequestExample
Attaches an example request body to a specific operation so Swagger UI’s “Try it out” pre-populates with realistic values. The operation is named by path and lower-case method:
server.Pipeline.OpenAPI.Generate.Register(
openapi.InjectRequestExample(
openapi.OperationTarget{Path: "/posts", Method: "post"},
"Example post", map[string]any{
"title": "Hello World",
"body": "My first post.",
"status": "draft",
}),
maniflex.After,
)
The example is only applied when the target operation and its request body exist in the generated spec; an unmatched target is a no-op.
AddExtension
A general-purpose escape hatch — receives the full *maniflex.OpenAPISpec and lets
you mutate any part of it:
server.Pipeline.OpenAPI.Generate.Register(
openapi.AddExtension(func(spec *maniflex.OpenAPISpec) {
spec.Info.Description += "\n\nContact the API team at [email protected]."
}),
maniflex.After,
)
Use sparingly — anything you can express through the typed helpers is easier to read.
Securing the spec itself
To gate access to /openapi.json, register an Auth middleware on
Pipeline.OpenAPI.Auth — it has the same shape as the model-route Auth step:
server.Pipeline.OpenAPI.Auth.Register(auth.RequireRole("internal"))
This is the standard pattern for keeping internal APIs out of the public view while leaving the public surface documented.
Querying
Every generated list and read endpoint accepts the same query parameters —
page, limit, filter, sort, include, and select. This page
documents their grammar and the fields that opt in to each.
page and limit
Standard offset pagination.
?page=2&limit=20
| Parameter | Default | Maximum |
|---|---|---|
page | 1 | unbounded |
limit | 20 | 200 |
Values above the maximum are clamped silently. Negative or non-numeric values
are rejected with 400 INVALID_QUERY.
The response carries meta.total, meta.page, meta.limit, and meta.pages
— see Response Envelope.
cursor (keyset pagination)
Offset pagination skips or duplicates rows when the dataset changes between page fetches — delete a row on page 1 and page 2 silently jumps a record. Keyset (cursor) pagination walks the data by a stable ordering key instead, so the window never shifts. Opt a model in by naming a sortable, effectively monotonic cursor column:
type Event struct {
maniflex.BaseModel `mfx:"cursor_field:created_at"` // created_at is sortable on BaseModel
Name string `json:"name" db:"name"`
}
Equivalently, set ModelConfig.CursorField: "created_at" at registration, or
put mfx:"...,cursor_field:<name>" on any of the model’s own fields.
The cursor column must be mfx:"sortable" and not nullable — a pointer field
(*time.Time) is rejected at registration. Keyset pagination needs a total order,
and NULL has no place in one: the boundary comparison is never true for it, and
Postgres and SQLite don’t even agree where NULLs sort. A nullable cursor column
would drop or repeat rows across pages, so the model fails to register rather than
paginating wrongly.
The presence of ?cursor= switches the request into keyset mode (it supersedes
?page). Send an empty value for the first page, then the meta.next_cursor
from each response to fetch the next:
GET /events?cursor=&limit=20 → first page
GET /events?cursor=<next_cursor>&limit=20 → following page
The walk is ordered by (cursor_field, id) — id is the implicit tiebreaker so
the order is total even when the cursor column ties. The default direction is
ascending; sort on the cursor field to reverse it:
GET /events?cursor=&sort=created_at:desc
Any ?sort= on a different field is rejected with 400 in cursor mode, since
the keyset order is fixed to the cursor column.
Cursor responses carry a different meta shape — no total/page/pages
(the count is skipped, which is the point on large tables):
{ "data": [ ... ], "meta": { "limit": 20, "next_cursor": "eyJ2Ijoi...", "has_more": true } }
has_more is false and next_cursor is omitted on the last page. The token is
opaque — treat it as a string and pass it back verbatim.
filter
Each filter is a colon-separated triple — field, operator, value:
?filter=status:eq:published
?filter=views:gt:100
?filter=created_at:gte:2025-01-01
Multiple filters combine with AND:
?filter=status:eq:published&filter=views:gt:100
Filters reference a field by its json name. Only fields tagged
mfx:"filterable" may be used; unknown or non-filterable references abort the
request with 400 INVALID_QUERY.
Operators
| Operator | Effect | Value |
|---|---|---|
eq | field = value | one value |
neq | field ≠ value | one value |
gt, gte, lt, lte | numeric and date comparisons | one value |
like | SQL LIKE, case-sensitive | one pattern — % and _ are wildcards |
ilike | SQL ILIKE, case-insensitive | one pattern — % and _ are wildcards |
contains | field contains the value, case-insensitive | one literal value |
starts_with | field starts with the value, case-insensitive | one literal value |
ends_with | field ends with the value, case-insensitive | one literal value |
in | field IN (…) | at least one comma-separated value |
not_in | field NOT IN (…) | at least one comma-separated value |
between | field ≥ lo AND ≤ hi (inclusive) | exactly two comma-separated values lo,hi |
is_null | field IS NULL | no value |
not_null | field IS NOT NULL | no value |
?filter=tag:in:go,rust,zig
?filter=amount:between:100,500
?filter=created_at:between:2025-01-01,2025-03-31
?filter=archived_at:is_null
?filter=title:ilike:%intro%
?filter=title:contains:intro
Patterns vs. literals
like and ilike take a pattern: % matches any run of characters and _
matches exactly one. That is what makes ?filter=title:ilike:%intro% work — but
it also means a value the user typed is interpreted rather than matched.
?filter=label:like:50% finds 500 units and 50 off as readily as the 50%
you were looking for, and there is no portable way to escape a % in a pattern
(SQLite has no escape character by default; Postgres has a backslash).
contains, starts_with, and ends_with take a literal: % and _ in the
value are escaped for you and match themselves, so ?filter=label:contains:50%
finds exactly the labels containing 50%. They are case-insensitive on both
backends. Use them for anything a user typed — a search box, a filename, an SKU —
and reach for like/ilike only when the caller genuinely is writing a pattern.
Note that % must still be percent-encoded in a URL (%25), as in any query
string:
?filter=label:contains:50%25 → matches the literal "50%"
?filter=label:like:50%25 → matches "50%", "500 units", "50 off", …
Filtering on related fields
When a relation is declared on the model, you can filter by a field on the related table using dot notation:
?filter=user.role:eq:admin
?filter=posts.status:eq:published
The related field must itself be filterable. The framework joins the related
table for the query; no separate ?include= is required to filter on it (but
you still need ?include= to return the related row).
q (full-text search)
?q= runs a native full-text search over every field tagged mfx:"searchable"
and orders the results by match relevance:
?q=hello world
?q=postgres&filter=tag:eq:db
This is distinct from filter: full-text search uses the database’s own
ranking, stemming, and tokenisation rather than literal comparison, so ?q=run
also matches running, and the densest match ranks first. The backend’s native
machinery does the work — a tsvector column and GIN index on PostgreSQL, an
FTS5 index on SQLite — both provisioned automatically during migration.
- Only models with at least one
mfx:"searchable"field accept?q=; on any other model it aborts with400 INVALID_QUERY. Searchable fields must be text columns. ?q=combines with?filter=(ANDed) and the usual?page=/?limit=offset pagination. It cannot be combined with?cursor=, since keyset order and relevance order are mutually exclusive.- An empty value (
?q=) is ignored — the list is returned unfiltered. - On PostgreSQL the text-search configuration defaults to
english; override it per model withModelConfig.SearchLanguage.
type Article struct {
maniflex.BaseModel
Title string `json:"title" db:"title" mfx:"required,searchable"`
Body string `json:"body" db:"body" mfx:"searchable"`
}
// GET /articles?q=keyset+pagination → relevance-ranked matches
sort
Each sort is field:direction:
?sort=created_at:desc
?sort=title:asc
Multiple sorts compose left-to-right (primary, secondary, …):
?sort=status:asc&sort=created_at:desc
Only fields tagged mfx:"sortable" may be used. BaseModel’s created_at and
updated_at are sortable by default.
Sorting on a relation field
Use relation.field to sort by a column on a BelongsTo parent. The server
adds the LEFT JOIN automatically — no filter or include on that relation is
required:
?sort=user.name:asc
?sort=vendor.name:desc&filter=status:eq:open
The related field must be tagged mfx:"sortable" on the parent model. Only
BelongsTo relations are supported; relation.field on a HasMany or
ManyToMany returns 400, as does an unknown relation or a non-sortable
related field.
include
Loads related records inline. The value is a comma-separated list of relation keys:
?include=user
?include=user,comments
Each key becomes a nested object (for BelongsTo) or array (for HasMany and
ManyToMany) on the returned row. See Relations for how
relation keys are derived.
Includes are populated by separate queries after the main query — they do not multiply rows or affect pagination.
One level of nesting
A key may carry a single dot to load a relation of the related model:
?include=author.company
?include=author.company,comments
?include=author.company implies author — the parent is what the child hangs
off — so you do not need to name both.
Two segments is the limit. ?include=a.b.c is refused with 400 INVALID_QUERY. Each level is one more batched query, and the tree comes from the
client, so leaving it uncapped would let a caller choose how much work a request
costs.
Every segment must name a real relation; a typo is a 400, not a silently
missing key. Nested rows are scoped, decrypted and field-filtered exactly as the
first level is — a forced filter (db.Tenancy, db.ForceFilter) applies at every
level, and hidden / writeonly fields on the nested model stay out.
Go callers: nesting applies to the JSON response. The typed relation structs (
post.Author) are still populated one level deep, sopost.Author.Companyis not filled in by a typed read. Use the JSON path, or a secondmaniflex.Read.
select
Request a subset of fields instead of the full row. Useful for wide tables (payroll, product catalogues with 40+ attributes) where most columns are irrelevant to the caller.
?select=id,name,department
?select=id,amount,status
The value is a comma-separated list of JSON field names. Unknown names
abort the request with 400 INVALID_QUERY. Fields tagged mfx:"hidden" or
mfx:"writeonly" are still stripped from the response even if explicitly
selected — the projection happens at the database layer, not as an ACL bypass.
?select= applies to both list (GET /:model) and read
(GET /:model/:id) endpoints. It can be combined freely with filter, sort,
and include.
Putting it together
A complete request that exercises all parameters:
GET /api/posts
?filter=status:eq:published
&filter=views:gte:100
&sort=created_at:desc
&include=user,comments
&select=id,title,views,status
&page=1
&limit=20
The framework parses the query string once in the Deserialize step into
ctx.Query (a *QueryParams), which middleware can read and modify before
the DB step. Tenant-scoping middleware, for example, appends a filter to
ctx.Query.Filters to enforce row-level access — see
Example 2.
Searching
maniflex has two layers of full-text search:
- Per-model search — the
?q=parameter on a model’s list endpoint, over itsmfx:"searchable"fields. Covered in Querying. - Cross-model search — search several models at once and merge the hits into
one relevance-ranked list. That is what this page documents: the
ctx.Searchprimitive and the built-inGET /searchendpoint.
Both use the database’s native full-text engine (PostgreSQL tsvector /
ts_rank, SQLite FTS5 / bm25), provisioned automatically for every model that
declares mfx:"searchable" fields. Free-form input is sanitised, so a query can
never be a syntax error.
The ctx.Search primitive
ctx.Search runs a cross-model search and returns the merged, relevance-ranked
hits. Use it from a custom Action to build search
endpoints scoped to exactly the models you choose, with your own authorisation:
server.Action(maniflex.ActionConfig{
Method: "GET", Path: "/search-community",
Middleware: []maniflex.MiddlewareFunc{communityAuth},
Handler: func(ctx *maniflex.ServerContext) error {
hits, err := ctx.Search(maniflex.SearchOptions{
Query: ctx.QueryParam("q"),
Models: []string{"Post", "Comment"}, // explicit, app-authorised set
Limit: 20,
})
if err != nil {
ctx.Abort(400, "SEARCH_ERROR", err.Error())
return nil
}
ctx.Response = &maniflex.APIResponse{StatusCode: 200, Data: hits}
return nil
},
})
type SearchOptions struct {
Query string // the search text; blank → no-op (no results)
Models []string // models to search; empty → all GlobalSearchable models
Limit int // max merged results; <= 0 → 20
PerModelLimit int // fairness cap (see Merge order); <= 0 → pure relevance
}
type SearchResult struct {
Model string `json:"model"` // the model the hit came from
ID string `json:"id"` // primary key of the matched row
Snippet string `json:"snippet"` // excerpt of the matched text
Score float64 `json:"score"` // relevance, higher = more relevant
}
func (c *ServerContext) Search(opts SearchOptions) ([]SearchResult, error)
With an explicit Models list each named model only needs mfx:"searchable"
fields — it does not need GlobalSearchable. That flag governs only the
built-in endpoint below; the Action path is yours to authorise. ctx.Search
participates in ctx.Tx when one is active and excludes soft-deleted rows.
The built-in GET /search endpoint
Enable it explicitly, then opt models in with ModelConfig.GlobalSearchable:
server.EnableGlobalSearch() // mounts GET {PathPrefix}/search
server.MustRegister(
Post{}, maniflex.ModelConfig{GlobalSearchable: true},
Comment{}, maniflex.ModelConfig{GlobalSearchable: true},
Product{}, maniflex.ModelConfig{GlobalSearchable: true},
)
GlobalSearchable requires the model to declare at least one mfx:"searchable"
field; registration fails otherwise.
GET /api/search?q=wireless+headphones
GET /api/search?q=invoice&limit=10&models=Product
| Parameter | Default | Notes |
|---|---|---|
q | — | Required. Blank → 400 INVALID_QUERY. |
limit | 20 | Clamped to the configured maximum (default 100). |
models | all | Comma-separated subset; each name must be a GlobalSearchable model, else 400. |
The response is the standard envelope with a flat array of hits, ordered by score descending:
{
"data": [
{"model": "Product", "id": "9f8…", "snippet": "wireless …", "score": 0.61},
{"model": "Post", "id": "1a2…", "snippet": "… wireless", "score": 0.18}
]
}
Configure the route and limits via EnableGlobalSearch:
server.EnableGlobalSearch(maniflex.GlobalSearchConfig{
Path: "/search",
DefaultLimit: 20,
MaxLimit: 100,
})
Authorization
The endpoint runs only the global Auth pipeline step — it does not apply
per-model auth or tenancy middleware. Gate it with Pipeline.Auth middleware,
either globally or scoped to the search operation:
server.Pipeline.Auth.Register(requireLogin, maniflex.ForOperation(maniflex.OpSearch))
Because per-model row-level rules are not applied, only set GlobalSearchable on
models that are safe to expose this way. When you need per-model authorisation,
build a scoped Action with ctx.Search instead (see above) and attach your own
middleware. Middleware registered for OpSearch on the Deserialize, Validate,
Service, or DB steps never runs (the endpoint skips them) and is reported with a
startup warning.
Merge order
A deployment uses one database driver, so every model shares one ranking function and the scores are directly comparable; results merge by score descending.
By default the merge is pure relevance — if one model’s hits dominate, the
result can be entirely from that model. Set PerModelLimit to give each model a
fair share: the merge first takes up to PerModelLimit of each model’s
top-scoring hits, then backfills any remaining slots up to Limit from the
leftovers (best score first, regardless of model). It is a fair-chance floor, not
a hard ceiling — the result still fills to Limit when some models have fewer
hits.
Cross-model scores are a heuristic: bm25 and
ts_rankdepend on each table’s own corpus statistics, so a common term can score higher in a table where it is rarer. UsePerModelLimitwhen you want guaranteed representation across models rather than a pure score ranking.
Response Envelope
Every response from a generated route follows one of two shapes — the data envelope or the error envelope. This page documents both, along with the status codes the framework emits.
Success envelope
A successful single-row response — OpRead, OpCreate, OpUpdate:
{
"data": {
"id": "8c1a…",
"title": "First post",
"created_at": "2026-05-19T12:34:56Z",
"updated_at": "2026-05-19T12:34:56Z"
}
}
A successful list response carries the same data key plus a meta block:
{
"data": [
{ "id": "8c1a…", "title": "First post", ... },
{ "id": "9d2b…", "title": "Second post", ... }
],
"meta": {
"total": 137,
"page": 1,
"limit": 20,
"pages": 7
}
}
meta field | Meaning |
|---|---|
total | total matching rows across all pages |
page | page number returned (1-based) |
limit | rows per page |
pages | total page count, computed as ceil(total/limit) |
When a request uses cursor (keyset) pagination
(?cursor=), the meta block takes a different shape — no total/page/pages
(the count is skipped):
{ "data": [ ... ], "meta": { "limit": 20, "next_cursor": "eyJ2Ijoi...", "has_more": true } }
DELETE returns 204 No Content with no body.
Error envelope
Every error response uses:
{
"error": {
"code": "VALIDATION_ERROR",
"message": "one or more fields failed validation",
"details": [
{ "field": "email", "message": "field \"email\" is required" },
{ "field": "password", "message": "must be at least 8 characters" }
]
}
}
| Field | Meaning |
|---|---|
code | machine-readable identifier (e.g. NOT_FOUND, CONFLICT) |
message | human-readable summary |
details | optional structured payload — an array of {field, message} objects for per-field errors |
details is an array wherever it is present, including on a 409 CONFLICT from
a unique violation. That one was a bare object until v0.3.0, so a duplicate value
answered in two shapes depending on whether the database or validate.UniqueField
caught it; a client that ranged over details had to type-switch first.
A composite unique constraint contributes one entry per column, so a form can highlight every input involved:
{"error": {"code": "CONFLICT", "message": "unique constraint violation",
"details": [
{"field": "phone_number", "message": "the combination of phone_number, owner_id is already taken"},
{"field": "owner_id", "message": "the combination of phone_number, owner_id is already taken"}
]}}
The message names the combination rather than a column because neither column is in violation on its own — that phone number is fine, the pair is not.
The catalogue of built-in codes is in Error Handling.
Status codes
| Operation | Success | Notable errors |
|---|---|---|
OpList | 200 OK | 400 INVALID_QUERY |
OpRead | 200 OK | 404 NOT_FOUND |
OpCreate | 201 Created | 400 INVALID_JSON, 409 CONFLICT, 422 VALIDATION_ERROR |
OpUpdate | 200 OK | 404 NOT_FOUND, 409 CONFLICT, 422 VALIDATION_ERROR |
OpDelete | 204 No Content | 404 NOT_FOUND |
HEAD mirrors the GET for the same URL with the body suppressed: same status
(including 404 for a record that does not exist), same headers, same middleware
— just no body.
OPTIONS returns 204 No Content with an Allow header listing the methods the
route accepts.
Headers
Every response carries:
| Header | Source |
|---|---|
Content-Type: application/json | always |
X-Request-Id | echoed from chi’s RequestID middleware |
X-Service-Name | when Config.ServiceName is set |
Custom middleware can add more — see Response Middleware
for AddHeader, CORSHeaders, Cache, and friends.
Computed (virtual) fields
Server.AddComputedField registers a derived field that appears in every
read response (create echo, single read, update echo, list rows) without
being stored:
server.MustAddComputedField("Product", "stock_level",
func(ctx *maniflex.ServerContext, row map[string]any) (any, error) {
return stockService.CurrentLevel(ctx.Ctx, row["id"].(string))
})
The function runs in the Response step after the DB row has been converted
to JSON keys, so row’s keys are JSON field names. It receives the
*ServerContext, so it can reach ctx.Tx, ctx.GetModel and ctx.Auth.
Computed fields:
- Cannot be filtered or sorted — they’re materialised on output only.
- Are name-collision-checked at registration: a name that matches a real model field, or that’s already registered as computed, is rejected.
- Tolerate errors per-row — a non-nil error from the function is logged and the field is omitted from that row; the rest of the response is unaffected.
- Tolerate a panic the same way — a panicking callback is logged
(with its stack, at
ERROR) and its field omitted, exactly as a returned error is. It costs that one field on that one row: the record is still returned and the model’s other computed fields still resolve. This holds for batch callbacks too, so converting a per-row field to a batch one does not change what a bad row costs. - Run on every read path that goes through the default Response step, including the create and update echoes.
- Must be goroutine-safe. On a multi-row page the per-row callbacks run concurrently (bounded to 8 at a time), so a callback that writes to a captured variable, a shared map, or any other state must synchronise it. Batch callbacks are called once for the whole page and are not affected.
Do not use
ctx.Txfrom a per-row callback. A*sql.Txis not safe for concurrent use, and the callbacks for one page share the sameServerContext. Reads throughctx.GetModelormaniflex.Readenlistctx.Txwhen one is open, so a computed field that reads while the request is in a transaction is exactly this case. Resolve such a field withAddBatchComputedField, which runs once and sequentially.
- Appear in the OpenAPI spec as read-only properties of the model’s response schema (never in a create or update body).
Use them for derived values that change too often to denormalise (stock level, leave balance, account balance) or that depend on external systems.
Batch resolution (AddBatchComputedField)
AddComputedField runs once per row, so a resolver that queries is an
N+1: a 50-row page costs 50 round-trips. AddBatchComputedField resolves
the whole page in one call instead — this is what lets a generated
GET /store-sites return an item_count without a hand-written action:
server.MustAddBatchComputedField("StoreSite", "item_count",
func(ctx *maniflex.ServerContext, rows []map[string]any) ([]any, error) {
ids := make([]any, len(rows))
for i, r := range rows {
ids[i] = r["id"]
}
counts, err := itemCountsBySite(ctx, ids) // ONE query for the page
if err != nil {
return nil, err
}
out := make([]any, len(rows))
for i, r := range rows {
out[i] = counts[r["id"].(string)]
}
return out, nil
},
maniflex.ComputedSchema(&maniflex.OASSchema{Type: "integer"}))
The callback must return exactly one value per row, positionally aligned
to rows. A length mismatch is logged and the field is omitted from the
whole response rather than landed on the wrong records — an absent column is
diagnosable, a misaligned one is not.
One registration serves every read path: a single read and the create/update echo call it with a one-row slice, and an export calls it once per chunk of rows (so a batch field costs one call per 500 records there, not one per record).
Prefer the batch form for anything that touches a database. Per-row
resolvers run concurrently across a page, but bounded at 8 at a time — the
fan-out used to be one goroutine per row with no ceiling, so a 100-row page
fired 100 concurrent round-trips and the load scaled as page-size ×
concurrent-requests. The bound stops that from being unbounded; it does not
stop it from being an N+1. Note too that work through ctx.Tx is serialised
by the transaction’s single connection, so per-row parallelism buys nothing
there.
Declaring the type
Both callbacks return any, so the framework cannot infer a computed
field’s type. Without ComputedSchema the field still appears in the spec
(read-only) but carries no type — a generated client knows it exists but not
what it holds. maniflex.ComputedSchema(&maniflex.OASSchema{…}) declares it.
Typed variants
maniflex.AddComputedField[T] and maniflex.AddBatchComputedField[T] take
the loaded record(s) as *T / []*T instead of JSON maps:
maniflex.AddBatchComputedField(server, "StoreSite", "item_count",
func(ctx *maniflex.ServerContext, sites []*StoreSite) ([]any, error) {
// …one query, one value per site
})
Replacing the envelope
The default shape is good enough for most APIs, but if you integrate with a
client that expects a different layout, register response.Envelope from the
catalogue:
import "github.com/xaleel/maniflex/middleware/response"
server.Pipeline.Response.Register(
response.Envelope(func(ctx *maniflex.ServerContext, data any, meta *maniflex.ResponseMeta) any {
return map[string]any{
"result": data,
"paging": meta,
"trace_id": ctx.TraceID,
}
}),
)
Error responses are unaffected — they always use the {"error": {…}} shape so
clients can distinguish success from failure with a single key check.
OpenAPI Spec
maniflex generates an OpenAPI 3.1 specification from the registered models and
serves it at GET /openapi.json. The spec is derived from the same
struct tags that drive validation and querying, so it cannot drift from the
actual behaviour of the API.
The endpoint
The spec lives at /openapi.json under the configured PathPrefix:
curl localhost:8080/api/openapi.json
The response is a full OpenAPI 3.1 document — info, paths, components,
the lot. It updates automatically every time you register a model, change a
tag, or alter maniflex.Config. There is no separate codegen step.
What is generated
For every registered model, the spec includes:
- Five paths —
/<table>(GET,POST) and/<table>/{id}(GET,PATCH,DELETE). - One attachment path per
mfx:"file"field when storage is configured:GET /<table>/{id}/<file_field>withapplication/octet-stream(plus any MIME types from the field’saccept:list). See Per-model attachment routes. - Three schemas — a full response shape, a create body shape, and an
update (patch) body shape. The three differ by which fields are visible: the
create shape drops
readonlyfields; the update shape additionally dropsimmutablefields. - Standard query parameters for list endpoints —
page,limit,filter,sort,include. - Field metadata taken from
mfx:tags —enum,min,max,required,readOnly,writeOnly. - JSON / map fields — a field whose type is a
mapwith string keys is documented as a free-form{"type": "object"}. This coversmap[string]any,map[string]string, and named types over them such astype JSONObject map[string]any— the value type is not inspected. To emit a more precise shape instead, give the type its own schema (see Schemas for custom types). - Relation fields — for each relation to a registered model, the full
response schema embeds the related schema by reference (
$ref), shown when?include=requests it. A relation whose target model is not registered — for example a bareRelatedIDfield with noRelatedmodel — is omitted rather than emitting a dangling reference that would break spec validators and client generators. - Error response shapes for
400,404,409,422, and500.
hidden fields are excluded entirely from every schema. writeonly fields
appear in the create and update schemas with writeOnly: true, but not in the
response shape.
Schemas for custom types
Field types are mapped to OpenAPI schemas by their Go kind: strings, booleans,
the integer and float families, time.Time (as date-time), and any
string-keyed map (as a free-form object). A field whose type falls outside
these — a custom struct, or any type with a non-obvious JSON representation — is
omitted from the generated schema rather than guessed at.
To document such a type, make it implement the ObjectWithSchema interface:
type ObjectWithSchema interface {
Schema() *maniflex.OASSchema
}
Whenever the generator encounters a field of that type it calls Schema() and
uses the returned value verbatim — taking precedence over the built-in kind
mapping, so this also lets you override the default object shape a JSON/map
column would otherwise get. Either a value or a pointer receiver works.
For example, to document a Geo JSON column as a structured object instead of a
free-form one:
type Geo struct {
Lat float64 `json:"lat"`
Lng float64 `json:"lng"`
}
func (Geo) Schema() *maniflex.OASSchema {
return &maniflex.OASSchema{
Type: "object",
Properties: map[string]*maniflex.OASSchema{
"lat": {Type: "number", Format: "double"},
"lng": {Type: "number", Format: "double"},
},
}
}
A Geo field now renders with that exact shape in the response, create, and
update schemas. A pointer field (*Geo) is additionally made nullable.
Custom actions
Actions — custom endpoints registered with
server.Action — are included in the spec alongside the generated model
routes. Each contributes its method, path, and any {...} path parameters
automatically. Fill in ActionConfig.OpenAPI to document request and response
bodies (inferred directly from Go structs), extra query parameters, security
requirements, and a long-form description. See
Documenting an action in OpenAPI.
The OpenAPI pipeline
The spec endpoint has its own three-step pipeline, parallel to the model-route pipeline:
OpenAPI.Auth → OpenAPI.Generate → OpenAPI.Response
| Step | Purpose |
|---|---|
| Auth | Same shape as the model-route Auth step. Gate /openapi.json here. |
| Generate | Builds the spec from the registry. After-position middleware mutates it. |
| Response | Serialises the spec to JSON. |
This is reached via server.Pipeline.OpenAPI.*. See
OpenAPI Middleware for the catalogue of
spec-shaping helpers — SetTitle, AddServer, AddSecurityScheme,
AddExtension.
Securing the spec
The endpoint is public by default. To restrict it — say, to internal users only — register an Auth middleware on the OpenAPI pipeline:
import "github.com/xaleel/maniflex/middleware/auth"
server.Pipeline.OpenAPI.Auth.Register(
auth.JWTAuth("my-secret"),
)
server.Pipeline.OpenAPI.Auth.Register(
auth.RequireRole("internal"),
)
These run only for /openapi.json; they do not affect the model routes.
Viewing the spec
The framework ships a Scalar API Reference viewer at
static/openapi.html. Point StaticDir
at the directory holding it (maniflex.Config{StaticDir: "static"}) and it is
served at http://localhost:8080/static/openapi.html, loading /api/openapi.json
directly — no extra setup needed for human browsing of the generated spec.
For tooling integration, the JSON document at /openapi.json is consumable by
any OpenAPI 3.1-compatible client generator, mock server, or contract testing
framework.
Customising the spec
Most customisation is one-line, through the OpenAPI Middleware helpers. For deeper edits, write your own middleware:
server.Pipeline.OpenAPI.Generate.Register(func(ctx *maniflex.OpenAPIContext, next func() error) error {
if err := next(); err != nil {
return err
}
// ctx.Spec is the just-generated *OpenAPISpec — mutate freely.
ctx.Spec.Info.Description = "Contact the API team at [email protected]."
return nil
}, maniflex.After)
The full set of types (OpenAPISpec, OpenAPIInfo, OASSecurityScheme, …) is in
the maniflex package.
Database Backends
maniflex ships two database adapters, both built on database/sql and sharing a
single SQL core (db/sqlcore). They expose the same interface; switching
between them is one import line.
| Adapter | Module | Driver |
|---|---|---|
| SQLite | maniflex/db/sqlite | modernc.org/sqlite — pure Go, no CGo |
| PostgreSQL | maniflex/db/postgres | github.com/lib/pq |
Each adapter lives in its own Go module so a project only pulls in the driver it actually uses.
SQLite
The default choice for development, tests, and small deployments. The pure-Go
driver means no CGo and no external service — go run . is enough to start a
local server with a working database.
import "github.com/xaleel/maniflex/db/sqlite"
db, err := sqlite.Open("./app.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
Common DSNs:
| DSN | Effect |
|---|---|
./app.db | persistent file in the working directory |
:memory: | per-process in-memory database; vanishes on shutdown |
SQLite is single-writer by design. The framework serialises writes through one connection internally; reads run on a pool. This is plenty for most internal tools and many production APIs.
Write connections open their transactions with BEGIN IMMEDIATE (the
_txlock=immediate DSN option, applied for you). That is what makes a
read-then-write transaction — LockForUpdate, an If-Match check,
mfx:"lock_scope" — behave the way it does on Postgres: a second transaction
waits at its BEGIN rather than reading the same stale row and then failing, or
overwriting, on the way out. Spell out your own _txlock= in the DSN and yours
is kept.
PostgreSQL
The recommended adapter for any multi-process deployment. It supports
genuine concurrent writers, real FOR UPDATE locks, and read replicas.
import "github.com/xaleel/maniflex/db/postgres"
// Open(writeDSN, readDSN, registry) — positional arguments.
db, err := postgres.Open(
"postgres://user:pass@host/db?sslmode=require", // write DSN
"postgres://user:[email protected]/db?sslmode=require", // read DSN (optional)
server.Registry(),
)
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
Pass an empty read DSN ("") to route reads to the primary. The adapter selects
the appropriate pool per request based on the operation — OpList and OpRead
go to the read pool, everything else to the write pool.
For connection-pool and session tuning use postgres.OpenWithConfig(writeDSN, readDSN, registry, writePool, readPool, session) — see
PostgreSQL in Production for replica lag
handling and SSL.
Switching between them
The adapter is the only thing that changes; nothing else in the application needs to know which database is in use:
- import "github.com/xaleel/maniflex/db/sqlite"
+ import "github.com/xaleel/maniflex/db/postgres"
- db, err := sqlite.Open("./app.db", server.Registry())
+ db, err := postgres.Open(os.Getenv("DB_URL"), "", server.Registry())
Models, middleware, and queries are portable across both backends because they
go through database/sql + the shared db/sqlcore adapter. Migrations
emitted by AutoMigrate use a portable subset of SQL.
AutoMigrate
Unless Config.DisableAutoMigrate is set (migration runs by default), the adapter:
- Creates any table that does not yet exist for a registered model.
- Adds any column that exists on the struct but not in the table.
- Logs a warning for columns that exist in the table but not on the struct (the framework never drops columns automatically).
- Logs a warning for a column whose type no longer matches its model field (the framework never rewrites columns automatically).
- Creates indexes declared in
ModelConfig.Indicesor auto-generated formfx:"scheduled"fields.
AutoMigrate adds; it does not rewrite. Change a field’s Go type — int to
string, say — and the existing column keeps the type it was created with. You
get a warning naming the table, the column, and both types on every startup, but
the schema is left alone: rewriting a column can lose data and locks the table
while it runs, so the conversion (and whatever backfill it needs) belongs in an
explicit, versioned migration you run against the database, not in a startup
routine that has to guess. Until that migration runs, reads and writes of the
field can fail against the old column.
AutoMigrate is suitable for development and many small deployments. For
larger systems, set DisableAutoMigrate: true and manage the schema with a
dedicated migration tool.
The DBAdapter interface
Both shipped adapters implement maniflex.DBAdapter. Custom backends — an HTTP
data service, a remote API, a different SQL database — implement the same
interface and inject the result through server.SetDB(myAdapter). The
interface is in db.go.
Adapters are compared by identity
The framework uses == on DBAdapter values to decide whether two models share
a database — that is what stops a transaction from silently spanning two of them.
So a custom adapter must:
- use a pointer receiver — return
*MyAdapter, notMyAdapter. Two separately constructed value-type adapters with equal fields compare equal, so the framework would treat two databases as one and let a transaction span them. That fails silently, not loudly. - stay comparable — comparing interface values whose dynamic type is not comparable (a struct holding a map, slice, or func, used as a value) panics at run time. A pointer receiver gives you this for free.
Nothing in the type system enforces either, which is why it is written down here
and on the DBAdapter godoc.
Per-model adapter routing
Config.DB sets the default adapter. Individual models can override it by
passing ModelConfig.Adapter:
ordersDB, _ := postgres.Open(ordersDSN, "", server.Registry())
inventoryDB, _ := postgres.Open(inventoryDSN, "", server.Registry())
server.MustRegister(
Order{}, maniflex.ModelConfig{Adapter: ordersDB},
InventoryItem{}, maniflex.ModelConfig{Adapter: inventoryDB},
User{}, // unrouted — falls back to Config.DB
)
“Distinct” here means distinct under ==. Two models that share a database must
be given the same adapter value, not two opened against the same DSN — those
are separate connection pools and separate transactions, and nothing tells the
framework they point at one database.
The framework treats each distinct adapter as its own database:
- AutoMigrate runs once per adapter, with a filtered registry view so each
adapter only sees the models routed to it. Tables for
Orderare never created on the inventory DB and vice-versa. - CRUD requests (
GET /orders,POST /orders) route throughOrder.Adapter. The DB step picks the per-model adapter automatically. ctx.BeginTx/ctx.RawQuery/ctx.RawExecuse the request’s model adapter, so middleware and custom actions stay on the right DB.ctx.GetModel("OtherModel")uses the target model’s adapter — handy for cross-DB reads — but it cannot share a transaction across adapters: ifctx.Txwas opened ondbAand you callGetModel("X")whereXlives ondbB, the accessor falls back to a non-transactional read againstdbB.
Config.DB is optional when every registered model has its own Adapter.
The server starts cleanly with DB: nil and routes everything through the
per-model overrides. If any model is unrouted and Config.DB is also nil,
startup fails with a clear error naming the unrouted models.
Constraint: transactions are adapter-scoped
A single database transaction cannot span two adapters. Two consequences:
maniflex.Batchrejects ab.Create("X", ...)call whereXroutes to a different adapter than the batch transaction was opened on. The error message points topkg/sagaas the cross-adapter pattern.- Manually-opened
ctx.Txonly protects writes against the request’s own model adapter. Cross-adapter writes throughctx.GetModel(...)happen outside that transaction.
For coordinated writes across databases, use pkg/saga —
compensating transactions are the supported pattern.
Choosing
| Need | Pick |
|---|---|
| Quick start, tests, small single-process services | SQLite |
| Multi-process deployment, real concurrency, replicas | PostgreSQL |
| Both (the codebase will outgrow SQLite) | SQLite locally, Postgres in production — same code |
Configuration
maniflex.Config is the single struct passed to maniflex.New. Every field has a
sensible default; populate only the ones that differ from those defaults.
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
Server
| Field | Default | Purpose |
|---|---|---|
Port | 8080 | TCP port the HTTP server binds to |
PathPrefix | /api | URL prefix prepended to every generated model route and /openapi.json |
ServiceName | "" | service identifier added to logs, audit records, and the X-Service-Name response header |
StaticDir | "" | filesystem directory served as static files; empty serves nothing (opt-in). Relative paths resolve against cwd |
StaticPrefix | /static | URL prefix the static directory is mounted under, at the router root |
StaticDisabled | false | turn static file serving off even when StaticDir is set |
PathPrefix does not affect /static, /files, or /health. Those are
mounted at the router root. See Static Files for the static
serving options.
HTTP timeouts
The framework owns the http.Server, and net/http gives that struct no
deadlines by default. These are the ones it sets on your behalf:
| Field | Default | Purpose |
|---|---|---|
ReadHeaderTimeout | 10s | how long a connection may take to send its request headers |
IdleTimeout | 120s | how long a keep-alive connection may sit idle between requests |
ReadTimeout | 0 (unbounded) | time to read an entire request, headers and body |
WriteTimeout | 0 (unbounded) | time to write a response |
ReadHeaderTimeout is the slowloris defence. Without it a client can hold a
connection open forever by dribbling one header byte at a time, and enough such
connections exhaust the server’s file descriptors without a single request ever
reaching the pipeline. Set a negative value to disable a timeout (which is
what net/http reads as “no deadline”) — only sensible behind a proxy that
already bounds header reads.
ReadTimeout and WriteTimeout are deliberately left unset, because both are
whole-request deadlines rather than idle deadlines:
- A
ReadTimeoutcaps how long a client may take to upload, so a large file over a slow link is severed mid-transfer. - A
WriteTimeoutcovers the entire response, so any value at all would cut a long-lived stream —realtime.SSEHandler, a large download — off at that mark.
Set them when you know your request sizes and have no streaming endpoints. The
header phase stays bounded by ReadHeaderTimeout either way.
Limits
| Field | Default | Purpose |
|---|---|---|
MaxConcurrentExports | 4 | how many GET /:model/export requests may run at once, server-wide; negative disables the limit |
An export holds its entire result set in memory until the last byte reaches the
client, so concurrency multiplies the largest allocation the server makes. The
per-model MaxExportRows bounds one export’s rows but not the row width nor the
number in flight; this bounds the product. Requests over the limit are refused
immediately with 503 EXPORT_BUSY and a Retry-After, not queued. See
CSV / XLSX Export.
Database
| Field | Default | Purpose |
|---|---|---|
DB | nil | the default DBAdapter. Usually set via server.SetDB(db) after MustRegister. Optional when every model has its own ModelConfig.Adapter — see Per-model adapter routing |
DisableAutoMigrate | false | skip schema migration on startup (migration runs by default) |
DBWriteURL | "" | DSN for the primary database (informational; populated by ConfigFromEnv) |
DBReadURL | "" | DSN for the read replica (informational) |
QueryTimeout | 0 (unlimited) | per-request deadline applied to all DB calls; exceeding it produces 504 TIMEOUT |
See Database Backends for adapter construction.
File storage and encryption
| Field | Purpose |
|---|---|
FilesConfig.Storage | maniflex.FileStorage implementation for mfx:"file" fields and the /files endpoints. Required if any model uses file uploads. See File Fields & Uploads. |
FilesConfig.BeforeMiddlewares | []maniflex.MiddlewareFunc wrapping the standalone /files endpoints. Empty = no auth (backward-compatible default); production deployments should populate this with at least an auth middleware. See File Fields & Uploads. |
KeyProvider | maniflex.KeyProvider for mfx:"encrypted" fields. Without one, encrypted fields refuse writes with 500 ENCRYPTION_NOT_CONFIGURED. |
Logging
| Field | Default | Purpose |
|---|---|---|
Logger | slog.Default() | logger used for lifecycle, per-request, and adapter messages |
PanicLogger | falls back to Logger | sink for the panic recoverer’s structured panic records |
Trace | zero (off) | pipeline tracing — see below |
Logger is used by ctx.Logger(), which adds request_id, trace_id, and
service attributes per request. Route it to a JSON handler in production.
Pipeline tracing
Config.Trace enables verbose debug output of the request pipeline. All trace
output is at DEBUG level through Logger, so the handler must accept
DEBUG records to see anything.
| Sub-flag | Effect |
|---|---|
Enabled | shorthand for Steps + Timings + Aborts |
Steps | enter/exit record per middleware |
Timings | per-middleware elapsed time on exit records |
Aborts | the source file:line of every ctx.Abort call |
Bodies | log field names present in ctx.ParsedBody (opt-in; may expose sensitive field names) |
Skips | log middleware skipped by ForModel/ForOperation filters |
cfg.Trace = maniflex.PipelineTrace{Enabled: true, Skips: true}
Enabled expands into the three standard flags unless one of those three is
already set, in which case it stays out of the way and you get exactly what you
named. Bodies and Skips are additive: setting one does not suppress the
expansion, so the example above gives all four.
cfg.Trace = maniflex.PipelineTrace{Enabled: true, Steps: true} // Steps only
cfg.Trace = maniflex.PipelineTrace{Bodies: true} // Bodies only
Leave Bodies off in production.
Lifecycle
| Field | Default | Purpose |
|---|---|---|
ShutdownTimeout | 30s | maximum time Start() waits for in-flight requests to finish on SIGINT / SIGTERM before forcing the listener closed |
See Graceful Shutdown.
Health probe
| Field | Default | Purpose |
|---|---|---|
HealthCheckDB | false | when true, GET /health pings every distinct registered adapter (Config.DB plus any per-model overrides) and returns 503 on failure. Driver error messages are logged, not echoed in the response body, so DSN fragments can’t leak. |
HealthTimeout | 3s | maximum time the health handler waits for the DB ping |
Set HealthTimeout shorter than your probe’s timeoutSeconds so the handler
can return 503 cleanly before the probe times out.
Reading from environment
maniflex.ConfigFromEnv(prefix) populates a Config from a conventional set of
environment variables. Use it for twelve-factor deployments, then override
individual fields in code where needed.
cfg, err := maniflex.ConfigFromEnv("") // or "ORDERS" → ORDERS_PORT, ORDERS_DB_WRITE_URL, …
if err != nil {
log.Fatal(err)
}
cfg.DisableAutoMigrate = true // disable for production
server := maniflex.New(cfg)
These are the variables it reads, and the only ones — anything else on Config
is set in code:
| Variable | Field | Value |
|---|---|---|
PORT | Port | integer, 1–65535 |
DB_WRITE_URL | DBWriteURL | string |
DB_READ_URL | DBReadURL | string |
QUERY_TIMEOUT_MS | QueryTimeout | positive integer, milliseconds |
SHUTDOWN_TIMEOUT_S | ShutdownTimeout | positive integer, seconds |
SERVICE_NAME | ServiceName | string |
HEALTH_CHECK_DB | HealthCheckDB | true/false, 1/0, yes/no, on/off |
A variable that is unset leaves its field at the zero value, for
ApplyDefaults to fill in. A variable that is set but unreadable is an
error — PORT=808O, QUERY_TIMEOUT_MS=abc, HEALTH_CHECK_DB=ture — naming the
variable and the value it could not read. Every bad variable is reported at once,
so two typos take one deploy to find rather than two. Don’t discard this error: a
mistyped PORT that is quietly ignored gives you a healthy-looking server
listening on 8080, and nothing anywhere says why.
Graceful Shutdown
server.Start() blocks on the HTTP listener and additionally listens for
SIGINT and SIGTERM. When either signal arrives, the server stops accepting
new connections and gives in-flight requests up to
Config.ShutdownTimeout to finish before forcing the listener closed.
How it works
- A signal arrives.
http.Server.Shutdown(ctx)is called with a deadline ofConfig.ShutdownTimeout(default: 30 seconds).- The listener stops accepting new connections immediately.
- In-flight requests are allowed to complete — including their pipeline middleware, transaction commits, and Response writes.
- When all requests have finished, or the deadline elapses, the database
adapter’s
Close()is called. Start()returns.
If the deadline passes with requests still running, the underlying TCP connections are closed — those requests fail mid-flight but the process exits cleanly.
Tuning ShutdownTimeout
Pick the value based on the longest legitimate request your service serves:
| Environment | Suggested ShutdownTimeout |
|---|---|
| Tests | 0–1s — exit instantly |
| Lambdas / fast-cycling containers | 5–10s |
| General OLTP API | 30s (default) |
| Bulk import or large file uploads | 60s+ |
Setting ShutdownTimeout shorter than your slowest request will sever it on
shutdown. Setting it longer makes deploys slower with no benefit beyond the
slowest real request.
Why graceful shutdown matters
Cutting a request mid-write produces inconsistent state at the boundary — a write that may or may not have committed, a webhook that may have fired but not been recorded, a client that may or may not have seen the response. The graceful path:
- ensures transactions commit or roll back cleanly,
- lets the Response step write its envelope before the connection drops,
- gives
maniflex.WithTransaction’s deferred rollback a chance to run.
For Kubernetes deployments, set terminationGracePeriodSeconds on the pod to
a value larger than ShutdownTimeout, otherwise the orchestrator will send
SIGKILL before the graceful handler completes.
Manual shutdown
For tests or custom lifecycle code, the same graceful path is available without waiting for a signal:
go server.Start()
// ... run tests ...
server.Shutdown(ctx)
Shutdown uses the supplied context as the deadline. Pass context.Background
for “wait as long as it takes”; pass a context.WithTimeout for an explicit
budget.
It is safe to call from any goroutine at any point in the server’s life — the
snippet above races Start by construction, and the outcome does not depend on
who wins. A Shutdown that lands while the server is still booting (migrating,
or waiting on a service that dials its backend) countermands the boot: the
listener is never opened, Start unwinds whatever it had already brought up and
returns nil, and Shutdown waits for that to finish before returning. On a
server that was never started at all, it does nothing.
Shutdown is terminal rather than a pause: a server that has been shut down will
not open a listener afterwards, so a Start following a Shutdown returns nil
without serving. A Server is not restartable — build a new one.
Background writes
Audit-log writes, cache invalidations (db.Invalidate), and async file
cleanups (Config.FilesConfig.Storage with mfx:"auto_delete" fields) run on
goroutines tracked by the server. Shutdown waits for those to drain
within the same deadline as the HTTP listener. If the deadline elapses
with goroutines still in flight, the server logs a warning with the
in-flight count and proceeds — the goroutines see their context cancelled
and exit on the next checkpoint.
Custom middleware can opt into the same lifecycle via
ctx.GoBackground(fn func(context.Context)); the supplied context is
independent of the request (which has already returned) but IS cancelled
when shutdown’s deadline hits.
Supervised services & lifecycle hooks
Applications often own long-lived background components — a poller, cache
warmer, queue consumer, or an in-memory pool manager — that must start after
the database is ready and stop cleanly before the process exits. Register
them as services and the framework folds them into the boot and shutdown
lifecycle instead of you hand-supervising them around Start.
type Service interface {
Start(ctx context.Context) error // ctx is cancelled at shutdown
Stop(ctx context.Context) error // carries what's left of the shutdown budget
}
server.AddService(pool) // a custom Service
server.AddService(maniflex.ServiceFunc(startFn)) // adapter for a bare start func
AddService must be called before Start — it panics once Start has been
entered, including while the server is still migrating or bringing earlier
services up. Boot fixes the service list when it begins, so a service registered
after that point would never be started, and a panic is a better answer than a
component that silently never runs. The same applies to Action, RealtimeDoc
and EnableGlobalSearch, which fix the routing table.
For app-scoped fire-and-forget work (e.g. a periodic reconciler) that doesn’t
need an ordered Stop, use server.Go. Its context is cancelled when shutdown
begins, and the goroutine is drained before Start returns:
server.Go(func(ctx context.Context) {
t := time.NewTicker(time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
reconcile(ctx)
}
}
})
Callers that want a hook without defining a Service type can set the
lightweight Config.OnStart / Config.OnShutdown functions.
Boot order: migrate → OnStart → Service.Start (registration order) → listen. A Start (or OnStart) error aborts boot exactly like a failed
migration; services that already started are stopped in reverse first.
A failed boot still tears down. Whatever boot managed to bring up is put back
down before Start returns the error — including the server.Go loops, which
run from the moment you call server.Go, not from Start. If the listener fails
to bind (the port is taken), the services stop and the goroutines drain exactly as
they do on the graceful path, in a ShutdownTimeout window of its own; if the
migration or a service refused to start, the loops are cancelled and awaited. Your
goroutines are never abandoned mid-write, however boot ends.
Shutdown order: http.Shutdown → Service.Stop (reverse order) → OnShutdown → drain server.Go + ctx.GoBackground goroutines. The Start context is
cancelled when shutdown begins so loops wind down on their own.
One budget, shared. Every phase runs on the same deadline context —
ShutdownTimeout, or whatever deadline you pass to server.Shutdown(ctx). The
phases are sequential, so they draw down a single window rather than each getting
a fresh one: a drain that eats 25 of 30 seconds leaves Stop, OnShutdown and
the goroutine drain 5 seconds between them. Honour the ctx you are handed — that
is what keeps total shutdown inside the window your orchestrator allows before it
escalates to SIGKILL.
AddService, OnStart, and server.Go are inert for apps that register
nothing — there is no behavioural change unless you opt in.
Health probes during shutdown
Once shutdown begins, /health continues to respond for a brief window
because in-flight requests are honoured. Configure your readiness probe to
stop directing traffic to the pod as soon as termination begins (Kubernetes
does this automatically when it sends SIGTERM).
Satellite Modules
maniflex is a multi-module monorepo. The core module — also named maniflex —
carries only chi and uuid. Every heavy dependency (a database driver, a
message broker client, a crypto library) lives in its own satellite module
under the same repository, so a consumer pulls only the dependencies it
imports.
Layout
maniflex/ # core module — chi + uuid
├── maniflex/ # framework
├── storage/ # local disk file storage
└── ...
storage/
└── s3/ # aws-sdk-go-v2 — S3, MinIO, R2, Spaces, etc.
db/
├── sqlite/ # modernc.org/sqlite — pure-Go SQLite
├── postgres/ # lib/pq — PostgreSQL
└── sqlcore/ # shared SQL adapter used by both
events/
├── kafka/ # segmentio/kafka-go
├── nats/ # nats.go
├── rabbitmq/ # rabbitmq/amqp091-go
└── redis/ # go-redis
jobs/
├── inproc/ # goroutine pool — tests and single-binary apps
├── sql/ # *sql.DB-backed (Postgres + SQLite) with transactional outbox
├── redis/ # Redis Streams / BRPOP — high-throughput worker fleets
├── cron/ # scheduled EnqueueAt ticker
└── maniflex/ # StatusModel + Mount helper — REST polling for job status
middleware/
├── auth/, body/, db/, … # catalogue middleware (see Middleware Catalogue)
├── service/bcrypt/ # golang.org/x/crypto for password hashing
└── db/redis/ # Redis cache invalidation
examples/ # runnable example apps — its own module
tests/ # e2e suite — its own module
Why split modules
The split keeps the core dependency graph minimal:
- A project that uses SQLite imports
maniflex/db/sqliteand gets the pure-Go driver. PostgreSQL’slib/pqis not in its build. - A project that publishes events to Kafka imports
maniflex/events/kafkaand pulls insegmentio/kafka-go. NATS and RabbitMQ stay out of the build. - A project that does not authenticate doesn’t import
middleware/authand pays nothing for the JWT library.
This matters most for binary size, attack surface, and CI build time. It also keeps the core stable: changing a database driver does not require a release of the framework itself.
Importing satellites
Each satellite is a normal Go module — add it with go get:
go get github.com/xaleel/maniflex # core
go get github.com/xaleel/maniflex/db/sqlite # SQLite adapter
go get github.com/xaleel/maniflex/middleware/auth # auth helpers
go get github.com/xaleel/maniflex/events/kafka # Kafka publisher
In code:
import (
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
"github.com/xaleel/maniflex/middleware/auth"
"github.com/xaleel/maniflex/events/kafka"
)
There are no required satellites for the framework itself to function —
maniflex alone gives you the registry, pipeline, and HTTP layer. You need
at least a database adapter (sqlite or postgres) before server.Start() can
serve a request.
Workspace mode
The repository ships a go.work file that includes every satellite module.
For consumers, this is invisible — Go modules resolve normally through
go.mod. For contributors working across modules, go.work makes
cross-module changes possible without replace directives.
go build ./... and go test ./... operate per-module. To build or test
every module at once, use the helper scripts in scripts/:
bash scripts/test-all.sh
# or
powershell scripts/test-all.ps1
Versioning
Each satellite carries its own v0.x tags. The core module is the only one a
typical app depends on by name; satellites are usually pulled in transitively
or by direct import as needed. Pin satellite versions in go.mod when
reproducibility across machines is required.
Admin Panel
maniflex/admin is an opt-in satellite module that mounts a server-rendered
administration panel on top of any maniflex server. It introspects the model
registry to build its navigation and views, and reads/writes data by issuing
in-process HTTP requests against the server’s own REST API — so every
operation travels the full auth/validate/pipeline stack. The admin never
touches the database directly.
Adding the module
go get github.com/xaleel/maniflex/admin
Because it is a satellite, importing it is the only thing needed to bring the
admin into your binary. The core maniflex module has no dependency on it.
Quick start
package main
import (
"net/http"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/admin"
"github.com/xaleel/maniflex/db/sqlite"
"github.com/go-chi/chi/v5"
)
func main() {
server := maniflex.New(maniflex.Config{PathPrefix: "/api"})
server.MustRegister(User{}, Post{}, Comment{})
db, _ := sqlite.Open(":memory:", server.Registry())
server.SetDB(db)
adminHandler := admin.Mount(server, admin.Config{
Title: "My App Admin",
AllowUnauthenticated: true, // local dev only
})
r := chi.NewRouter()
maniflex.Mount(r, server)
r.Mount("/admin", http.StripPrefix("/admin", adminHandler))
http.ListenAndServe(":8080", r)
}
Mount must be called after all models are registered and the DB adapter
is set, and before the server starts handling requests. It panics early if
neither Config.Auth nor Config.AllowUnauthenticated is set, so an
unprotected panel can never be shipped by accident.
Config reference
| Field | Type | Default | Description |
|---|---|---|---|
PathPrefix | string | "/admin" | Mount path; the returned handler serves routes under this prefix |
Title | string | "maniflex admin" | Displayed in the panel header |
Auth | func(http.Handler) http.Handler | — | Wraps the whole panel with an auth gate; required unless AllowUnauthenticated is set |
AllowUnauthenticated | bool | false | Skips the auth requirement; local dev only |
Models | []string | (all) | Struct names to show; empty means every registered model |
ReadOnly | bool | false | Hides create/edit/delete UI and unmounts those routes |
Templates | fs.FS | — | Override FS for custom templates (see Templates) |
StaticFS | fs.FS | — | Replaces the embedded CSS/asset bundle |
Authentication
Set Config.Auth to any func(http.Handler) http.Handler middleware — for
example the JWT middleware from maniflex/middleware/auth:
import "github.com/xaleel/maniflex/middleware/auth"
adminHandler := admin.Mount(server, admin.Config{
Title: "My Admin",
Auth: auth.JWTAuth(secret, auth.JWTOptions{}),
})
The Auth wrapper runs before every panel request. Because data reads and
writes go through the API pipeline, the upstream Auth middleware registered
on your model endpoints also enforces field-level and operation-level rules —
the admin doesn’t bypass them.
Never set
AllowUnauthenticated: truein a production deployment. Mount panics at startup if neither option is provided, so there is no way to accidentally omit auth and only discover it at runtime.
Views
Dashboard
GET /admin/ — one summary card per visible model showing its total row
count, fetched in-process from the API.
List
GET /admin/{model} — a paginated table of records.
- Pagination — 20 rows per page;
?page=Nnavigates. - Sorting — a dropdown built from fields tagged
mfx:"sortable". The current direction is preserved across filter changes. - Filtering — one input per field tagged
mfx:"filterable". Enum fields render a<select>; other fields render a text input. Active filters persist in the URL as?f_<field>=<value>.
Detail
GET /admin/{model}/{id} — all readable fields for one record.
- FK fields rendered as links to the related record (
/admin/{related}/{fk_id}). HasManyrelations shown as “View {related}” links (list pre-filtered to this record’s ID).- Edit and Delete actions, each protected by a CSRF token.
Create form
GET /admin/{model}/new — an empty form; POST /admin/{model} submits it.
Edit form
GET /admin/{model}/{id}/edit — the form pre-filled from the existing record;
POST /admin/{model}/{id} submits it.
Both forms share the same template and widget logic:
| Widget | When used |
|---|---|
text | default string fields |
textarea | long-text / text DB type |
number | integer and float fields |
checkbox | boolean fields |
select | fields with mfx:"enum:…" |
relation | BelongsTo FK fields — a <select> populated from the target model |
file | fields tagged mfx:"file" — includes a preview/download link when a file is already stored |
datetime | time.Time fields — rendered as an <input type="datetime-local"> |
Fields tagged mfx:"hidden" or mfx:"writeonly" are excluded from the list
and detail views. Fields tagged mfx:"readonly" appear on the edit form as
disabled inputs (they are server-managed). mfx:"immutable" fields are
editable on create but disabled on edit.
Delete
POST /admin/{model}/{id}/delete — deletes the record via the API and
redirects to the list. Requires a valid CSRF token (present on the detail
page’s Delete button).
CSRF protection
The panel uses double-submit cookies. On first form load a random 32-byte
hex token is written to the CSRF cookie and mirrored in a hidden _csrf form
field. Every mutating POST verifies that both match before forwarding to the
API. The cookie is HttpOnly and SameSite=Lax, and is marked Secure when the
panel is served over TLS — directly or behind a proxy that sets
X-Forwarded-Proto: https. There is nothing to configure — it is always on.
Model whitelist
To show only a subset of registered models:
admin.Mount(server, admin.Config{
Models: []string{"User", "Post"},
// "Comment" will not appear in the panel
})
Model names are Go struct names, not table names. Models omitted from the whitelist are hidden from navigation, list, and detail views — they are still served by the API.
Read-only mode
admin.Mount(server, admin.Config{
ReadOnly: true,
})
In read-only mode the create/edit/delete routes are not mounted at all, and the corresponding controls are hidden in the UI. Useful for support teams that need visibility without write access.
Templates
Drop in a replacement for any individual template by providing a fs.FS on
Config.Templates. Any file not present in the override FS falls back to
the embedded default. The template file names are:
| File | View |
|---|---|
layout.html | outer chrome (header, sidebar, <head>) |
dashboard.html | model summary cards |
list.html | paginated table |
detail.html | single-record field list |
form.html | shared create/edit form |
error.html | error page |
Example — override only the layout to inject custom branding:
//go:embed templates
var myTemplates embed.FS
admin.Mount(server, admin.Config{
Templates: myTemplates,
})
The templates receive the viewData struct. Consult the admin package source
(view.go) for the full shape of each page’s data.
Static assets
The embedded asset bundle is served under {PathPrefix}/static/. To replace
it entirely with a custom CSS file:
//go:embed assets
var myAssets embed.FS
admin.Mount(server, admin.Config{
StaticFS: myAssets,
})
StaticFS replaces the whole bundle — include any assets the templates
reference (or adjust the templates to match).
How it works
The panel is self-contained: it holds a reference to server.Handler() and
issues normal http.Request objects against it in-process. There is no
separate HTTP round-trip.
browser → GET /admin/users
→ admin handler
→ apiClient.list(r, "users", "limit=20&sort=…")
→ server.Handler().ServeHTTP(rw, r') // in-process
→ full pipeline (Auth → Validate → DB → Response)
← []map[string]any
← rendered list.html
This means:
- Pipeline middleware on the model (tenant isolation, field redaction, soft- delete visibility) is enforced on every admin read and write.
- Auth cookies or tokens present on the browser request are forwarded unchanged to the API, so per-user permission checks work automatically.
- The admin has no SQL access of its own and cannot bypass business rules.
→ Satellite Modules
→ Field Tags Reference
→ File Fields & Uploads
→ Pipeline Overview
Custom Endpoints (Actions)
The five generated REST routes per model cover the standard CRUD shape, but
some endpoints don’t fit that shape — POST /orders/{id}/cancel,
POST /invoices/{id}/send, GET /reports/revenue. Actions are maniflex’s
mechanism for adding these.
Registering an action
An action is a method, a path, and a handler:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Handler: cancelOrder,
})
The handler receives the standard *maniflex.ServerContext:
func cancelOrder(ctx *maniflex.ServerContext) error {
orderID := ctx.URLParam("id")
if _, err := ctx.GetModel("Order").Update(orderID, map[string]any{
"status": "cancelled",
}); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusOK,
Data: map[string]any{"ok": true},
}
return nil
}
The trimmed pipeline
Action requests run a shorter pipeline than CRUD requests:
Auth → [per-action middleware...] → handler → Response
Deserialize, Validate, Service, and DB are skipped. The action handler
is responsible for parsing its own body (ctx.BindJSON) and performing its
own database work (via ctx.GetModel, ctx.RawExec, or directly).
ctx.Operation is OpAction inside the handler. Middleware registered on the
trimmed-out steps with ForOperation(maniflex.OpAction) does not run; only Auth
and Response middleware do.
DB-step middlewares don’t cover actions. Anything registered on
Pipeline.DB— includingdb.RateLimit,db.AuditLog,db.ForceFilter, anddb.Tenancy— is silently skipped for action routes. For an all-action service this means zero rate limiting, zero audit records, and no tenancy unless you wire them per action. Use the action-flavoured variants in the action’s ownMiddlewarelist:db.RateLimitAction(cfg)(keys on the caller + method/path),db.AuditLogAction(sink)(records actor/resource/result from the action context), anddb.TenancyAction(field, fn)/db.ForceFilterAction(field, fn)(row-level scoping — see Scoping Actions).
Under db.TenancyAction / db.ForceFilterAction the handler’s database work is
scoped through ctx.GetModel, the typed generics, ctx.Aggregate,
ctx.LockForUpdate, and the Tx from ctx.BeginTx (so WithTransaction and
Batch work too) — and ctx.RawQuery, ctx.RawExec, ctx.Search and
ctx.RecursiveQuery refuse, because a scope cannot be applied to raw SQL and
running it anyway would return every tenant’s rows. ctx.Unscoped() bypasses
that deliberately where a path genuinely must. An action with no scope registered
is unaffected: every path behaves as it always has.
Per-action middleware
Actions can carry their own middleware list, which runs between Auth and the
handler:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Handler: cancelOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.RequireRole("admin"),
idempotency.Key("Idempotency-Key"),
},
})
This is the equivalent of the Service step for an action — anything that should run before the handler but after authentication.
Reading input
The action handler does its own request parsing:
type RefundReq struct {
Amount float64 `json:"amount"`
Reason string `json:"reason"`
}
func refundOrder(ctx *maniflex.ServerContext) error {
var req RefundReq
if err := ctx.BindJSON(&req); err != nil {
return nil // ctx.Abort already called
}
// ... work ...
ctx.Response = &maniflex.APIResponse{StatusCode: http.StatusOK}
return nil
}
ctx.BindJSON enforces the same 4 MB body limit as the default Deserialize
step. ctx.URLParam and ctx.QueryParam read URL and query parameters.
Multipart uploads:
ctx.Filesis populated by the Deserialize step, which actions skip — so it is always empty inside an action. To accept a file upload in an action, parse the request yourself:if err := ctx.Request.ParseMultipartForm(32 << 20); err != nil { ctx.Abort(http.StatusBadRequest, "BAD_REQUEST", "invalid multipart form") return nil } for _, headers := range ctx.Request.MultipartForm.File { // headers[0].Open() → the uploaded file }
Transactional actions
ctx.BeginTx works inside an action just as it does in middleware. For most
actions, wrap the handler body in a BeginTx / Commit block:
func cancelOrder(ctx *maniflex.ServerContext) error {
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
// ... transactional work via ctx.GetModel / ctx.RawExec ...
return tx.Commit()
}
Because the action does not pass through the Service step, maniflex.WithTransaction
registered there does not apply — actions manage their own transactions.
SQLite deadlock — fetch the model accessor after setting
ctx.Tx. The SQLite adapter uses a single write connection (MaxOpenConns(1)). Actx.GetModel(...)accessor binds to whateverctx.Txis at the time you callGetModel. If you grab the accessor beforeBeginTxand then write with it afterctx.Tx = tx, the write opens a second writer connection and blocks forever behind the transaction holding the single writer — the request hangs with no error. Always callctx.GetModel(...)afterctx.Tx = tx:tx, _ := ctx.BeginTx(ctx.Ctx, nil) defer tx.Rollback() ctx.Tx = tx orders := ctx.GetModel("Order") // bound to ctx.Tx now — safe orders.Update(id, patch) if err := tx.Commit(); err != nil { return err } ctx.Tx = nil // reset: reads built for the response after commit must not // route through the finished tx
Streaming a raw (non-JSON) response
An action normally returns JSON by setting ctx.Response. For a binary or
streaming endpoint — serving an image, a generated PDF, a CSV export — write
directly to ctx.Writer (the raw http.ResponseWriter) and leave
ctx.Response nil. After the handler returns, the framework writes nothing
further when ctx.Response is nil, so the bytes you wrote are the whole
response:
func downloadEvidence(ctx *maniflex.ServerContext) error {
f, err := openEvidence(ctx.URLParam("id"))
if err != nil {
ctx.Abort(http.StatusNotFound, "NOT_FOUND", "evidence not found")
return nil
}
defer f.Close()
ctx.Writer.Header().Set("Content-Type", "image/png")
ctx.Writer.WriteHeader(http.StatusOK)
_, err = io.Copy(ctx.Writer, f) // leave ctx.Response nil
return err
}
When the bytes are an mfx:"file" model field, prefer the built-in per-model
attachment route (GET /{model}/{id}/{field}) instead — it runs the read
pipeline (auth, soft-delete, tenancy) and streams for you. If you scope a
db.ForceFilter to OpList/OpRead, remember to add OpReadAttachment too, or
downloads bypass the filter.
Documenting an action in OpenAPI
Actions appear in the generated OpenAPI spec automatically. By
default each one contributes its method, path (with path parameters extracted
from {...} segments), Summary, Tags, and Deprecated flag:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Summary: "Cancel an order",
Tags: []string{"Orders"},
Handler: cancelOrder,
})
For request/response bodies, query parameters, and security, fill in the
optional OpenAPI block. Its most useful feature is schema inference: point
RequestSchema / ResponseSchema at a Go struct tagged with the same json
and mfx tags you already use on models, and maniflex reflects it into a JSON
schema — no hand-written OpenAPI types:
type RescheduleReq struct {
NewTime string `json:"new_time" mfx:"required"`
Reason string `json:"reason"`
}
type RescheduleResp struct {
ID string `json:"id"`
Status string `json:"status" mfx:"enum:scheduled|cancelled"`
}
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/appointments/{id}/reschedule",
Summary: "Reschedule an appointment",
Handler: reschedule,
OpenAPI: maniflex.ActionOpenAPI{
Description: "Moves an appointment to a new time.",
RequestSchema: RescheduleReq{},
ResponseSchema: RescheduleResp{},
ResponseStatus: http.StatusOK, // status the response schema documents; defaults to 200
QueryParams: []maniflex.OASParameter{{
Name: "notify", In: "query",
Schema: &maniflex.OASSchema{Type: "boolean"},
}},
Security: []map[string][]string{{"bearerAuth": {}}},
},
})
The reflected schemas honour the field tags you already use on models —
required, enum, min, max, readonly, writeonly — and skip hidden
fields. RequestSchema and ResponseSchema each accept a struct value, a
pointer, or a reflect.Type.
Security names a scheme you register separately with
openapi.AddSecurityScheme.
If you’d rather build the OpenAPI types by hand, set RequestBody and
Responses directly on the ActionConfig — those take precedence over the
inferred schemas when both are present.
Serving a model’s own path from an action
An action and a model cannot both own the same method + path. Registering an
action at a path a model already owns (e.g. GET /threads when a model’s table
is threads) is rejected at startup with a clear panic, rather than letting
the router silently mount two handlers and serve whichever one it happens to
match first:
panic: maniflex: action GET /threads would shadow the auto-generated collection route for model "Thread"
The check covers every route a model mounts, not just the five CRUD ones: the
export and aggregate endpoints, per-field attachment paths
(GET /{model}/{id}/{field}), and a singleton’s GET / PATCH on its bare path.
A conflict is reported when the action resolves the same path and method as a
model route — a path parameter’s name is irrelevant (GET /threads/{threadId}
collides with the read route just as GET /threads/{id} does), while a method the
model does not serve at that path is free to take (POST /threads/{id} is fine —
the item route has no POST).
When you want to serve a model’s collection path yourself — returning a custom shape, or composing several models — mark the model headless so it mounts no REST routes at all, freeing its path for the action:
server.MustRegister(Thread{}, maniflex.ModelConfig{Headless: true})
server.Action(maniflex.ActionConfig{
Method: "GET",
Path: "/threads", // no collision: Thread mounts no routes
Handler: listThreads,
})
A headless model is still registered in full — it migrates, participates in
relations, and is reachable through ctx.GetModel("Thread") and typed CRUD — it
simply has no auto-generated HTTP surface (and no auto-generated OpenAPI paths;
its schema is still emitted for $refs). Use it whenever the generated CRUD
shape doesn’t match the contract you want to expose for that resource.
When to use an action
| Need | Use |
|---|---|
| Standard CRUD | The generated routes |
One-off state transitions (/cancel, /publish) | Action |
| Aggregations and reports | Action, or Raw Queries & Query Models |
| Bulk operations | Batch Operations & Sagas |
| Background processing | Events & Background Jobs |
| Reaching the pipeline from Go, rather than exposing Go to HTTP | Execute |
An action and Execute are mirror images: an action lets HTTP reach your logic,
and Execute lets your logic reach the pipeline — with a typed
principal instead of a header, and your transaction instead of N of them. If you
have ever been tempted to have your server make an HTTP request to itself, that is
the tool.
Reserve actions for endpoints that genuinely don’t fit CRUD. Resist the temptation to use them as a general-purpose handler API — the framework’s strength is in the generated routes; every action is one more thing to test and document by hand.
In-Process Invocation (Execute)
server.Action lets HTTP reach your logic. server.Execute is the other half: it
lets your logic reach the pipeline.
res, err := srv.Execute(ctx.Ctx, maniflex.Invocation{
Model: "Item",
Operation: maniflex.OpUpdate,
ID: itemID,
Body: map[string]any{"status": "approved"},
Auth: &approver, // a typed principal, not a header
Tx: ctx.Tx, // joins the caller's transaction
})
Every step runs — Auth, Deserialize, Validate, Service, DB, Response — in the same order, through the same middleware, as the equivalent request from a client. The result is the same envelope that request would have produced.
Why it exists
Without it, HTTP is the only way in, and code that needs to run a model operation from Go has to make a request to itself:
// Don't do this.
req, _ := http.NewRequest("PATCH", path, body)
req.Header.Set("X-Replay", "1")
req.Header.Set("X-Requester-ID", requesterID)
router.ServeHTTP(httptest.NewRecorder(), req)
Both defects that follow are structural, not careless:
- A principal passed as a header is a principal any client can send. The gates then grow bypasses keyed on that header, and those bypasses are reachable from the internet by construction. No amount of care fixes it at that layer — you cannot pass an identity to a router except through the request.
- N requests cannot be one transaction. A loop that replays N items over HTTP commits each separately, so a failure at item 3 leaves 1 and 2 written while the batch is marked unfinished — and re-running it re-applies the prefix.
Invocation.Auth is a typed *AuthInfo, so there is no header to forge and no
bypass to grow. Invocation.Tx is your transaction, so N invocations commit or
roll back together.
Maker–checker, atomically
The use case this was built for: a staged write, captured earlier, executed on approval as one unit.
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() // no-op once committed
for _, item := range request.Items {
if _, err := srv.Execute(ctx.Ctx, maniflex.Invocation{
Model: item.Model,
Operation: maniflex.OpUpdate,
ID: item.ResourceID,
Body: item.Payload, // the bytes captured at request time
Auth: &requester, // whoever the write is attributed to
Tx: tx,
}); err != nil {
return err // the whole batch rolls back — no committed prefix
}
}
return tx.Commit()
A non-2xx answer comes back as an error, not a value, which is what makes that
loop correct. if err != nil { return err } is the natural Go loop, and it has to
be the one that rolls back — handing a 422 back as (res, nil) would make the
naive loop commit items 1 and 2. Inspect the status with errors.As:
var execErr *maniflex.ExecuteError
if errors.As(err, &execErr) && execErr.StatusCode == http.StatusNotFound {
// the record is gone
}
The *APIResponse is returned either way, so you can read Data and Meta on a
success and the error envelope on a failure.
Authentication: ctx.InProcess()
An in-process call has no Authorization header, because there is no client. The
shipped auth.JWTAuth and auth.JWKSAuth stand aside when — and only when — the
request came from Execute and already carries a principal:
if ctx.InProcess() && ctx.Auth != nil {
return next() // identity already established, by trusted code
}
Everything else on the Auth step still runs against that principal: RequireRole,
RequireOwner, ABAC policies, your own middleware. An Execute with no Auth is
anonymous and is refused exactly as an anonymous request is — being internal is
not being authenticated.
ctx.InProcess() reads an unexported field that only Execute sets, so no client
and no middleware can claim it. Over HTTP it is always false, and the JWT header
path behaves bit-for-bit as it always has. Use it in your own middleware where a
check is about the transport rather than the caller:
server.Pipeline.Auth.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if ctx.InProcess() { return next() } // no browser to protect from CSRF
return csrfCheck(ctx, next)
}, maniflex.WithName("csrf"))
That is also the supported way to exempt a middleware from in-process calls. There
is deliberately no SkipMiddlewares list on Invocation: a named list of security
checks to switch off is the shape of the very bypass Execute exists to delete,
and middleware names are labels for trace logs, not identifiers (six shipped
middlewares share the name otel.span). Writing the exemption at the registration
site keeps it type-checked, greppable, and rename-proof.
No step can be skipped
There is no Steps bitmask. Execute runs the whole pipeline, so it cannot become
a quieter way into the database than the HTTP route it mirrors:
- Validate runs, so
mfxrules —required,enum,readonly,immutable— bind anExecuteexactly as they bind a request. AnExecuteis not a mass-assignment hole. - Auth runs, so authorisation registered there cannot be walked around by
naming yourself in
Invocation.Auth. - Service runs, so your business hooks fire.
Skipping Auth in particular would remove the isolation the injected principal is meant to be constrained by — which is the vulnerability, not the fix.
Invocation
| Field | Meaning |
|---|---|
Model | Registered model’s struct name, e.g. "Order". Required. |
Operation | OpCreate, OpRead, OpUpdate, OpDelete, OpList. Required. |
ID | Primary key. Required for read, update, delete. |
Body | Request body for create/update. []byte/json.RawMessage/string verbatim; anything else is marshalled. |
Query | url.Values carrying filters, sort, include, pagination. |
Auth | The principal. Typed, unforgeable, not a header. |
Tx | Transaction to join. The caller owns it — Execute neither commits nor rolls back. |
Header | Request headers for middleware that reads them (If-Match, Accept-Language). Not Authorization — use Auth. |
An update is a PATCH, not a replace. A field absent from Body is left alone,
exactly as over HTTP — so send a map when you mean to patch some columns:
Body: map[string]any{"status": "approved"}, // every other column untouched
This is not a re-implementation of presence semantics; Execute synthesises the
request and the real Deserialize step binds the body, so the rule is the one the
HTTP path has, not a second copy of it that agrees today and drifts later.
Refused operations. OpExport and OpReadAttachment stream bytes to a
response writer rather than producing a value, so there is nothing to return.
OpAction and OpSearch run trimmed pipelines of their own — call an action’s
handler directly, or ctx.Search.
Registration window. Execute builds the router on first use, exactly as
Handler() does, so Pipeline.Register and Server.Action must come first.
When to use it
| Need | Use |
|---|---|
| Serve a client | The generated routes |
| Run a model operation from Go, with auth and validation | Execute |
| Read or write from Go, without the pipeline | Typed CRUD / Model Accessor |
| Several operations, one transaction | Execute with a shared Tx, or Batch |
Reach for typed CRUD or ctx.GetModel when you want the database. Reach for
Execute when you want the rules — the same auth, validation, hooks, scoping
and response an HTTP caller would get, minus the socket.
Raw Queries & Query Models
The generated CRUD routes cover one-table reads. Anything that needs joins, aggregates, or custom SQL goes through one of the framework’s escape hatches: raw queries from middleware, or query models — read-only models backed by a hand-written SELECT.
Raw queries from middleware
ctx.RawQuery and ctx.RawExec run parameterised SQL through the active
transaction or the bare adapter:
rows, err := ctx.RawQuery(
`SELECT status, COUNT(*) AS n
FROM orders
WHERE organization_id = ?
GROUP BY status`,
ctx.Auth.TenantID,
)
rows is a []map[string]any with column-name keys. Placeholders are rebound to
the adapter’s dialect, so ? works on both SQLite and Postgres ($1, $2).
Never interpolate values into the query string — that’s a SQL injection.
ctx.RawExec is the same shape for non-SELECT statements and returns the
number of rows affected. ctx.RawQuery also returns the rows from a
data-modifying statement with a RETURNING clause (e.g. UPDATE … RETURNING id).
When ctx.Tx is non-nil, both methods participate in the active transaction
automatically. The built-in SQLite and Postgres adapters support this; if a
custom adapter’s transaction cannot run raw SQL, the call fails with
maniflex.ErrRawNotSupportedInTx rather than quietly running on a different
connection outside the transaction — where the write would commit on its own and
survive the rollback.
Portability pitfalls
Hand-written SQL runs on both SQLite and Postgres, which differ in ways the ORM normally hides:
- Parameterise booleans.
WHERE active = 1works on SQLite but errors on Postgres (operator does not exist: boolean = integer). Bind a Goboolinstead:WHERE active = ?,true. - Know your column names. A column’s name comes from the field’s
dbtag, else itsjsontag, else the snake-cased field name. A camelCasejsontag (json:"orderId") produces a camelCase column (orderId) — and Postgres folds unquoted identifiers in hand-written SQL to lowercase, soorderIdsilently won’t match. Keep raw SQL to snake_case columns, or set an explicitdb:"snake_case". SQLite is case-insensitive, so this only bites on Postgres. - Pin table names. Physical table names are pluralised implicitly
(
VisitorDay→visitor_days). When you reference tables in raw SQL, setModelConfig.TableNameso the name can’t drift from what your SQL expects.
Structured aggregation: ctx.Aggregate
For typed, validated aggregations there’s a structured builder that doesn’t require hand-written SQL:
rows, err := ctx.Aggregate("Order", maniflex.AggregateQuery{
Select: []maniflex.AggregateField{
{Op: maniflex.AggCount, As: "n"},
{Op: maniflex.AggSum, Field: "total", As: "revenue"},
},
GroupBy: []string{"status"},
Where: []*maniflex.FilterExpr{
{Field: "created_at", Operator: maniflex.OpGte, Value: "2026-01-01"},
},
Having: []maniflex.HavingClause{
{Alias: "revenue", Operator: maniflex.OpGt, Value: 1000},
},
OrderBy: []maniflex.SortExpr{{DBName: "revenue", Direction: maniflex.SortDesc}},
Limit: 100,
})
Each AggregateField.Op is one of AggCount, AggCountDistinct, AggSum,
AggAvg, AggMin, AggMax. Leave Field empty on AggCount to mean
COUNT(*). As overrides the alias used in the result row and in Having
or OrderBy; if omitted the default is <op>_<field> (or count for
COUNT(*)).
All DB column names — in Select.Field, GroupBy, and Where.Field — are
validated against the registered model. A typo fails fast with a clear
error rather than emitting bad SQL. OrderBy.DBName may reference either
an aggregate alias or a GroupBy column. Nested-relation filters are not
yet supported in Aggregate — use the raw-query escape hatch when you need
them.
When ctx.Tx is active the aggregate participates in the transaction,
matching RawQuery/QueryModel.
To keep an aggregate on a parent column rather than compute it per request —
Order.PaidAmount maintained as SUM(OrderPayment.amount) — see
Maintained Rollups, the write-side counterpart of ctx.Aggregate.
Auto-generated aggregate endpoint
Opt a model into a built-in HTTP aggregation route with
ModelConfig.AggregateEnabled:
server.MustRegister(Order{}, maniflex.ModelConfig{AggregateEnabled: true})
This mounts GET /:model/aggregate. The aggregation is described by a JSON
document passed URL-encoded in the ?aggregate= query parameter, and the
group rows come back under the usual {"data": [...]} envelope:
GET /api/orders/aggregate?aggregate=<url-encoded JSON>
# where the JSON is:
{
"select": [{"op": "count", "as": "n"}, {"op": "sum", "field": "amount", "as": "total"}],
"group_by": ["status"],
"where": [{"field": "created_at", "operator": "gte", "value": "2026-01-01"}],
"having": [{"alias": "total", "operator": "gt", "value": 1000}],
"order_by": [{"field": "total", "direction": "desc"}],
"limit": 100
}
const spec = {
select: [{ op: "sum", field: "amount", as: "total" }],
group_by: ["status"],
};
const res = await fetch(
`/api/orders/aggregate?aggregate=${encodeURIComponent(JSON.stringify(spec))}`,
);
The spec travels in the query string, not in a request body, because this is a
GET: a body on a GET is dropped by many proxies and CDNs and cannot be sent
by fetch() at all, so an endpoint that needed one worked in development and
failed in production. A request body is not read; sending one gets a
400 INVALID_AGGREGATE pointing you at ?aggregate=.
op is one of count, count_distinct, sum, avg, min, max (omit
field on count for COUNT(*)). Field names use the same convention as
?filter=/?sort= — the JSON name (DB column name also accepted) — and every
referenced field must be mfx:"filterable" or mfx:"sortable", so the public
endpoint can never aggregate a hidden or sensitive column. WHERE operators are
the flat comparison set plus
in/not_in/like/ilike/contains/starts_with/ends_with/is_null/not_null
(no between).
The endpoint runs as the list operation: any auth or tenancy middleware you
registered for OpList applies unchanged (no separate registration needed), and
request ?filter= conditions — including middleware-injected tenancy
force-filters — are AND-ed into the aggregate WHERE alongside the spec’s own
where.
Tree traversal: ctx.RecursiveQuery
For self-referential models — categories, org charts, threaded comments, bill of
materials — ctx.RecursiveQuery issues a WITH RECURSIVE CTE without hand-writing
SQL:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: "some-uuid",
ParentField: "parent_id",
MaxDepth: 5,
})
// rows[0]["_depth"] == int64(0) is the root; rows[1..n] are descendants.
Every returned row is a map[string]any with all the model’s columns plus a
synthesised _depth integer (0 = the root node). Rows are ordered by _depth
ascending.
Fields
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
RootID | string | yes | — | Primary key of the starting node |
ParentField | string | yes | — | DB column that holds the parent’s ID, e.g. "parent_id" |
Direction | RecursiveDirection | no | RecursiveDescendants | Walk downward (RecursiveDescendants) or upward (RecursiveAncestors) |
MaxDepth | int | no | 0 → DefaultRecursiveMaxDepth (100) | Stop after this many levels; negative means unlimited |
Where | []*FilterExpr | no | nil | Additional filters applied in both the anchor and recursive members |
Descendant vs. ancestor traversal
Descendants (default) — walks down the tree. Given a root category it returns all children, grandchildren, etc.:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: rootID,
ParentField: "parent_id",
// Direction defaults to RecursiveDescendants
})
Ancestors — walks up the tree. Starting from a leaf, it returns the node itself, its parent, grandparent, and so on up to the root:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: leafID,
ParentField: "parent_id",
Direction: maniflex.RecursiveAncestors,
})
Limiting depth
MaxDepth: 1 returns the root plus its immediate children only — no further
descendants:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: rootID,
ParentField: "parent_id",
MaxDepth: 1, // depth 0 (root) + depth 1 (children)
})
Left at its zero value, MaxDepth applies maniflex.DefaultRecursiveMaxDepth
(100) rather than running unbounded. Pass a negative value for a genuinely
unlimited traversal:
MaxDepth: -1, // the whole hierarchy, however deep
Before v0.2.5,
0meant unlimited. If you relied on that for a hierarchy deeper than 100 levels, setMaxDepth: -1explicitly.
Cyclic data
A parent chain that loops — a row that is its own ancestor, whether directly
(parent_id pointing at itself) or through a chain — is not an error. The
traversal tracks the ids it has visited and stops at the repeat, so each node is
returned once:
A.parent_id = B, B.parent_id = A, root = A
→ A (_depth 0), B (_depth 1)
This holds in both directions. It matters because such data is not exotic: a category tree with no application-level guard against re-parenting a node under its own descendant will produce one eventually, and before v0.2.5 a single such request looped until it exhausted the process.
Note that a cycle is the only pathology here. Because the traversal follows one
scalar ParentField, every node has exactly one parent and so exactly one path
from the root — the same node cannot be reached twice by different routes.
Filtering nodes
Where filters are applied in both the anchor and recursive members, so a node
that fails the filter is excluded regardless of depth, and the traversal does
not continue through it:
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{
RootID: rootID,
ParentField: "parent_id",
Where: []*maniflex.FilterExpr{
{Field: "status", Operator: maniflex.OpEq, Value: "active"},
},
})
Nested-relation filters are not supported in RecursiveQuery — use
ctx.RawQuery for those cases.
Soft-delete awareness
When a model uses WithDeletedAt or a boolean soft-delete field, the recursive
query automatically excludes deleted records from both the anchor and recursive
members. No extra filter is needed.
Transaction participation
RecursiveQuery participates in ctx.Tx exactly like RawQuery:
tx, _ := ctx.BeginTx(ctx.Ctx, nil)
ctx.Tx = tx
defer tx.Rollback()
rows, err := ctx.RecursiveQuery("Category", maniflex.RecursiveQuery{...})
tx.Commit()
Database support
Both Postgres ($N placeholders) and SQLite (since 3.8.3, ? placeholders)
are handled transparently.
Stable read-only endpoints for aggregates
maniflex has no SQL-backed “query model” — a struct cannot be registered with a SQL body. For a stable, repeatable read endpoint over computed data you have two real building blocks:
- The auto-generated aggregate endpoint. Every model already exposes
GET /{model}/aggregate(see Auto-generated aggregate endpoint), driven byctx.Aggregate. Grouping, counts, sums/averages, and the standard?filter=all apply, and it is in the OpenAPI spec — reach for it first for counts/sums/averages over a registered model. - A custom action running raw SQL. For a shape the aggregate endpoint cannot
express (a multi-table join, a window function), mount a
custom action whose handler runs
ctx.RawQueryand returns the rows; you own filtering and pagination inside the handler. On Postgres, back an expensive aggregate with a materialised view maintained in your migrations andSELECTfrom it in the handler.
When to use which
| Need | Tool |
|---|---|
| One-off aggregate inside an action or middleware | ctx.RawQuery |
| Counts / sums / averages as a stable, filterable endpoint | GET /{model}/aggregate (ctx.Aggregate) |
| A bespoke read endpoint (joins, window functions) | custom action + ctx.RawQuery |
| Tree traversal (descendants, ancestors, depth limit) | ctx.RecursiveQuery |
| Bulk mutation inside a single request | ctx.RawExec (inside a transaction) |
| Per-row business logic across many rows | Batch Operations & Sagas |
Performance notes
- Raw queries do not cache; each request executes the SQL. For a frequently-hit
aggregate exposed through a custom action, wrap it with
response.Cache(see Response Middleware) or maintain a summary table (the write-side Maintained Rollups are built for this). - Avoid unbounded scans — add
WHEREandLIMITclauses to any hand-written SQL when the underlying table is large. - For Postgres, a materialised view often beats recomputing an expensive aggregate per request; refresh it on a schedule and read it from the handler.
Maintained Rollups
A rollup is a denormalised aggregate column on a parent that the framework keeps
in step with its children. Order.PaidAmount as SUM(OrderPayment.amount),
StoreSite.ReviewsCount as COUNT(Review) — the columns an app would otherwise
recompute by hand in every write path, and which drift the moment one path
forgets.
Where ctx.Aggregate
computes an aggregate on read, a Rollup maintains one on write.
srv := maniflex.New(cfg)
srv.MustRegister(Order{}, OrderPayment{})
// Every OrderPayment write recomputes its Order's paid_amount.
srv.MustRegisterRollup(maniflex.Rollup{
Parent: "Order", ParentField: "paid_amount", Op: maniflex.AggSum,
Child: "OrderPayment", ChildField: "amount", On: "order_id",
})
// A rollup needs the child write to be transactional.
srv.Pipeline.Service.Register(
maniflex.WithTransaction(nil),
maniflex.ForModel("OrderPayment"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Configuration
| Field | Meaning |
|---|---|
Parent | model carrying the denormalised column |
ParentField | JSON name of that column |
Op | AggSum, AggCount, AggAvg, AggMin, or AggMax |
Child | model whose rows are aggregated |
ChildField | JSON name of the aggregated column (omit only for AggCount) |
On | JSON name of the foreign key on Child pointing to Parent’s id |
Field names are resolved and validated at registration — a typo is a startup
error naming the field, not a silently drifted total. This is the whole reason
the config is a typed struct and not a mfx:"rollup:sum(...)" tag: a tag would
be a mini query-language inside a string, invisible to go vet and failing in
the worst possible way.
RegisterRollup returns an error; MustRegisterRollup panics. Both must be
called before Start()/Handler().
How it stays correct
On every create, update or delete of a child, the affected parent is
recomputed from scratch — Op(ChildField) over that parent’s live children —
and written to the parent column, inside the child write’s transaction.
Recomputing rather than applying a +delta/-delta is what makes it correct by
construction:
- Delete — the parent recomputes without the deleted row.
- Re-parenting — a PATCH that changes the child’s foreign key recomputes both the old and the new parent.
- Soft delete — a soft-deleted child is excluded, matching what a fresh aggregate returns.
- No drift — the column is always exactly the aggregate of the rows it summarises, never an accumulator that can diverge.
Empty sets follow SQL: a sum or count of no children is 0; a min/max/avg of no
children is null.
Transactions are required
A rollup refuses a child write that is not in a transaction, with
500 ROLLUP_NO_TX. Without one, a child insert could commit while the parent
update fails — the exact drift the rollup exists to prevent. Register
maniflex.WithTransaction on the Service step for the child’s writes, as above.
This follows the same fail-loud rule as mfx:"lock_scope".
Backfilling
Adding a rollup to a table that already has children, or reconciling a column edited out of band, needs a one-time recompute:
if err := srv.BackfillRollups(context.Background()); err != nil {
log.Fatal(err)
}
BackfillRollups recomputes every registered rollup for every parent from the
current child rows. It reconciles rather than locks — each parent is recomputed
independently, and a concurrent live write is simply picked up by its own rollup
— so prefer a quiet window for very large tables.
Cost and limits
- Each child write costs one extra aggregate query plus one parent update, inside the transaction. Concurrent children of the same parent serialise on that parent’s row, which is what keeps the total consistent.
- The rollup fires on the generated CRUD routes and any write that runs the DB
step. A write that bypasses the pipeline (a raw
INSERT, a direct adapter call) does not trigger it — runBackfillRollupsafter such a bulk load. AggCountDistinctis not supported as a rollup op; write it by hand with an After-DB middleware if you need it.
Model Accessor
The generated REST routes expose a model over HTTP. Application code (action handlers, middlewares, event listeners, etc.) frequently needs to read or write a different model than the one the request targets. The model accessor is the in-process door for that work.
ctx.GetModel(name) returns an object exposing the five standard CRUD
operations bound to one registered model. Every call routes through the active
transaction, respects per-model database overrides, and returns
map[string]any rows without the HTTP round-trip. These are the same underlying
rows the REST layer reads, but keyed by DB column name and not run through
response marshalling — so hidden / writeonly columns are present and the
result is not the REST envelope shape. The main difference is the interaction
with middlewares, more on that below.
row, err := ctx.GetModel("User").Read(userID)
if err != nil {
return err
}
name := row["name"].(string)
The ModelAccessor interface
ctx.GetModel(name) yields a *ModelAccessor. All methods use
map[string]any, keyed by DB column name (each field’s mfx DB name), not the
JSON name — see Relationship to pipeline middleware
below.
| Method | Signature | Returns |
|---|---|---|
List | List(q *QueryParams) ([]map[string]any, error) | A page of rows |
Read | Read(id string) (map[string]any, error) | One row, or ErrNotFound |
Create | Create(data map[string]any) (map[string]any, error) | The stored row |
Update | Update(id string, data map[string]any) (map[string]any, error) | The updated row |
Delete | Delete(id string) error | — |
// List - q may be nil (page 1, limit 20, no filters or sorts).
admins, err := ctx.GetModel("User").List(&maniflex.QueryParams{
Filters: []*maniflex.FilterExpr{
{Field: "role", Operator: maniflex.OpEq, Value: "admin"},
},
Page: 1,
Limit: 100,
})
// Create - returns the stored representation, id and defaults populated.
created, err := ctx.GetModel("User").Create(map[string]any{
"name": "Carol",
"email": "[email protected]",
"role": "viewer",
"password": "secret",
})
// Update - a partial patch; only the supplied keys are written.
updated, err := ctx.GetModel("User").Update(userID, map[string]any{
"role": "admin",
})
// Delete - soft-deletes when the model opts into soft delete, else hard-deletes.
err := ctx.GetModel("User").Delete(userID)
List accepts the same QueryParams the query
parser builds from ?filter= / ?sort= / ?page=; filters, sorts, and
pagination all apply. Read, Update, and Delete return
maniflex.ErrNotFound when the id is absent. Create and Update return a
*maniflex.ErrConstraint on unique or check violations, allowing DB errors to
be mapped to HTTP responses the same way the pipeline does (see
Error Handling).
Errors surface on first use
GetModel never returns an error directly. When the name is not registered — or
the registry is not wired onto the context — it returns an error accessor that
surfaces the failure on the first method call rather than at construction:
_, err := ctx.GetModel("NoSuchModel").List(nil)
// err: maniflex: model "NoSuchModel" is not registered
This keeps call sites terse (ctx.GetModel("User").Read(id)) without a nil check
on the accessor; the error is handled at the method call as usual.
Transactions
Accessor operations route through ctx.Tx whenever a transaction is active, so
work performed through the accessor commits or rolls back atomically with the
rest of the request:
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
ctx.Tx = tx
defer tx.Rollback() // no-op after a successful Commit
if _, err := ctx.GetModel("Order").Create(order); err != nil {
return err // the deferred Rollback undoes everything
}
if _, err := ctx.GetModel("Inventory").Update(sku, dec); err != nil {
return err
}
return tx.Commit()
The accessor must be obtained after
ctx.Txis set. An accessor captures whateverctx.Txholds at the momentGetModelis called. Obtaining it beforeBeginTxleaves its writes outside the transaction — and under SQLite they can deadlock against the open tx. Always callctx.GetModel(...)afterctx.Tx = tx:ctx.Tx = tx orders := ctx.GetModel("Order") // bound to ctx.Tx — safe
Per-model database routing
When a model is pinned to a non-default adapter (see Database Backends), the accessor resolves that model’s own adapter, so a cross-model read reaches the correct database automatically. The request transaction, however, belongs to the request’s adapter. When the target model lives on a different adapter, the accessor cannot enlist that transaction and instead runs the operation outside it against the correct adapter. Within a single adapter, transactional routing behaves exactly as above.
Relationship to pipeline middleware
The accessor talks directly to the database adapter (through the active
transaction when one is set). It does not run the request pipeline. Response
transforms, dynamic redaction, field-visibility rules, and any other registered
middleware never observe an accessor read or write — they operate on
ctx.Response/ctx.Body, which the accessor does not populate.
The one exception is the model’s own value constraints. enum, min and
max are checked on accessor and typed writes just as they are on an HTTP
request, and a violation returns an error before anything reaches the database:
_, err := ctx.GetModel("Account").Create(map[string]any{"role": "superuser"})
// err: maniflex: Account: field "role" must be one of: [user admin]
Pass maniflex.SkipValidation() where the violation is deliberate — backfilling
rows that predate an enum, say:
_, err := ctx.GetModel("Account").Create(legacyRow, maniflex.SkipValidation())
readonly and immutable are not applied here, deliberately. Those tags
say a value may not come from a client, and the accessor is not a client —
stamping such a column from a background job is much of why these APIs exist.
The split is between data integrity, which is a bug whoever writes it, and
access control, which depends on who is asking.
Before v0.2.5 no constraint was checked on these paths, so the same model accepted a value in Go that it answered
422for over HTTP.
This is not a substitute for validating untrusted input. Value constraints
are the model’s own rules, not an authorization boundary: decoding a request
body into a struct and handing it to maniflex.Create still writes whatever
fields it contains, including readonly ones. Untrusted bodies belong on the
HTTP path, or must be checked before they reach here.
The practical consequence: an accessor returns the record as stored, before any response-layer shaping. Given a redacting transform on the Response stage —
s.Pipeline.Response.Register(
response.TransformField("secret", func(any) any { return "[redacted]" }),
maniflex.ForModel("Account"),
maniflex.AtPosition(maniflex.After),
)
— a REST read of Account returns "secret": "[redacted]", because the
transform rewrites ctx.Response.Data after the DB step. An accessor read does
not:
row, _ := ctx.GetModel("Account").Read(id)
row["secret"] // the raw stored value — NOT "[redacted]"
Two further differences follow from bypassing the response marshaller:
- Keys are DB column names. Accessor maps are keyed by each field’s
mfxDB name, not the JSON name that response middleware and the API envelope use. Where the two names differ, the DB name is the one present in the map. hiddenandwriteonlyfields are included. The response marshaller stripsmfx:"hidden"andmfx:"writeonly"columns from every payload; the accessor does not. Their raw values are present in accessor maps.
This is by design — the accessor is trusted, in-process access for application logic, not a client-facing surface. It does mean that sanitisation implemented in response middleware (redaction, field hiding, CDN rewriting) is not inherited by data read through the accessor. When accessor output is forwarded outside the process, that shaping is the caller’s responsibility.
QueryModel is the former name for List
ctx.QueryModel(name, q) remains available and now delegates to
ctx.GetModel(name).List(q). The accessor is preferred: it exposes Read,
Create, Update, and Delete on the same object rather than reading alone.
rows, err := ctx.QueryModel("User", nil) // deprecated
rows, err := ctx.GetModel("User").List(nil) // preferred
Typed accessor: map[string]any → *T
The string-named accessor is dynamic — suited to cases where the model name is data, or where maps are the natural representation. When the type is known at compile time, the typed CRUD free functions provide the same five operations against concrete structs. They resolve the model from the type parameter, so no name string must be kept in sync:
users, err := maniflex.List[User](ctx, nil) // []*User
u, err := maniflex.Read[User](ctx, id) // *User
created, err := maniflex.Create(ctx, &User{Name: "Jane"}) // *User
updated, err := maniflex.Update(ctx, id, &User{Name: "J."}) // *User
err := maniflex.Delete[User](ctx, id)
These route through ctx.Tx, honour per-model adapters, and return the same
ErrNotFound / ErrConstraint errors as the string-named accessor — they are
its typed counterpart, not a separate code path.
One difference is significant: maniflex.Update[T] performs a full-record
update. Every writable column except id is written from the supplied struct,
so any zero-valued field overwrites the stored value. For a partial patch — only
the specified keys — use the map-based ctx.GetModel(name).Update(id, data)
instead.
Columns the model marks mfx:"readonly" or mfx:"immutable" are not among
them, and neither are id, created_at and updated_at. Those tags used to be
enforced only by the Validate step, which the typed helpers do not run, so a
caller building a fresh struct to change one field stamped the zero time over the
row’s real created_at and blanked every readonly column along with it. Note
this is a guard against accidental blanking on a full-struct update, not an
access rule: maniflex.Create writes readonly columns from the struct, and
ctx.GetModel(name).Update writes whatever key you name. To write such a column
deliberately, name it:
ctx.GetModel("User").Update(id, map[string]any{"origin": "import"})
An explicit key is a different statement of intent from a struct that happens to carry a zero value — which is the whole distinction the two accessors draw.
Choosing between the accessors
| Requirement | Accessor |
|---|---|
| Dynamic access where the model name is a variable | ctx.GetModel(name) |
| Concrete structs and compile-time field names | maniflex.List[T] / Read[T] / … |
| A partial patch (only some fields) | ctx.GetModel(name).Update(id, data) |
| A full-record replace from a struct | maniflex.Update[T](ctx, id, &rec) |
| Joins, aggregates, custom SQL | Raw Queries & Aggregates |
Both accessors serve in-process cross-model work inside a request. Exposing a model over HTTP is a matter of registering it and letting the generated routes handle it; the accessor is the tool a handler reaches for once execution is already inside the pipeline.
Batch Operations & Sagas
The generated REST routes work on one row at a time. When the work fans out — inserting hundreds of rows from a CSV, fulfilling an order across inventory, payment, and shipping — two patterns appear: batch for atomic same-table work, and saga for multi-step workflows that span services.
Batch inside a single transaction
The simplest “bulk write” is a single transaction that issues many inserts or updates. Use an action endpoint so the request does not pass through the per-row Validate/Service hooks:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/users/import",
Handler: importUsers,
})
func importUsers(ctx *maniflex.ServerContext) error {
var req struct {
Users []map[string]any `json:"users"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
users := ctx.GetModel("User")
inserted := 0
for _, row := range req.Users {
if _, err := users.Create(row); err != nil {
ctx.Abort(http.StatusConflict, "IMPORT_FAILED",
fmt.Sprintf("row %d: %s", inserted, err.Error()))
return nil
}
inserted++
}
if err := tx.Commit(); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusCreated,
Data: map[string]any{"inserted": inserted},
}
return nil
}
Either every row commits or none does. Validation still runs because
ctx.GetModel(...).Create goes through the adapter — but per-row middleware
on the Service or Validate steps does not, since this is an action.
The maniflex.Batch helper
The hand-rolled BeginTx / ctx.Tx / defer Rollback dance above is
canonicalised by maniflex.Batch(ctx, func(*maniflex.Batcher) error)
(batch.go). It opens a transaction (or joins ctx.Tx if one is already set),
points ctx.Tx / ctx.Ctx at it for the duration of the callback, commits on
success, and rolls back on any returned error or ctx.Abort. The *Batcher
exposes the same five CRUD operations, all enlisted in the shared transaction:
err := maniflex.Batch(ctx, func(b *maniflex.Batcher) error {
inv, err := b.Create("Invoice", invoiceData)
if err != nil {
return err
}
for _, line := range lines {
line["invoice_id"] = inv["id"]
if _, err := b.Create("InvoiceLine", line); err != nil {
return err // rolls the whole batch back
}
}
return nil
})
Prefer maniflex.Batch over the manual transaction plumbing shown above — it
gets the rollback, abort, and ctx.Tx restoration semantics right. A single
batch transaction cannot span adapters; use a saga for cross-database work.
For larger imports, batch the inserts (INSERT … VALUES (…), (…), … via
ctx.RawExec) and commit every N rows.
Cross-service workflows: sagas
When a workflow touches more than one downstream — charge a payment provider, reserve inventory, notify a partner — a single database transaction is no longer enough. The standard pattern is a saga: a sequence of forward steps, each with a compensating undo step.
maniflex ships a lightweight saga coordinator in pkg/saga:
saga.New(name).Step(name, do, undo).Execute(ctx, state) runs the forward
steps in order and, on any failure, runs the compensating undo functions in
reverse order.
err := saga.New("dispense_charge").
Step("create_invoice", createInvoice, voidInvoice).
Step("debit_ar_ledger", debitAR, reverseAR).
Step("deduct_stock", deductStock, restock).
Execute(ctx, saga.State{"patient_id": pid, "items": items})
It is an in-process, non-crash-safe coordinator: if the process dies
mid-saga, no compensation runs. For durable, crash-safe workflows the
transactional outbox pattern below remains the option to reach for (or wrap the
saga’s Execute inside a pkg/jobs job so a retry re-drives it). The outbox
mechanics fit naturally on the pipeline:
- Start a request transaction with
maniflex.WithTransaction. The local database changes commit or roll back atomically. - Make external calls from the Service step, recording an outbox row in the same transaction for each call that needs a follow-up.
- Process the outbox asynchronously with a background runner (from
jobs/redisor a similar) that performs the external call, marks the outbox row done, and triggers compensation on failure.
server.Pipeline.Service.Register(maniflex.WithTransaction(nil),
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil {
return err
}
// Both writes are in the same transaction as the Order insert.
_, err := ctx.GetModel("OutboxEvent").Create(map[string]any{
"kind": "charge-payment",
"payload": ctx.DBResult,
"status": "pending",
})
return err
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate), maniflex.AtPosition(maniflex.After))
A separate worker reads pending OutboxEvent rows and processes them. If the
payment provider fails, the worker records the failure and enqueues a
compensating action (e.g. “cancel the order”).
This pattern — transactional outbox + asynchronous worker — is the practical alternative to two-phase commit. It costs you one table and one background job but gains durability and isolation that distributed transactions cannot offer.
When to use which
| Workload | Pattern |
|---|---|
| Bulk same-table writes | One transaction, one action endpoint |
| Multi-table writes touching only your database | maniflex.WithTransaction on the request |
| Writes that depend on external systems | Transactional outbox + saga |
| Long-running background work | Background job (see Events & Background Jobs) |
See also
- Events & Background Jobs — running the outbox worker and emitting domain events.
- Transactions —
maniflex.WithTransaction, manualBeginTx, andLockForUpdate. - Custom Endpoints (Actions) — the right place to host a bulk endpoint.
Events & Background Jobs
maniflex offers two complementary mechanisms for work that happens outside the request pipeline: an event bus for lightweight domain-event fan-out, and a job queue for durable, retriable background work.
| Mechanism | When to use |
|---|---|
Event bus (events/*) | Notify other services or modules that something happened. Fire-and-forget. |
Job queue (jobs/*) | Do something reliably after a request — report generation, email, reconciliation. Needs retry and status tracking. |
Event bus
The event bus lets pipeline middleware publish domain events that any number of
subscribers consume independently. An events.Emit call on the DB-After step
publishes user.created, order.placed, etc. to whichever bus is wired up:
import (
"github.com/xaleel/maniflex/events"
"github.com/xaleel/maniflex/events/redis"
)
bus := redis.New(redisClient, "myapp") // prefix namespaces the Redis stream keys
server.Pipeline.DB.Register(
events.Emit(bus),
maniflex.ForModel("Order"),
maniflex.AtPosition(maniflex.After),
)
Publishing under a transaction
Emit never publishes before the write is durable. Which mechanism it uses
depends on the bus:
| Bus | Under WithTransaction |
|---|---|
outbox.Bus (a TxPublisher) | the event row is INSERTed inside the transaction, so event and write commit or roll back together |
| a direct broker bus (redis, kafka, nats, rabbitmq) | the publish is deferred to after the commit, and dropped if the transaction rolls back |
The second row is the weaker guarantee of the two: the commit can succeed and the
broker still be unreachable, and there is no record left to retry from. Use an
outbox.Bus when losing an event is worse than storing one — see
Example 3 for the pattern end to end.
If you register your own side effect from a middleware — a webhook, a cache
invalidation — reach for ctx.AfterCommit rather than firing it inline:
ctx.AfterCommit(func() { go notify(orderID) })
It runs the callback immediately when no transaction is active, so it is safe to use unconditionally. It runs synchronously after the commit, so start a goroutine for anything slow.
Subscribers register a Subscription:
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: func(ctx context.Context, e events.Event) error { /* ... */ return nil },
})
For WebSocket fan-out, connect a realtime.Hub to the bus — see
Realtime / WebSockets.
What the payload carries
Event.Data is the written row, keyed by database column name — not by
json name, and not the response shape. It is deliberately not the response
projection: locale resolution and ctx.RedactResponseField masking are
decisions made for one requesting caller, and an event is durable, replayable,
and read by subscribers who never made that request.
Four kinds of column are stripped before the event leaves:
| Excluded | Why |
|---|---|
mfx:"hidden" | never leaves the server |
mfx:"writeonly" | never read back — password hashes and the like |
mfx:"encrypted" | the row reaching Emit is already decrypted, so emitting it would publish the plaintext |
{field}_hmac | the searchable digest companion of an encrypted+unique column |
This matters because the payload is not a transient in-memory value: it is
persisted verbatim to event_outbox.payload, replayed by the outbox relayer,
pushed to every WebSocket and SSE client through the hub, and written to
whatever the broker retains. An encrypted column emitted in the clear defeats
the at-rest guarantee everywhere downstream at once.
maniflex.RedactRecord(model, row) applies the same exclusion set, if you build
an event by hand (see the custom-action note below) or serialize ctx.DBResult
in your own middleware.
Idempotent delivery
Every broker adapter here is at-least-once, so a handler can see the same event
twice. That is by design and not a rare edge: a consumer that crashes with work
in flight replays it on restart, because the alternative — treating in-flight
work as consumed — loses it. events.Dedupe wraps a handler to suppress the
repeat:
store := events.NewSQLDedupeStore(db, "sqlite") // or NewInMemoryDedupeStore
store.Migrate(ctx)
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: events.Dedupe(store)(myHandler),
})
The ID is claimed before the handler runs, so two workers handed the same event concurrently do not both process it. If the handler then returns an error, the claim is released, so the retry is not mistaken for a duplicate — a transient failure retries normally and only a genuine redelivery is dropped.
Releasing is an optional capability: a custom DedupeStore may also implement
events.DedupeReleaser. Both bundled stores do. A store that does not cannot
undo its claim, so a handler that fails transiently loses the event rather than
retrying it — Dedupe logs a warning naming the store when you wrap one.
A claim outlives a process crash mid-handler: the ID stays recorded and the event is not reprocessed.
InMemoryDedupeStorebounds this with its TTL; forSQLDedupeStore, pruneevent_dedupeon whatever window you can tolerate replaying.
Dead-lettering
Set Subscription.DLQ (or RelayOptions.DLQType on the outbox relayer) to
re-publish an event that exhausted its attempts under a separate type, through
the same broker. Both paths produce the same payload:
ID | a fresh one — the original was already published under its ID, so reusing it gets the dead-letter dropped by any downstream deduper |
Type | the configured DLQ type |
Headers | every original header, plus original_type and original_id |
Everything else is copied unchanged, so the dead-letter carries the same Data,
Model, RecordID and TenantID as the event it came from.
A DLQ publish that itself fails is logged and the event is then lost — the DLQ rides the same broker that was already failing, so a persistent outage takes the dead-letters with it.
Custom actions emit manually.
events.Emitruns on the DB step, which custom actions skip — so aserver.Actionhandler never fires the middleware and must publish to the bus itself:data, _ := json.Marshal(maniflex.RedactRecord(ctx.Model, ctx.DBResult)) err := bus.Publish(ctx.Ctx, events.Event{ Type: "order.cancelled", Model: "Order", RecordID: orderID, Data: data, })Nothing redacts a hand-built payload for you — marshal
ctx.DBResultdirectly and you publish the plaintext of every encrypted column.For the transactional outbox, publish inside the action’s own transaction so the event commits atomically with the write.
Available adapters: events/redis, events/kafka, events/nats,
events/rabbitmq. The in-process adapter (inproc.New() from
github.com/xaleel/maniflex/events/inproc) ships in the core module for tests.
events/rabbitmqdoes not reconnect. It is handed an*amqp.Connectionit does not own, and amqp091-go connections do not self-heal, so a connection or channel drop ends every subscription on it permanently while the process keeps serving. A dead subscription logs an ERROR naming the queue and callsOptions.OnSubscriptionClosed; supervise that callback and rebuild the bus on a fresh connection if consumer downtime matters. ItsPublishwaits for a broker confirm, so a failed publish is reported rather than assumed delivered.
Broker adapters are nested modules. Adapters with heavy dependencies (e.g. NATS) ship as their own Go modules —
go get github.com/xaleel/maniflex/events/nats— so the core module stays dependency-light. Pin each one explicitly ingo.mod.
NATS: one bus binds one JetStream stream.
nats.New(nc, stream)ties a bus to a single stream for both publish and subscribe, and JetStream rejects two streams whose subjects overlap. A service that publishes its own subjects while consuming another service’s subjects needs a stream-ownership decision — either a shared stream, or consume from the owning service’s stream. Create the stream yourself (scoped to your business subjects, not">");Newdoes not create it.
Job queue
The jobs/ packages provide a producer/consumer queue with retries, dead-letter
routing, and optional status persistence through the REST layer.
Adapters
| Package | Backing store | Transactional enqueue | Best for |
|---|---|---|---|
jobs/inproc | goroutine pool | no (best-effort) | tests, single-binary dev |
jobs/sql | Postgres or SQLite | yes — enqueue in the same ctx.Tx | production (recommended) |
jobs/redis | Redis Streams / BRPOP | no | high-throughput fleets |
All three share the same jobs.Queue and jobs.Source interfaces so swapping
adapters is a one-line change.
Defining and enqueueing a job
import (
"database/sql"
"github.com/xaleel/maniflex/jobs"
jobssql "github.com/xaleel/maniflex/jobs/sql"
)
// jobs/sql takes a database/sql handle, not the maniflex DB adapter.
db, _ := sql.Open("sqlite", "./app.db")
queue := jobssql.New(db)
if err := jobssql.Migrate(ctx, db, "sqlite"); err != nil { /* ... */ } // "postgres" on PG
Lanes and secrets
-
Separate lanes: run an isolated queue on its own table so a type-restricted worker can’t interfere with other jobs. Pass
WithTableNameto bothNewandMigrate(indexes are renamed to match, so two queues share one DB):otp := jobssql.New(db, jobssql.WithTableName("otp_jobs")) jobssql.Migrate(ctx, db, "sqlite", jobssql.WithTableName("otp_jobs")) -
Encrypt payloads at rest: payloads are stored as cleartext JSON by default. Pass
WithPayloadCipher(cipher)(anyEncrypt([]byte)/Decrypt([]byte)implementation) to encrypt the payload column; stored values are prefixedencq:and decrypted transparently on dequeue. -
Unhandled types are requeued, not killed: a worker that lacks a handler for a job’s type now requeues it (so another worker can claim it) instead of dead-lettering it — safe for a type-restricted worker sharing a table.
// Inside a pipeline middleware or action handler:
id, err := queue.Enqueue(ctx, jobs.Job{
Type: "send_receipt",
ActorID: ctx.Auth.UserID,
TenantID: ctx.Auth.TenantID,
Payload: json.RawMessage(`{"order_id":"abc"}`),
})
Fields worth knowing:
| Field | Effect |
|---|---|
Type | Selects the handler on the Worker (required) |
MaxRetry | Max attempts before dead. Default 3. |
NotBefore | Delay execution until this time (use EnqueueAt as a shortcut) |
GroupKey | At most one job with this key runs at a time — useful for per-tenant serialisation |
TraceID | Propagated to the handler context for end-to-end trace correlation |
The Worker
import "github.com/xaleel/maniflex/jobs"
w, err := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Handlers: map[string]jobs.Handler{
"send_receipt": func(ctx context.Context, j jobs.Job) (jobs.Result, error) {
var p struct{ OrderID string `json:"order_id"` }
json.Unmarshal(j.Payload, &p)
return jobs.Result{}, mailer.SendReceipt(ctx, p.OrderID)
},
},
Concurrency: 8, // goroutines; default = GOMAXPROCS
Logger: slog.Default(),
})
ctx, cancel := context.WithCancel(context.Background())
go w.Run(ctx)
// On shutdown:
cancel()
w.Shutdown(shutdownCtx)
Migrate before you launch background goroutines.
server.Go(fn)(and a barego w.Run(ctx)) starts running immediately, butAutoMigrateonly runs insideStart(). A worker that touches a table beforeStart()migrates it races table creation. When you launch workers yourself, callserver.MigrateOnly(ctx)afterSetDBand before starting them so the tables exist.
Result carries an optional URL (pre-signed storage URL for file outputs)
and Output (small structured JSON). Both are surfaced through the status model
below.
StatusModel — REST polling
Mount the status model once, alongside other model registrations:
import jobsmaniflex "github.com/xaleel/maniflex/jobs/maniflex"
sink, queue, err := jobsmaniflex.Mount(server, rawQueue)
if err != nil { log.Fatal(err) }
// Pass sink to the worker:
w, _ := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Status: sink,
Handlers: handlers,
})
Mount registers a StatusModel (table job_statuses) and returns:
sink— ajobs.StatusSinkto pass toWorkerConfig.Status; the worker writes a row for every lifecycle transition.queue— a wrappedjobs.Queue; everyEnqueuecall creates an initialenqueuedstatus row so clients can poll immediately.
The REST layer exposes these endpoints automatically (no extra code):
GET /api/job_statuses list (filterable, paginated)
GET /api/job_statuses/:id single row
POST /api/job_statuses → 405 (worker-only)
A typical client flow after an action returns {"job_id": "abc"}:
GET /api/job_statuses/abc
→ {"data": {"status": "enqueued", ...}}
GET /api/job_statuses/abc (poll until done)
→ {"data": {"status": "succeeded", "result_url": "https://...", "completed_at": "..."}}
Status values: enqueued → running → succeeded | failed | dead | cancelled.
Scope
By default, unauthenticated requests see all rows. When a caller is
authenticated, the built-in force-filter restricts the list to their own
actor_id; callers with the admin role see everything. Override the role
name with MountOptions.AdminRole.
Atomic enqueue with jobs/sql
When jobs/sql is the adapter and a maniflex.WithTransaction middleware is
active, queue.Enqueue runs its INSERT through the same *sql.Tx:
// Service step:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { // DB write commits first
return err
}
_, err := queue.Enqueue(ctx.Ctx, jobs.Job{
Type: "reconcile_inventory",
Payload: json.RawMessage(`{"product_id":"` + productID + `"}`),
})
return err
}, maniflex.ForModel("Order"), maniflex.AtPosition(maniflex.After))
If the transaction rolls back, the job row never appears. If the process crashes after commit, the job row is durable and the worker will pick it up. This eliminates the “DB committed but job lost” race that an in-memory queue cannot prevent.
GroupKey — serialised execution
Set GroupKey to ensure at most one job for a given key runs at a time:
queue.Enqueue(ctx, jobs.Job{
Type: "generate_payroll",
GroupKey: "tenant:" + tenantID, // one payroll run per tenant at a time
})
The jobs/sql adapter enforces this via SELECT … SKIP LOCKED; jobs/inproc
tracks running keys in memory.
Retry and dead-letter
When a handler returns an error the worker re-queues the job after an
exponential backoff (base 1 s, cap 5 min). After Job.MaxRetry attempts the
job is marked dead and the status row records the final error. Set
WorkerConfig.DLQType to route dead jobs to a separate handler for inspection
or alerting.
Cancellation
When the inner queue implements jobs.Cancellable (both jobs/inproc and
jobs/sql do), the wrapped queue returned by Mount also implements it:
c := queue.(jobs.Cancellable)
c.Cancel(ctx, jobID) // marks the job cancelled in the queue and updates the status row
Only jobs that have not yet started can be cancelled; a running job must finish or fail before the status row moves.
Completion events (optional)
Set WorkerConfig.EventBus to publish job.{type}.completed and
job.{type}.failed events on every terminal transition. Pair with a
realtime.Hub to push completion notifications to connected clients without
polling:
w, _ := jobs.NewWorker(jobs.WorkerConfig{
// ...
EventBus: bus, // any value implementing Publish(ctx, type, payload) error
})
Scheduled jobs with jobs/cron
jobs/cron provides a minimal ticker that calls Queue.EnqueueAt on a fixed
interval. It does not offer durable cron (if a replica is down at fire time the
tick is missed); for durable scheduling, combine jobs/sql with a
next_fire_at column in your model. For field-based transitions (auto-publish,
auto-expire), see Scheduled Fields & Runner.
Schedules are fixed intervals (Every), not cron expressions:
import (
"time"
"github.com/xaleel/maniflex/jobs"
"github.com/xaleel/maniflex/jobs/cron"
)
cr := cron.New(queue, nil) // nil logger → slog.Default()
cr.Add(cron.Entry{
Every: 24 * time.Hour,
Job: jobs.Job{Type: "daily_report"},
})
cr.Start(ctx) // returns immediately; cr.Stop() halts the tickers
Realtime / WebSockets
maniflex is a synchronous request/response framework, but the realtime package
ships a first-class event hub that pushes domain events to browsers over
WebSocket and Server-Sent Events. It is a pure consumer of the
event bus: producers publish through events.Emit exactly as
they would for any other subscriber, and the hub fans those events out to
connected clients.
Nothing about realtime leaks into a CRUD-only app — the hub is mounted by your
own code outside server.Handler(), so a blog that never imports realtime
pays no websocket dependency, goroutine, or shutdown phase.
The shape of it
import (
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/events"
"github.com/xaleel/maniflex/events/inproc"
"github.com/xaleel/maniflex/realtime"
)
bus := inproc.New() // or events/redis, events/nats, … for multi-replica
// Producer: every create/update/delete publishes a domain event.
server.Pipeline.DB.Register(
events.Emit(bus, events.EmitConfig{Source: "billing"}),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
// Consumer: the hub fans those events out to clients.
hub, err := realtime.NewHub(realtime.HubConfig{Bus: bus})
if err != nil {
log.Fatal(err)
}
r := chi.NewRouter()
r.Mount("/api", server.Handler())
r.Handle("/ws", hub.Handler()) // WebSocket upgrade
r.Handle("/sse", hub.SSEHandler()) // Server-Sent Events fallback
http.ListenAndServe(":8080", r)
Removing realtime is a one-line revert: drop the two r.Handle lines and the
events.Emit registration.
Topics
Events are addressed by their CloudEvents type — a dotted string like
invoice.created or queue.position_changed. Clients subscribe with glob
patterns (the same matcher the event bus uses):
| Pattern | Matches |
|---|---|
invoice.* | invoice.created, invoice.updated, … |
*.created | any ….created event |
* | every event |
HubConfig.AllowPatterns is an optional whitelist of subscribable patterns; an
empty list allows any. A client that asks for a forbidden pattern gets a
FORBIDDEN_PATTERN error (WS) or a 403 (SSE).
WebSocket protocol
The client speaks a tiny JSON protocol over the socket:
client → server server → client
{"op":"subscribe","patterns":["invoice.*"]} {"op":"ack","subId":"s_1"}
{"op":"unsubscribe","subId":"s_1"} {"op":"event","subId":"s_1","data":<event>}
{"op":"ping"} {"op":"pong"}
{"op":"error","code":"…","msg":"…"}
The data field is the full CloudEvents JSON document, so a
browser can parse it with any CE SDK.
SSE protocol
SSE is push-only and subscribes via query parameters — ideal for corporate networks that break WebSockets:
GET /sse?subscribe=invoice.*&subscribe=queue.position_changed
Each event arrives as a standard data: frame whose body is the same
CloudEvents JSON.
The SSE response always sets X-Accel-Buffering: no — a safe default that stops
NGINX from buffering the stream and delivering events in batches.
Authentication
Connections are authenticated once, on connect (never per message). Supply
an Authenticator; the default AnonymousOnly{} accepts everyone.
hub, _ := realtime.NewHub(realtime.HubConfig{
Bus: bus,
Authenticator: realtime.BearerToken(func(tok string) (*realtime.Principal, error) {
claims, err := verifyMyJWT(tok)
if err != nil {
return nil, err
}
return &realtime.Principal{UserID: claims.Sub, TenantID: claims.Tenant, Roles: claims.Roles}, nil
}),
})
BearerToken pulls the token from the Authorization: Bearer … header, the
?access_token= query parameter (browsers can’t set headers on WebSocket()),
or the Sec-WebSocket-Protocol: access_token.<token> subprotocol. Composite
tries several authenticators in order.
Per-event authorisation
AllowPatterns controls which topics a client may subscribe to; Visibility
controls which individual events it actually receives. The hook runs once per
(event, client) pair and can also redact the payload:
HubConfig{
Visibility: func(p *realtime.Principal, e events.Event) (bool, *events.Event) {
if e.TenantID != p.TenantID {
return false, nil // suppress cross-tenant events
}
return true, nil
},
}
Return (true, ©) to deliver a transformed event — the hub clones before
mutation so each client sees its own view.
Heartbeat
Idle connections are kept alive automatically so L7 proxies (ALB, NGINX, with their typical 30–60s idle timeouts) don’t drop them:
- WebSocket — the server sends a ping frame every
PingInterval(default 30s); compliant clients answer with a pong. - SSE — the server emits a
: keepalivecomment on the same interval.
Resumable streams (lastEventId)
By default delivery is ephemeral: a client that disconnects misses whatever was published while it was away. Enable resume to give clients a replay buffer.
hub, _ := realtime.NewHub(realtime.HubConfig{
Bus: bus,
ResumeBuffer: 1024, // retain the most recent 1024 events for replay
})
With resume enabled, every delivered event carries a cursor:
- SSE — the cursor is the standard
id:line. On reconnect the browser’sEventSourceautomatically sendsLast-Event-ID, and the hub replays everything after it before resuming the live stream. (You can also pass?lastEventId=<cursor>explicitly.) - WebSocket — events include a
"cursor"field; resume by adding it to your subscribe message:{"op":"subscribe","patterns":["invoice.*"],"after":"<cursor>"}.
If the cursor is older than the retained buffer (or the hub restarted), the
client receives a resync signal — event: resync on SSE, {"op":"resync"}
on WebSocket — telling it to refetch current state instead of silently missing
events. Across the reconnect seam delivery is at-least-once; because cursors are
monotonic, clients drop anything at or below their last applied cursor.
ResumeBuffer installs an in-process ring buffer, so resume works when the
client reconnects to the same replica (WebSocket affinity). For
cross-replica resume, supply your own ResumeStore (e.g. backed by a Redis
stream) via HubConfig.ResumeStore.
Schema-emitting events (AsyncAPI)
Just as /openapi.json lets clients codegen typed REST clients, the hub’s event
catalogue can be published as an AsyncAPI 2.6 document so clients codegen
typed event payloads. Declare it once:
server.RealtimeDoc(maniflex.AsyncAPIConfig{
Title: "Billing events",
Servers: []maniflex.AsyncAPIServerConfig{
{Name: "ws", URL: "ws://localhost:8080/ws", Protocol: "ws"},
},
// Derive invoice.created|updated|deleted channels from registered models:
AutoModelEvents: true,
// …and/or declare custom events with a Go struct payload:
Events: []maniflex.EventDoc{
{Type: "payment.received", Title: "Payment received", Payload: PaymentReceived{}},
},
})
This mounts GET {PathPrefix}/asyncapi.json. The payload struct is reflected
with the same json + mfx tags as models and actions
(Actions). The endpoint is opt-in — apps that never call
RealtimeDoc get no new route.
Backpressure & slow clients
Each connection has a bounded outbound queue (SendBuffer, default 64). If a
client can’t keep up within SendTimeout (default 5s) it is kicked — a WebSocket
close 1013 Try Again Later, or an SSE disconnect that triggers EventSource
reconnection. Hub.Stats() exposes the live connection count and cumulative
kick count for monitoring. A frame larger than MaxMessageSize (default 64 KiB)
is rejected with close 1009.
Scaling out
The hub is single-process by design; cross-replica fan-out is the bus’s job:
- inproc (single binary) — one hub, all clients local.
- redis / nats / kafka — every replica subscribes to the bus, so an event published anywhere reaches local clients on every replica. Pair with a sticky load balancer so each client stays on one replica (WebSocket affinity).
The hub does not create a consumer group per connection — per-client filtering happens server-side, downstream of one shared bus subscription, so broker load doesn’t scale with connection count.
Graceful shutdown
Hub.Shutdown(ctx) stops accepting connections, sends a 1001 Going Away close
to every client, drains in-flight writes until the deadline, then cancels the
bus subscription. Call it alongside *http.Server.Shutdown from the same signal
handler — the hub is mounted by your code, so it isn’t part of
server.Shutdown.
HubConfig reference
| Field | Default | Purpose |
|---|---|---|
Bus | — (required) | the events.Bus the hub consumes |
Authenticator | AnonymousOnly{} | connection auth |
Visibility | allow-all | per-event authorisation / redaction |
AllowPatterns | allow-all | subscribable topic whitelist |
ResumeStore | nil (disabled) | replay buffer for lastEventId resume |
ResumeBuffer | 0 (disabled) | shortcut: install an in-memory store of this size |
PingInterval | 30s | WS ping / SSE keepalive cadence |
SendBuffer | 64 | per-client outbound queue depth |
SendTimeout | 5s | slow-client kick threshold |
MaxMessageSize | 64 KiB | inbound frame size limit |
Origins | allow-all | allowed Origin values for the WS upgrade |
Encryption at Rest
A field tagged mfx:"encrypted" is automatically encrypted before it
reaches the database and decrypted on read. The plaintext never appears in
the table; the column stores a self-describing envelope. This page covers
the full subsystem: the tag, the key provider interface, the storage
format, unique-constraint handling, and key rotation.
Declaring an encrypted field
type Patient struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required,filterable,sortable"`
SSN string `json:"ssn" mfx:"encrypted,key:patient-pii"`
}
| Sub-option | Effect |
|---|---|
encrypted | mark the field for envelope encryption |
key:NAME | the key identifier passed to the KeyProvider; defaults to "default" |
The column’s Go and DB types remain string. Storage is the prefix
enc: followed by a base64-encoded binary envelope:
enc:Aa1z... (envelope bytes embed the keyID)
The enc: prefix lets the framework distinguish ciphertext from any
legacy plaintext that may exist in the column — useful for incremental
migration of an existing table.
What encryption costs you
The trade-offs are deliberate and worth being explicit about:
- No filtering. A
WHERE ssn = ?would have to match an envelope that includes a random nonce. Encrypted fields cannot befilterable. - No sorting. Same reason. Encrypted fields cannot be
sortable. - Uniqueness via HMAC. A
mfx:"encrypted,unique"field gets a companion{field}_hmacTEXT UNIQUEcolumn. See the next section. - The KeyProvider is required. Reads degrade to returning the raw
stored ciphertext; writes are rejected with
500 ENCRYPTION_NOT_CONFIGUREDuntil a provider is configured.
For columns that need to be queryable but contain sensitive data, store a non-sensitive lookup key (a hashed identifier) in a separate field and encrypt only the payload.
Access paths
Encryption is applied on every access path, not just HTTP:
-
HTTP pipeline — automatic.
-
Typed helpers (
maniflex.Create/Read/Update/List) andctx.GetModel(name)— encrypt on write and decrypt on read using theServerContext’s KeyProvider (set automatically inside a request). -
Background workers / CLIs using
maniflex.NewBackground(...)— callbg.SetKeyProvider(srv.KeyProvider())so typed access to encrypted models encrypts and decrypts:bg := maniflex.NewBackground(ctx, srv.DB(), srv.Registry()) bg.SetKeyProvider(srv.KeyProvider()) p, _ := maniflex.Read[Patient](bg, id) // decrypted
The raw adapter (srv.DB().Create/FindMany) is deliberately encryption-agnostic
— it stores and returns ciphertext verbatim, which is what RotateEncryptionKey
relies on. Use the typed helpers or ctx.GetModel for transparent crypto.
Configuring a KeyProvider
maniflex.Config.KeyProvider must be set before any model with encrypted
fields is exercised. Two implementations ship in pkg/encryption, both
constructed as plain struct literals.
EnvKeyProvider — keys from environment variables
import "github.com/xaleel/maniflex/pkg/encryption"
server := maniflex.New(maniflex.Config{
KeyProvider: &encryption.EnvKeyProvider{Prefix: "MYAPP_KEY"},
// ...
})
The env var name for a given keyID is derived as
{Prefix}_{KEYID_UPPER}, with hyphens replaced by underscores and the
result uppercased:
Prefix | keyID | Env var read |
|---|---|---|
MYAPP_KEY | default | MYAPP_KEY_DEFAULT |
MYAPP_KEY | patient-pii | MYAPP_KEY_PATIENT_PII |
MFX_KEY (default) | billing | MFX_KEY_BILLING |
Each variable holds a base64-encoded 32-byte (256-bit) AES key. Generate one with:
openssl rand -base64 32
The provider accepts either standard or URL-safe base64.
VaultKeyProvider — HashiCorp Vault Transit
server := maniflex.New(maniflex.Config{
KeyProvider: &encryption.VaultKeyProvider{
Address: "https://vault.example.com",
Token: os.Getenv("VAULT_TOKEN"),
Mount: "transit", // optional, default "transit"
// Client: customHTTPClient, // optional, defaults to http.DefaultClient
},
})
keyID maps to a Vault Transit key name; the plaintext is sent to
/v1/{mount}/encrypt/{keyID} and Vault returns its own
vault:v1:... ciphertext, which the provider embeds in the envelope.
A Vault key rotation is transparent — Vault decrypts ciphertexts
encrypted with any prior version of the key automatically.
The shipped provider uses a static token. For production, wrap it with a refresher that obtains a fresh token from AppRole, Kubernetes auth, or JWT auth before each operation.
A custom backend implements the maniflex.KeyProvider interface:
type KeyProvider interface {
Encrypt(ctx context.Context, keyID string, plaintext []byte) ([]byte, error)
Decrypt(ctx context.Context, envelope []byte) ([]byte, error)
KeyIDOf(envelope []byte) (string, error)
HMAC(ctx context.Context, keyID string, data []byte) ([]byte, error)
}
Encrypt returns a self-describing binary envelope that embeds the
keyID. Decrypt reads the keyID from the envelope, so callers don’t
supply it. HMAC produces a deterministic keyed digest used for unique
indexes.
Unique encrypted fields
A normal UNIQUE constraint on an envelope is useless — each envelope
contains a random nonce, so two encryptions of the same plaintext are
different ciphertexts. The framework solves this with an HMAC companion
column.
Email string `json:"email" mfx:"encrypted,unique"`
AutoMigrate emits two columns:
| Column | Type | Purpose |
|---|---|---|
email | TEXT | the enc:<base64> envelope |
email_hmac | TEXT UNIQUE | a keyed HMAC of the plaintext |
On every write, the DB step calls KeyProvider.HMAC(ctx, keyID, plaintext)
and stores the result in the companion. The HMAC is deterministic for a
given (key, plaintext) pair, so the database can enforce uniqueness
without ever seeing the plaintext.
Reads strip the HMAC column from responses automatically; clients see only
the decrypted plaintext on email and never the digest.
When the unique check fires, the adapter returns *maniflex.ErrConstraint and
the DB step converts it to 409 CONFLICT — same path as any other unique
violation.
If the index cannot be created
The UNIQUE index over the HMAC column is what actually enforces the
constraint. If AutoMigrate cannot create it, the migration fails rather
than starting a server whose mfx:"unique" is claiming a guarantee the database
is not enforcing:
AutoMigrate: could not create unique index uidx_users_email_hmac on
users(email_hmac): UNIQUE constraint failed — the model declares these
column(s) unique, so the server will not start without the constraint. The
usual cause is rows that already violate it; de-duplicate them and start again
The usual cause is exactly that: rows already holding duplicate values in the field. De-duplicate them and start again. A plain (non-unique) index that fails to build is only warned about — a missing one costs a table scan, not a constraint.
Per-domain keys
The key:NAME sub-option routes a field to a specific key identifier:
type Record struct {
maniflex.BaseModel
PaymentToken string `json:"payment_token" mfx:"encrypted,key:billing"`
MedicalNote string `json:"medical_note" mfx:"encrypted,key:medical"`
}
A KeyProvider that backs different keys with different secrets (or a Vault transit mount) lets you scope access by domain — the billing team holds the billing key; medical staff hold the medical key; the application process holds both. Rotating one does not affect the other.
When key: is omitted, the framework uses the keyID "default". Either
configure a key under that name or always tag with an explicit key.
Decryption on the read path
The DB step runs the decryption pass after every read:
- For list and read operations,
decryptFieldsreplaces everyenc:<base64>value with the decrypted plaintext. - HMAC companion columns are always stripped from the response.
- Values that do not have the
enc:prefix are left as-is — important for gradual adoption: enable encryption on a column whose existing rows are plaintext, and only new writes get encrypted. - Rows pulled in by
?include=are decrypted too. A relation’s row reaches the serializer straight from the adapter rather than through the DB step’s own pass, so before v0.2.5 an encrypted field on an included model came back as base64 ciphertext while the same model read directly returned plaintext. If a relation cannot be decrypted the field is left as stored and a warning is logged, rather than failing the whole request over one included row.
If KeyProvider is nil but a model has encrypted fields, reads return
the raw stored ciphertext (so the application still functions in some
read-only sense), but writes are refused. Configuring a provider is the
only way to write encrypted columns.
Key rotation
maniflex.RotateEncryptionKey(ctx, server, modelName, oldKeyID, newKeyID)
re-encrypts every row of a model whose envelopes were encrypted with
oldKeyID:
n, err := maniflex.RotateEncryptionKey(ctx, server, "Patient", "v1", "v2")
if err != nil {
log.Fatal(err)
}
log.Printf("re-encrypted %d rows", n)
The function pages through the table 100 rows at a time, decrypts each
value with the old key, re-encrypts with the new key, and updates the
HMAC companion for any unique encrypted fields. Both keys must remain
available in the KeyProvider until the rotation completes — partial
rotations leave the table with a mix of old-key and new-key envelopes
until you finish.
The operation is not atomic across all rows. On failure, run it again —
it skips envelopes whose keyID already matches newKeyID, so a partial
rotation is safe to resume.
For large tables, run the rotation as a background job rather than at
startup. Each row is a separate UPDATE, so the operation is bound by
the database’s write throughput.
What an envelope looks like
The exact envelope format is the KeyProvider’s concern. The two
shipped providers use slightly different layouts, but both put a
self-describing header in front of the ciphertext so KeyIDOf can
extract the keyID without decrypting.
EnvKeyProvider — AES-256-GCM with an inline nonce:
[ version:1 ][ keyIDLen:2 (BE) ][ keyID:N ][ nonce:12 ][ gcmCiphertext+tag:M ]
New envelopes use version 0x02, which binds the version byte and keyID into
the GCM tag as additional authenticated data (AAD), so neither can be altered
without failing decryption. Legacy 0x01 envelopes — sealed without that
binding — still decrypt, and a key rotation re-writes them as 0x02.
VaultKeyProvider — Vault returns a vault:v1:... ciphertext that
embeds its own versioning, so the envelope carries no nonce:
[ version:1 ][ keyIDLen:2 (BE) ][ keyID:N ][ vaultCiphertext:M ]
Vault envelopes use version 0x01 (Vault Transit authenticates internally).
The keyID length is a 16-bit big-endian integer, and the framework stores the
binary envelope as the string enc:<base64> in the column.
Encrypt produces the blob; Decrypt parses the header to recover the
keyID, then routes to the right key. KeyIDOf reads the keyID without
decrypting — useful for audit logging and for the rotation loop above.
A custom provider need not follow either format; the framework only
cares that Encrypt and Decrypt are inverses and that KeyIDOf works
on the output of Encrypt.
Compatibility with other features
| Feature | Interaction |
|---|---|
mfx:"encrypted" + unique | HMAC companion column; standard unique violation as 409 |
mfx:"encrypted" + filterable / sortable | not allowed — a query that filters an encrypted field is rejected at query time with ENCRYPTED_FIELD_NOT_FILTERABLE; the tag itself is not stripped |
mfx:"encrypted" + soft-delete | independent — soft-delete operates on a separate marker column |
mfx:"encrypted" + versioning | encrypted fields are excluded from both the diff and the snapshot, along with their _hmac companions. The history row is built from the decrypted record, so anything left in it would be stored as plaintext |
mfx:"encrypted" + audit log | the audit Changes diff does not exclude encrypted fields automatically; use WithExcludeFields to keep them out |
mfx:"encrypted" + relations | a relation FK is never encrypted; relation joins remain unaffected. Encrypted fields on an included relation are decrypted like any other read |
mfx:"encrypted" + export | the export writes decrypted plaintext — it is a portable file of the data, not of the ciphertext. Exclude the column with response.RedactField where that is not wanted |
Operational checklist
- Set
Config.KeyProviderbefore any encrypted-field model is registered. - Back keys with a secret store (env vars from a vault, HashiCorp Vault Transit, AWS KMS). Never commit a key to source control.
- For staged rollout, deploy the schema (
email_hmaccolumn) before the application change that starts encrypting — and the application change before the migration that backfills existing rows. - Keep both keys active throughout a rotation; remove the old key only
after
RotateEncryptionKeyhas reported every row migrated. - Treat
KeyIDOf(envelope)as the source of truth for “which key encrypted this row” — useful for auditing the rotation.
Versioning & History
A model marked Versioned keeps an immutable history of every write to it.
The framework creates a sibling {model}_history table at migration time
and appends one row per Create / Update / Delete. History rows are
queryable through the same REST surface as any other model, with one
restriction: they are read-only.
Opting in
Set Versioned: true in ModelConfig:
server.MustRegister(
Invoice{}, maniflex.ModelConfig{
Versioned: true,
},
)
Equivalent declaration on the embedded BaseModel:
type Invoice struct {
maniflex.BaseModel `mfx:"versioned"`
Number string `json:"number" mfx:"required,unique"`
Amount float64 `json:"amount" mfx:"required,min:0"`
}
Either form triggers two effects at registration:
- A synthetic
InvoiceHistorymodel is added to the registry — same as any other model, but read-only. - Three DB middlewares are attached to
Invoice: a pre-image capture beforeOpUpdate/OpDelete, and an After-DB writer for every write that succeeded.
The history table
The sibling table has a fixed schema, regardless of the source model’s columns:
| Column | Type | Notes |
|---|---|---|
id | TEXT | UUID, primary key of the history row itself |
record_id | TEXT | id of the source row this entry describes |
version | INTEGER | 1-based, monotonic per record_id |
operation | TEXT | "create", "update", or "delete" |
actor_id | TEXT | ctx.Auth.UserID at the time of the write; nullable |
timestamp | TIMESTAMP | UTC, set by the framework |
request_id | TEXT | the X-Request-Id of the producing request |
diff | TEXT | JSON {field: {old, new}} map |
snapshot | TEXT | full row state as JSON — omitted when VersionedDiffOnly is set |
AutoMigrate also adds a unique index uidx_{table}_history_record_version
on (record_id, version DESC) for the standard “list history for one row”
query. The uniqueness guards against two concurrent writes computing the same
(record_id, version); the writer retries on the constraint violation.
What gets diffed
diff records every changed scalar field. The format is:
{
"amount": {"old": 99.0, "new": 105.0},
"status": {"old": "draft", "new": "sent"}
}
OpCreate— every field with a non-nil value is recorded as{"old": null, "new": value}; nil-valued fields are skipped.OpUpdate— only fields whose value differs between pre-image and post-image appear.OpDelete— every field is recorded as{"old": value, "new": null}.
Excluded by default:
- The primary key (
id) — from the diff only; the snapshot keeps it. hiddenfields.writeonlyfields.encryptedfields and their{field}_hmaccompanions.
This avoids leaking secrets into history while still capturing the
business-meaningful changes. The same exclusions apply to the
snapshot — history rows are built from the decrypted row, so an
encrypted column left in the snapshot would sit in the history table as
plaintext and quietly undo the at-rest guarantee. If you need a value
in history, don’t mark it encrypted, hidden or writeonly.
Snapshot vs. diff-only
By default each history row carries both the diff and the full
snapshot of the row state — convenient for “what did the record look
like on date X?” queries:
# The newest history row for one invoice.
curl 'localhost:8080/api/invoices/abc123/history?limit=1'
For high-write models the snapshot is the largest column by far.
VersionedDiffOnly: true skips the snapshot entirely:
server.MustRegister(
EventLog{}, maniflex.ModelConfig{
Versioned: true,
VersionedDiffOnly: true,
},
)
The trade-off: reconstructing the row state at version N requires walking all entries from version 1 to N and applying their diffs. For an audit trail used by humans (reading recent changes) this is fine; for point-in-time recovery, keep the snapshot.
Reading history
History is read through the record it belongs to:
# One invoice's history, newest first.
curl 'localhost:8080/api/invoices/abc123/history'
# Paginated.
curl 'localhost:8080/api/invoices/abc123/history?page=2&limit=50'
The response is the standard list envelope. Rows come back newest-first
(descending version); page and limit are honoured, and the default
page size is 20.
Why not a flat /invoice_history endpoint?
There used to be one, and it was a security hole (audit MS-4). The synthesized history model mounted the full read surface, but per-model middleware is registered against the parent’s name — an app that wrote:
server.MustRegister(Invoice{}, maniflex.ModelConfig{
Versioned: true,
Middleware: &maniflex.ModelMiddleware{Auth: []maniflex.MiddlewareFunc{requireLogin}},
})
protected GET /invoices and left GET /invoice_history open to
anyone. db.Tenancy("org_id", …) scoped with ForModel("Invoice") had
the same gap, so any caller could read every tenant’s history.
Copying the parent’s middleware onto the history model does not fix it.
The history table has none of the parent’s columns — it holds id,
record_id, version, operation, actor_id, timestamp,
request_id, diff and snapshot — so a tenancy filter on org_id
has nothing to filter. It would look scoped and enforce nothing.
So the history model is now Headless (registered and migrated, no
routes of its own) and reached only through the parent. The request runs
the parent’s read pipeline first: if you cannot read the record, you
cannot read its history, and you get the same 404 the record itself
would give you rather than a 403 that confirms it exists. Every auth,
tenancy and force-filter middleware you already registered applies, with
nothing new to configure.
Middleware scoped with ForOperation(OpRead) does match these requests,
and that is what makes the gate work: it reads the request’s forced filters, so
a tenancy middleware that never ran would leave it with nothing to scope by.
The implication runs one way only — ForOperation(maniflex.OpReadHistory)
means history requests alone.
Deleted records
A soft-deleted record keeps its history, including the delete entry.
The gate uses the adapter’s ScopeChecker capability, which counts
soft-deleted rows as present while still applying your scope — so a
deleted record’s history is visible to exactly the callers who could see
the record, and to nobody else.
A hard-deleted record’s history is not reachable over HTTP. The row
that said who was allowed to read it is gone, and answering from the
history table alone would mean showing it either to everyone or to no
one. The rows remain in {model}_history for an admin query or an
offline audit. If you need history to outlive deletion, use soft-delete
(maniflex.WithDeletedAt).
Custom adapters
ScopeChecker is optional. An adapter that does not implement it — a
third-party one written against the DBAdapter interface — keeps
working, and the endpoint stays exactly as scoped: the gate falls
back to an ordinary scoped read, so tenancy and force filters apply as
they always did. The one thing it gives up is the soft-delete case: that
read applies the soft-delete condition, so a soft-deleted record’s
history 404s, the same as a hard-deleted one.
There is no warning at startup, because nothing is misconfigured — you did not ask for a capability and fail to get it. To gain it, implement:
func (a *MyAdapter) ExistsInScope(
ctx context.Context, model *maniflex.ModelMeta, id string,
filters []*maniflex.FilterExpr,
) (bool, error)
Report whether a row with that id exists and satisfies filters,
including when it is soft-deleted. Apply the filters in full —
“deleted” must not become “unscoped” — and return only the boolean, never
the row: a method that returned soft-deleted records would be a general
bypass of the soft-delete condition, and this one is deliberately unable
to serve as one. Implementing it on your Tx type as well lets a
history read inside a transaction see the request’s own uncommitted
writes.
Filtering
Query parameters are parsed against the parent model, so
?filter=operation:eq:update is rejected — operation is not a field
of Invoice. Filtering within a record’s history is not currently
supported; a record’s history is bounded by that record’s edit count,
and pagination covers it. For cross-record queries (“everything alice
changed this week”), use audit logging, which is built for
exactly that question.
The history model is Headless, so it contributes a schema to
/openapi.json but no paths; the /{id}/history route is documented on
the parent.
Transactions and history
The history row is written in the same transaction as the source write — both succeed together or neither does. If the primary insert rolls back, no orphan history entry is left behind.
If the history write itself fails after a successful primary write, the
framework logs the error but does not fail the primary response.
Losing one history row is preferable to refusing a write that the user
already saw succeed. The error is logged via ctx.Logger() so an
operator can investigate.
That default is wrong when history is an audit record rather than a
convenience: the gap is silent, and it surfaces only when someone asks what
changed. Set VersionedRequired to make the failure fail the request:
server.MustRegister(Invoice{}, maniflex.ModelConfig{
Versioned: true,
VersionedRequired: true, // no history row → no write
})
Because the history row is written in the same transaction, returning the error rolls the primary write back with it — the change and its audit entry stand or fall together. Note the exception: with no transaction in force there is nothing to roll back, and the error only reports that history is missing after the fact.
Performance notes
- One additional
INSERTper write to a versioned model. Postgres handles this with a write multiplier of ~2x on the affected tables. - The
snapshotJSON is the dominant cost on row size. UseVersionedDiffOnlyfor verbose tables. - The
record_idindex is essential — every “history for one row” query uses it. Don’t drop it. - For very-high-write models, consider routing history to a separate table partition or a write-optimised store (TimescaleDB, ClickHouse) via a custom DB-After middleware instead of the built-in.
Comparison with audit logging
Audit Logging and Versioning solve different problems:
| Versioning | Audit Logging | |
|---|---|---|
| Storage | sibling DB table | configurable sink (DB, syslog, SIEM, …) |
| Granularity | per-row | per-row, optionally with diff |
| Transactional with the write | yes | yes (Before-DB) |
| Reconstruct prior state | yes — via snapshot or diff replay | no — only the change is recorded |
| Read API | the framework’s list/read on {model}_history | up to the sink |
| Best for | “what did this invoice look like a week ago?” | “who did what, when, across the whole system?” |
The two compose cleanly — turn on versioning for models that need reconstructable history, and audit-log everything for compliance.
Operational checklist
- Enable
Versionedon models whose change history matters for compliance, debugging, or undo. Don’t enable it on every model — the write multiplier adds up. - Choose
VersionedDiffOnly: truefor high-write tables where the diff alone is enough. - Plan storage growth: history is monotonic — older rows never go away unless you delete them out of band. Set up a retention job for very active models.
- Restrict access to the history endpoints with
auth.RequireRole— the diff and snapshot may contain values an end user shouldn’t see.
Scheduled Fields & the Runner
A mfx:"scheduled" tag on a *time.Time field declares a time-driven
transition: when the timestamp falls into the past, the framework applies
a configured action to the row. The mechanism is small but covers a
surprising number of real workflows — auto-publish, auto-archive,
soft-delete after expiry, scheduled status transitions.
This page covers both halves: the tag (declarative, per-model) and the runner (the background goroutine that actually applies transitions).
The tag
mfx:"scheduled" must appear on a *time.Time field (the pointer type
is required so “unset” is distinguishable from the zero time). The tag
takes one action and any number of qualifiers, separated by semicolons:
type Post struct {
maniflex.BaseModel
maniflex.WithDeletedAt
Title string `json:"title"`
Status string `json:"status" mfx:"required,enum:draft|published|archived,default:draft"`
// Auto-publish: set status=published when publish_at falls in the past.
PublishAt *time.Time `json:"publish_at" mfx:"scheduled;field=status;from=draft;to=published"`
// Auto-archive: set status=archived once archive_at falls in the past
// (no from= — applies regardless of current status).
ArchiveAt *time.Time `json:"archive_at" mfx:"scheduled;field=status;to=archived"`
// Auto-soft-delete: requires WithDeletedAt above.
ExpiresAt *time.Time `json:"expires_at" mfx:"scheduled;soft-delete"`
}
Actions
Exactly one action per scheduled field:
| Action | Effect when the timestamp passes |
|---|---|
soft-delete | sets the soft-delete marker — requires maniflex.WithDeletedAt or WithIsDeleted |
hard-delete | physically deletes the row, regardless of soft-delete config |
field=NAME;to=VALUE | sets the named field to the value |
Qualifiers
The field=...;to=... action accepts optional qualifiers:
| Qualifier | Effect |
|---|---|
from=VALUE | apply only when the named field currently equals this value |
to=VALUE | the value to assign (required for field=...) |
from= and to= are validated against the field’s enum (if any) at
registration time — a typo aborts the boot, not the first sweep.
Validation at registration
Every scheduled tag is resolved when ScanModel runs. Configurations
that don’t make sense are reported and the field is dropped from the
runner’s scope:
- Field type must be
*time.Time. - Exactly one of
soft-delete,hard-delete,field=is required. soft-deleterequires the model to be soft-deletable.field=requires ato=and references an existing column.from=/to=must be members of the target field’senum, if it has one.
A scheduled column automatically gets an IndexSpec added to the model
so the runner can locate due rows without a full scan.
The runner
The runner lives in maniflex/scheduled (its own satellite-style package).
It is opt-in — declaring scheduled tags makes the rows ready to be acted
on, but nothing happens until a runner is started.
import "github.com/xaleel/maniflex/scheduled"
runner, err := scheduled.New(server, scheduled.Config{
Interval: time.Minute,
BatchSize: 500,
})
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
runner.Start(ctx)
defer runner.Stop()
scheduled.New walks the registry, picks up every model that declares a
scheduled field, and binds them to the runner. A registry with no
scheduled fields produces a usable no-op runner — callers can wire it
unconditionally and pay no cost.
Config
| Field | Default | Purpose |
|---|---|---|
Interval | 1m | how often the loop ticks |
BatchSize | 500 | maximum rows processed per (model, spec) per tick |
Logger | slog.Default() | structured log sink |
Clock | time.Now().UTC | injectable; tests override |
OnDelete | nil | callback func(model, id string) after a delete commits |
OnSetField | nil | callback func(model, id, field, to string) after a set-field commits |
The two hooks fire once per affected row, after the per-model transaction has committed. They run outside the transaction, so a hook panic does not roll back the write.
What one tick does
For each registered model with scheduled specs, in turn:
- Run a
SELECT id, <column>, <conditional fields> FROM <table> WHERE <column> <= now() AND ...to find rows due for action. Thefrom=qualifier becomes an additionalAND field = 'value'clause. - Open a per-model transaction.
- Apply the action to each row in the batch:
soft-delete→UPDATE table SET deleted_at = now() WHERE id = ?hard-delete→DELETE FROM table WHERE id = ?(via the adapter’sHardDeleteif available)field=NAME;to=VALUE→UPDATE table SET name = ? WHERE id = ?
- Commit the transaction.
- Fire
OnDelete/OnSetFieldhooks for each row, in order. - Move to the next model.
The per-model transaction means a single bad row aborts only that
model’s batch, not the whole sweep. Errors are appended to the tick’s
Report.Errors and logged.
Sweep for one-shot ticks
runner.Sweep(ctx) runs exactly one tick and returns the Report:
report, err := runner.Sweep(ctx)
log.Printf("deleted %d, updated %d across %d models",
report.Deleted, report.Updated, len(report.PerModel))
Useful in tests and for cron-driven deployments where the framework’s
internal ticker is the wrong fit. Sweep blocks until the pass completes.
Distributed runners
A single runner per cluster is enough for most workloads — the operations it performs (soft-delete, status flip) are idempotent. Two runners processing the same batch simultaneously would do redundant work but no incorrect work.
For at-most-once semantics in a multi-process deployment, run the
runner in only one replica (a leader-elected pod, a sidecar, a separate
deployment). Or use the scheduled/jobsx adapter, which bridges the
runner to a jobs queue so the sweep is enqueued as a durable job and
dispatched by the worker pool:
import (
"time"
"github.com/xaleel/maniflex/jobs"
"github.com/xaleel/maniflex/jobs/cron"
"github.com/xaleel/maniflex/scheduled"
"github.com/xaleel/maniflex/scheduled/jobsx"
)
// Register the sweep handler on the worker, keyed by jobsx.JobType
// ("maniflex.scheduled.sweep").
w, _ := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Handlers: map[string]jobs.Handler{jobsx.JobType: jobsx.JobHandler(runner)},
})
go w.Run(ctx)
// A fixed-interval ticker enqueues one sweep job per minute; exactly one worker
// picks up any given tick.
sched := cron.New(queue, nil)
sched.Add(cron.Entry{Every: time.Minute, Job: jobs.Job{Type: jobsx.JobType}})
sched.Start(ctx)
In this setup the ticker drives the queue, not the runner directly — exactly one worker processes any given tick, even with many app replicas.
Hooks for events and audit
OnDelete and OnSetField are the natural place to emit events for
scheduled transitions, so downstream systems learn that a row’s status
changed even though no HTTP request caused the change:
runner, _ := scheduled.New(server, scheduled.Config{
OnSetField: func(model, id, field, to string) {
data, _ := json.Marshal(map[string]any{
"model": model, "id": id, "field": field, "to": to,
})
_ = bus.Publish(context.Background(), events.Event{
Type: "scheduled-transition",
Data: data, // Event.Data is json.RawMessage
})
},
})
The hook fires outside the database transaction. For at-least-once delivery semantics, write a row to an outbox table from inside the runner’s transaction (via a custom DB middleware on the affected models) rather than relying on the hook.
Interaction with versioning and audit
A scheduled transition is just an UPDATE (or DELETE) issued by the
runner. It flows through the model’s normal middleware:
Versionedmodels get a history row for the transition, withactor_id = NULL(noctx.Authexists in the runner).db.AuditLogrecords the write the same way.
This is intentional — a status change is a status change, regardless of whether a human or the runner triggered it.
When to use scheduled fields
| Need | Fit |
|---|---|
| Auto-publish at a fixed time | yes |
| Auto-archive / auto-expire | yes |
| Soft-delete on retention deadline | yes |
| Send an email at 9 AM tomorrow | not directly — use a job queue; the runner only mutates rows |
| Run a multi-step workflow at a deadline | not directly — hook into OnSetField to enqueue the workflow |
The runner is deliberately simple: timestamp + row-local change. For side-effecting work outside the database, use it as a trigger and delegate the actual work to a job queue.
Operational checklist
- One runner per cluster, started once, stopped on shutdown.
- Set
Intervalto the desired granularity —1mis plenty for most workflows; tighten if you have sub-minute deadlines. - Set
BatchSizeto a value the database can absorb in one transaction without blocking writers. 500 is a safe default; for very high-volume tables tune lower so each batch is shorter. - Use
OnDelete/OnSetFieldhooks for observability — emit events, increment metrics, log structured records. - For deployments with multiple app replicas, gate the runner to one
process or use
scheduled/jobsxto dispatch sweeps through your job queue. - Combine with
maniflex.WithDeletedAtfor the soft-delete-on-expiry pattern; the indexeddeleted_at IS NULLpredicate keeps the sweep query cheap as the table grows.
Audit Logging
db.AuditLog from the catalogue records every mutating operation to a
configured sink. Unlike Versioning — which writes to a
sibling table in the same database — audit log records are designed to be
shipped to an external system (a database table, a structured logger, a
SIEM). This page documents the record shape, the sink contract, and the
options that change what is captured.
Registering
The simplest registration captures the operation without per-field diffs:
import "github.com/xaleel/maniflex/middleware/db"
server.Pipeline.DB.Register(
db.AuditLog(mySink),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After),
)
mySink implements db.AuditSink:
type AuditSink interface {
Write(ctx context.Context, record AuditRecord) error
}
The audit record is emitted from a background goroutine with a 5-second timeout. Sink errors are logged but never fail the request — audit writes are fire-and-forget, by design. An audit pipeline that can fail the request is a liveness risk; an audit pipeline that occasionally drops a record is recoverable.
The record shape
Every audited write produces one AuditRecord:
type AuditRecord struct {
Timestamp time.Time `json:"timestamp"`
Model string `json:"model"`
Operation maniflex.Operation `json:"operation"`
ResourceID string `json:"resource_id,omitempty"`
Actor string `json:"actor,omitempty"`
TenantID string `json:"tenant_id,omitempty"`
RequestID string `json:"request_id,omitempty"`
TraceID string `json:"trace_id,omitempty"`
ServiceName string `json:"service_name,omitempty"`
Result any `json:"result,omitempty"`
Changes map[string]FieldChange `json:"changes,omitempty"`
}
type FieldChange struct {
From any `json:"from"`
To any `json:"to"`
}
| Field | Source |
|---|---|
Timestamp | UTC at the moment the record is built |
Model | ctx.Model.Name |
Operation | ctx.Operation |
ResourceID | ctx.ResourceID — empty on create until after the write |
Actor | ctx.Auth.UserID (empty for anonymous requests) |
TenantID | ctx.Auth.TenantID |
RequestID | ctx.RequestID (chi’s X-Request-Id) |
TraceID | ctx.TraceID (W3C traceparent) |
ServiceName | Config.ServiceName |
Result | ctx.DBResult — the row state returned by the adapter |
Changes | populated only when WithChanges() is set |
The minimum shape — Timestamp, Model, Operation, Actor,
RequestID — is enough to answer “who did what, when?” for compliance.
Adding Changes answers “what specifically was modified?”
Tracking changes
WithChanges() enables per-field diffs:
server.Pipeline.DB.Register(
db.AuditLog(sink, db.WithChanges()),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
// No AtPosition — defaults to Before.
)
Important: WithChanges() requires the middleware to run at
maniflex.Before (the default position), not maniflex.After. The middleware
needs to read the row state before the DB step writes, so the diff has
both sides.
With WithChanges(), Changes is populated as:
| Operation | Changes map |
|---|---|
| Create | {field: {from: null, to: new_value}} for each field returned by the write |
| Update | {field: {from: old, to: new}} for each changed field |
| Delete | {field: {from: value, to: null}} for each field on the pre-image |
Fields that didn’t change between pre-image and post-image are omitted. Fields excluded from the diff (see below) are also omitted.
The pre-image is read through ctx.ResolveResourceID(), so a scoped Singleton
diffs against the caller’s own row rather than against nothing. When that row has
not been provisioned yet, the update that provisions it is diffed as a create —
{from: null, to: value} — which is what it is.
Excluding fields from the diff
WithExcludeFields("password", "api_key", "session_token") keeps named
fields out of the Changes map:
server.Pipeline.DB.Register(
db.AuditLog(sink, db.WithChanges(), db.WithExcludeFields(
"password", "ssn", "api_token",
)),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
Use this for secrets that shouldn’t reach the audit pipeline even in
hashed form. Field names are matched against the DB column name (e.g.
api_token, not apiToken).
hidden, writeonly, and encrypted fields are not excluded
automatically — the diff is built from the raw row the write returns.
WithExcludeFields is the only mechanism that keeps a column out of the
Changes map, so name every sensitive column explicitly, including any
that carry those tags.
Common sinks
The sink interface is small enough to wire to anything that records structured events.
Database table
type DBAuditSink struct{ db *sql.DB }
func (s *DBAuditSink) Write(ctx context.Context, r db.AuditRecord) error {
changes, _ := json.Marshal(r.Changes)
_, err := s.db.ExecContext(ctx, `
INSERT INTO audit_logs
(timestamp, model, operation, resource_id, actor,
tenant_id, request_id, trace_id, service_name, changes)
VALUES (?,?,?,?,?,?,?,?,?,?)`,
r.Timestamp, r.Model, r.Operation, r.ResourceID, r.Actor,
r.TenantID, r.RequestID, r.TraceID, r.ServiceName, string(changes),
)
return err
}
A separate table — or a separate database — keeps audit volume from affecting the operational schema.
Structured logger
type LogSink struct{ log *slog.Logger }
func (s *LogSink) Write(ctx context.Context, r db.AuditRecord) error {
s.log.LogAttrs(ctx, slog.LevelInfo, "audit",
slog.String("model", r.Model),
slog.String("operation", string(r.Operation)),
slog.String("actor", r.Actor),
slog.String("request_id", r.RequestID),
slog.Any("changes", r.Changes),
)
return nil
}
The simplest sink — ships every audit event to the same log aggregator the rest of the app uses. Good for cold-storage compliance archives.
Async queue
For high-volume systems where the sink might back up, push records to a durable queue and process them out of band:
func (s *KafkaSink) Write(ctx context.Context, r db.AuditRecord) error {
b, _ := json.Marshal(r)
return s.producer.Produce(ctx, "audit-events", b)
}
A failed publish is logged but does not fail the request; the queue itself provides retry semantics.
Failure semantics
The middleware:
- Reads the pre-image (when
WithChanges()is set) before the DB step. - Calls
next(). - Checks the result. If
next()returned a non-nil error, the audit record is not written — we don’t audit failed operations. - Checks
ctx.Response. If status is>= 400, again no audit record. - Builds the record from the captured pre-image and
ctx.DBResult. - Spawns a goroutine that calls
sink.Writewith a 5-second background context.
This means:
- A failed write produces no audit entry. The framework’s other observability — request logs, error metrics — covers failed attempts.
- A successful write whose audit sink fails still succeeds. The audit record is lost.
- The audit write outlives the request context. A long-running audit write doesn’t block the HTTP response.
For at-least-once delivery, the sink must be backed by durable storage
(a database, a queue) — the in-process goroutine can be lost if the
process is killed before Write returns.
Audit log + versioning
Both record changes. Choose by where the records live and how they’re read:
| Concern | Audit log | Versioning |
|---|---|---|
| Storage | external sink | same DB, sibling table |
| Per-record reconstruction | no | yes (snapshot) |
| Compliance archive | yes | possible but awkward |
| Forensics across the whole system | yes | no — history is read one record at a time, through GET /:model/{id}/history |
| Cost | sink-dependent | one extra INSERT per write |
In a production system both are common: audit log feeds a SIEM for “who did what across everything”, versioning provides per-record history inside the app.
Operational checklist
- Pick a sink that matches your audit volume: database table for low volume, structured logs for medium, durable queue for high.
- Register at
maniflex.Beforewhen usingWithChanges(), atmaniflex.Afterotherwise. WithExcludeFieldsevery secret column — includingwriteonly,hidden, andencryptedones, which are not redacted from the diff automatically.- Treat the sink as best-effort. Don’t rely on the in-process goroutine for legal-grade audit retention; use a sink whose own storage is durable.
- Index your audit table on
timestamp,actor,model, andresource_id— they are the columns most queries filter by. - Restrict who can read the audit table. The diffs may contain values
the original endpoint hid behind
RedactField.
Idempotency
Idempotency middleware makes POST requests safe to retry over a flaky
network. The first request runs the pipeline normally; the response is
cached keyed on the Idempotency-Key header. Subsequent requests with
the same key and the same body short-circuit the pipeline and replay
the cached response.
The pattern is borrowed from Stripe’s API. This page documents the
shipped implementation in maniflex/middleware/idempotency.
The contract
An Idempotency-Key identifies one logical operation. Sending the
same key twice with the same body means “I’m retrying — give me the
same result as last time, do not run the operation again.” Sending the
same key with a different body means “I am confused about my own
state and you should refuse me.” Sending no key means “do not apply
idempotency to this request.”
Idempotency-Key: e3b0c442-98fc-1c14-9afb-...
The key is opaque to the framework — any string the client chooses. A UUID per logical operation is the conventional choice.
Registering
The middleware lives on the Deserialize step at maniflex.After
position, scoped to the operations that should be retryable:
import (
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/middleware/idempotency"
)
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{
Store: maniflex.NewMemoryCache(),
TTL: 24 * time.Hour,
}),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
Why After on Deserialize: the middleware needs ctx.RawBody to compute
the body hash, and the default Deserialize handler populates that. Running
after the default ensures the body is present.
Config:
| Field | Default | Purpose |
|---|---|---|
Store | required | the cache backend (anything implementing maniflex.CacheStore) |
TTL | 24h | how long a cached response is replayable |
KeyFunc | ctx.Auth.UserID then ctx.Request.RemoteAddr | derives the per-caller scope |
HeaderRequired | false | when true, requests without Idempotency-Key are rejected with 400 |
Locker | in-process singleflight | serialises concurrent first-misses on the same key. Supply a Redis-SETNX implementation of idempotency.Locker for multi-replica deployments; see Concurrent first-misses |
The cache key
The cache key is composed of four parts:
<KeyFunc(ctx)>:<model>:<operation>:<idempotency-key>
KeyFunc(ctx)— the per-caller scope. Defaults to the authenticated user ID, falling back to the remote IP for anonymous requests. Override to use the API token or any other identifier.model:operation— limits a key’s effect to one (model, op) pair. The sameIdempotency-Keycan be reused safely for, say,POST /api/ordersandPOST /api/refunds— they are different cache keys.idempotency-key— the client-supplied value.
The body hash is not part of the key — it is part of the cached
entry and compared on lookup. This is intentional: it lets the
middleware detect “same key, different body” and respond with
422 IDEMPOTENCY_KEY_REUSED.
What gets cached
Only successful responses (2xx). Failed responses are not cached, on purpose — retrying a failed write is the whole point of idempotency. A first attempt that 5xx’d should be re-run on the retry, not replayed.
The cached entry carries:
type Entry struct {
maniflex.APIResponse // StatusCode, Data, Error, Meta
BodyHash string
StoredAt time.Time
}
StatusCode— replayed verbatim.Data,Meta— replayed verbatim.BodyHash— SHA-256 ofctx.RawBody, used to detect body mismatch.
The replayed response carries the header Idempotent-Replayed: true
so the client can tell a replay from a fresh execution.
What happens on each call
| Request | Effect |
|---|---|
First request with Idempotency-Key: K | runs the pipeline; if 2xx, caches the response |
| Repeat with same key, same body | skips the pipeline; replays cached response; adds Idempotent-Replayed: true |
| Repeat with same key, different body | 422 IDEMPOTENCY_KEY_REUSED |
| Repeat with same key after TTL | runs the pipeline as if it were the first time |
Request with no Idempotency-Key | passes through (unless HeaderRequired is true) |
Choosing a store
Two implementations cover the common cases.
maniflex.NewMemoryCache
In-process, per-replica:
idempotency.Config{Store: maniflex.NewMemoryCache(), TTL: time.Hour}
Suitable for single-replica development. In a multi-replica deployment each replica has its own cache — a retry routed to a different replica gets a fresh run, defeating the purpose.
Redis (or any shared store)
There is no built-in Redis cache. maniflex.CacheStore is a three-method
interface you implement against whatever shared store you run:
type CacheStore interface {
Get(ctx context.Context, key string) (any, bool)
Set(ctx context.Context, key string, value any, ttl time.Duration)
Delete(ctx context.Context, key string)
}
The value the idempotency middleware stores is an idempotency.Entry, so a
cross-process backend must serialise it on Set and decode it back into an
idempotency.Entry on Get. A store that returns the raw bytes (or a decoded
map) instead makes the middleware abort with IDEMPOTENCY_CACHE_CORRUPT.
Register the middleware with your implementation exactly as with the in-process
cache:
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{
Store: myRedisCache, // your maniflex.CacheStore implementation
TTL: 24 * time.Hour,
}),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
Any backend that can store a TTL’d key/value (Redis, Memcached, DynamoDB with TTL) works, provided it round-trips the stored value’s type.
Concurrent first-misses
Two requests carrying the same Idempotency-Key and identical bodies that
arrive at exactly the same moment both miss the cache. Without
serialisation, both would run the full pipeline and both would write —
silently breaking the contract that one key represents one logical
operation. The middleware uses a Locker to serialise these first-misses.
The default Locker is in-process (singleflight-style): the second
goroutine blocks on a channel until the first releases, then re-checks
the cache and replays. This handles single-replica deployments correctly
out of the box.
For multi-replica deployments, supply Config.Locker with a backend that
synchronises across processes — typically Redis SETNX with a short TTL:
type Locker interface {
Acquire(ctx context.Context, key string, ttl time.Duration) (acquired bool, release func(), err error)
}
Acquire returns acquired=true to exactly one caller per key per TTL
window. The caller must invoke release once the cache entry is written
(or the work has failed). acquired=false means another caller holds (or
held) the lock — singleflight-style lockers block first, then return
false so the loser can replay from cache; SETNX-style lockers return
immediately and the loser re-checks the cache itself.
A Locker error (e.g. Redis network blip, request context cancelled)
fails open: the middleware runs the pipeline directly, mirroring the
pre-Locker behaviour rather than returning 503. This trades correctness
under partial outages for availability — appropriate for a feature whose
whole purpose is “make retries safe.”
Scoping to specific endpoints
For most APIs, idempotency belongs only on a handful of write endpoints —
payment, order placement, account creation. Scope with ForModel so
unrelated POST requests are unaffected:
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{Store: store}),
maniflex.ForModel("Payment", "Order"),
maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
The middleware passes through for unscoped requests with no measurable cost.
Requiring the header
For endpoints where retries without a key are dangerous, set
HeaderRequired: true:
idempotency.Middleware(idempotency.Config{
Store: store,
HeaderRequired: true,
})
A scoped registration is the right shape — make the header mandatory on payment but optional on lower-stakes resources:
server.Pipeline.Deserialize.Register(
idempotency.Middleware(idempotency.Config{
Store: store, HeaderRequired: true,
}),
maniflex.ForModel("Payment"), maniflex.ForOperation(maniflex.OpCreate),
maniflex.AtPosition(maniflex.After),
)
A missing header on a covered endpoint returns
400 IDEMPOTENCY_KEY_REQUIRED.
Use with custom actions
Action endpoints run a trimmed pipeline that skips Deserialize, so
pipeline-level idempotency does not apply automatically. To get the
same behaviour for an action, include the middleware in the action’s
Middleware list:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: placeOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
idempotency.Middleware(idempotency.Config{Store: store}),
},
})
An action skips the Deserialize step that populates ctx.RawBody, so here the
middleware reads the body itself (via ctx.EnsureRawBody) and restores it — the
action handler’s ctx.BindJSON then works exactly as it would without
idempotency, in whatever order. Before this was fixed (v0.2.3), the middleware
hashed an empty body in an action, so the same key, different body check
silently never fired and a reused key replayed the first response regardless of
the payload.
Edge cases
- Same key, different body. Returns
422 IDEMPOTENCY_KEY_REUSED. The contract is that one key represents one logical operation; reusing it for a different payload is almost certainly a client bug. - Request currently in flight when retry arrives. The default
in-process
Locker(singleflight-style) holds the second goroutine until the first finishes, at which point it replays from cache — only one pipeline execution runs per process. For multi-replica deployments, supply aConfig.Lockerthat uses RedisSETNXso two replicas don’t both run the pipeline. See Concurrent first-misses below. - Cache eviction before TTL. The retry runs the pipeline again. This is correct behaviour: the cache is a replay mechanism, not a deduplication mechanism. The application’s own uniqueness constraints handle “this thing was already created.”
- Operation that mutates external state. Idempotency caches the
response, not the side effect. A payment that charged a card once on
the first request will return the same
paidresponse on a retry without charging again — because the first request returned with the payment recorded as committed. The action handler is responsible for being idempotent against the external system; the middleware just prevents the framework from issuing duplicate writes.
Operational checklist
- One shared
Storeacross replicas. Don’t useMemoryCachein multi-replica deployments. - TTL longer than your client’s longest retry window. 24h is generous; for mobile clients on flaky networks, 7d is reasonable.
- Scope to the endpoints that benefit. Don’t blanket-apply.
- Pair with
HeaderRequired: trueon payment-like endpoints where client correctness depends on it. - Surface the
Idempotent-Replayedheader in client SDKs so consumers can tell a replay from a fresh execution. - Combine with a uniqueness constraint at the DB level for defence-in-depth. A retry that hits a different replica after cache eviction will run the pipeline; the DB constraint catches the duplicate.
Outbound Integrations: pkg/integration
maniflex/pkg/integration is a small toolkit for the integration patterns
that sit beside the framework: calling third-party HTTP APIs, polling
hardware, and receiving signed webhooks. It’s not a feature of the
framework — it’s three composable types you call from your own code.
Caller — JSON-over-HTTP with retry
import "github.com/xaleel/maniflex/pkg/integration"
var billing = &integration.Caller{
BaseURL: "https://api.billing.example.com",
Timeout: 10 * time.Second,
MaxRetry: 3,
Headers: map[string]string{
"Authorization": "Bearer " + secrets.Billing,
},
}
// Inside a handler / job / cron tick:
var resp struct {
InvoiceID string `json:"invoice_id"`
}
err := billing.Post(ctx, "/invoices", map[string]any{
"amount": total,
"patient": id,
}, &resp)
- Always JSON in, JSON out. Pass
out=nilto discard the response body. - Passing
[]byteorstringas the body skips JSON encoding — useful for upstreams that demand a specific wire format. - Retries fire on network errors, HTTP 5xx, and HTTP 429 with a
configurable backoff (
BackoffFn; default linear up to 1s). 4xx (other than 429) is final. - Non-2xx final responses surface as
*integration.ErrHTTPStatus. Useerrors.Asto inspectStatusCode, the parsed JSONBody, or the rawRawBody. - Always honours the request context — cancel ctx to abort an in-flight retry loop.
Poller — periodic background work
// Own a cancellable context and cancel it from your shutdown hook — there is
// no ShutdownContext() on Server; the poller stops when the context you pass
// is cancelled.
pollCtx, stopPolling := context.WithCancel(context.Background())
defer stopPolling() // or call it from your graceful-shutdown path
p := &integration.Poller{
Interval: 30 * time.Second,
Fn: func(ctx context.Context) error {
return terminal.SyncFingerprints(ctx)
},
}
go p.Start(pollCtx) // dies cleanly when stopPolling() is called
A failed tick is logged and the schedule continues — Poller is for
best-effort background work, not workflows where missing a tick is a bug.
For those, use pkg/jobs. Set RunOnStart: true to fire immediately rather
than waiting one Interval.
WebhookReceiver — HMAC-signed inbound
wh := &integration.WebhookReceiver{
Secret: secrets.PaymentWebhook,
Algorithm: "sha256", // or "sha512"
// Defaults are GitHub-style: X-Hub-Signature-256 + X-Event-Type
}
http.HandleFunc("/hooks/payments", wh.Handler(map[string]integration.WebhookHandler{
"payment.succeeded": handlePaymentSucceeded,
"payment.refunded": handlePaymentRefunded,
}))
The handler:
- Reads at most
MaxBodyBytes(default 1 MiB) from the request body. - Computes HMAC over the raw body and compares it constant-time to the
value in
HeaderKey. Commonalgo=hexprefixes (GitHub, Stripe) are tolerated. - Looks up the handler by the
EventHeaderKeyvalue. - Calls the handler with the raw body so it can decode whatever shape the upstream sends.
Failure modes:
- 400 — body read error
- 401 — missing or mismatching signature
- 404 — no handler registered for that event
- 500 — handler returned a non-nil error
WebhookReceiver.Handler panics if Secret is empty or Algorithm is
neither sha256 nor sha512 — both are configuration mistakes worth
catching at startup.
CSV / XLSX Export
Models can opt into an auto-generated export endpoint that streams CSV or XLSX of the same data the standard list endpoint returns, with the same filter and sort query parameters:
server.MustRegister(Invoice{}, maniflex.ModelConfig{
ExportEnabled: true,
MaxExportRows: 50_000, // optional cap; defaults to 100,000
})
This mounts:
GET /invoices/export → CSV (default)
GET /invoices/export?format=xlsx → XLSX
The endpoint reuses the full request pipeline — Auth, tenancy, soft-delete —
and middleware registered on ForOperation(maniflex.OpList) covers the
export too. An export is a list in another format, so whatever decides which
rows a caller may list decides which rows they may export:
server.Pipeline.Auth.Register(
auth.JWTAuth(secret, auth.JWTOptions{}),
maniflex.ForOperation(maniflex.OpList), // also covers OpExport
)
Naming OpExport as well is harmless but redundant. The implication runs one
way only: ForOperation(maniflex.OpExport) means the export alone, which is
what you want for export-specific middleware such as a rate limiter.
Before v0.2.5 this was not so — a middleware scoped to
OpListdid not run for exports, so tenancy written that way scoped the list and let the export return every tenant’s rows.
Query parameters
The same filter, sort, and include parameters work as on GET /invoices.
page and limit are ignored — the export reads every row that matches the
filters, up to MaxExportRows.
GET /invoices/export?filter=status:eq:posted&sort=created_at:desc
Column selection
Each model field appears as one column, named by its json tag (the same
identifier you see in API responses). Hidden, writeonly, and file-typed
columns are excluded:
| Field tag | In export? |
|---|---|
| (default) | yes |
hidden | no |
writeonly | no |
file | no (raw storage keys are useless to the recipient) |
A field masked for this caller by a response middleware
(response.RedactField) is
excluded too — column and values both, so the export does not advertise a field
it will not fill. Those tags are the same for every caller; a masking middleware
decides per request, and the export honours both.
Computed fields registered via Server.AddComputedField are not included
yet — they require runtime evaluation per row and the export is read-only at
the storage layer.
Row cap
MaxExportRows (default 100,000) bounds the result. Requests whose filtered
result would exceed the cap return 413 Request Entity Too Large with a
suggestion to tighten the filters; no partial data is written. Increase the
cap if you have a deliberate need; consider an async job (pkg/jobs) for
multi-million-row exports.
Concurrency cap
An export reads its whole result set into memory and holds it until the last
byte has been written to the client. MaxExportRows bounds one export’s row
count, but not how wide a row is, and not how many exports run at once — so
the memory an export costs is rows × width, and the memory the server costs
is that again times however many are in flight. A handful of concurrent exports
of a wide model is enough to exhaust the heap with every one of them
individually inside its cap.
Config.MaxConcurrentExports (default 4) bounds the product:
server := maniflex.New(maniflex.Config{
MaxConcurrentExports: 8, // 0 uses the default of 4; negative disables the limit
})
The limit is server-wide rather than per-model, because the heap it protects is
shared. An export arriving when every slot is taken is rejected immediately
with 503 Service Unavailable, code EXPORT_BUSY, and a Retry-After header —
it is not queued. Queuing would hold the connection open behind work that is
slow by nature, so the caller is told to come back instead.
The slot is taken before the pipeline runs and released when the request returns, so it spans the database read and the write — the whole window the rows are live. Note that this means the refusal happens before Auth: a request that would have failed authentication gets the 503 rather than a 401 while the server is saturated.
Set a negative value to remove the limit if you have admission control in front of the service.
Response shape
| Format | Content-Type | Content-Disposition |
|---|---|---|
| CSV | text/csv; charset=utf-8 | attachment; filename="<model>-<ts>.csv" |
| XLSX | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | attachment; filename="<model>-<ts>.xlsx" |
The XLSX writer produces a minimal-but-valid .xlsx workbook with one sheet
named Data. Strings are written as inline string cells — no shared-string
table, no styles, no formulas — which keeps the writer dependency-free
(stdlib archive/zip and encoding/xml only) at the cost of slightly
larger files than a heavyweight library would produce.
Unsupported ?format= values return 400 INVALID_FORMAT.
What’s not in v1
- Async / job-backed exports for multi-million-row datasets. Today’s
endpoint streams synchronously; the upper bound is
MaxExportRows. - Constant-memory exports. The CSV and XLSX writers hold nothing per row —
they serialise and write one row at a time — but the rows themselves are read
from the database in one go before the write begins, so one export still costs
memory proportional to its result set.
MaxConcurrentExportsbounds how many of those can coexist; a true row-by-row cursor would remove the per-export cost too, at the price of pinning a database connection for the length of the client’s download and turning a mid-stream database error into a silently truncated file. - Localised column headers. Headers always use the JSON field name.
- Style hints (number formats, frozen headers, autofilter). The XLSX is intentionally plain.
Open an issue if any of these matter for your use case.
Auth & Security Hardening
The defaults are safe to deploy, but production APIs benefit from a few extra layers. This page collects the practical checklist.
Authentication
- Use
auth.JWTAuthwith an asymmetric algorithm (RS256/ES256) when tokens are issued by an external provider. SymmetricHS256works when the signing service and the API share infrastructure. - Use
auth.JWKSAuth(jwksURL, opts…)when the issuer publishes a rotating JWK Set (/.well-known/jwks.json). It fetches and caches the keys, selects the signing key by the token’skid, and refetches on an unknownkidso key rotation needs no redeploy. AllJWTOptions(Issuer, Audience, claim mappings, ClockSkew) apply. Prefer this over pinning a single staticPublicKeyagainst an issuer that rotates. RSA (RS256/384/512) and EC (ES256/384/512) supported. - Set
JWTOptions.IssuerandAudienceso tokens issued for another audience are rejected. - Set
JWTOptions.TenantClaimfor multi-tenant APIs — the verified value ends up onctx.Auth.TenantIDand feedsdb.Tenancy. - Never accept anonymous writes by default. Register
auth.JWTAuth(orauth.APIKeyAuth) on the Auth step scoped toOpCreate,OpUpdate,OpDelete. Useauth.AllowPublicReadwhen reads are truly public.
server.Pipeline.Auth.Register(auth.AllowPublicRead())
server.Pipeline.Auth.Register(auth.JWTAuth(secret, auth.JWTOptions{
Issuer: "https://accounts.example.com",
Audience: "https://api.example.com",
TenantClaim: "org_id",
}), maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete))
Verifying tokens from an issuer that rotates its signing keys via JWKS:
server.Pipeline.Auth.Register(auth.JWKSAuth(
"https://accounts.example.com/.well-known/jwks.json",
auth.JWTOptions{
Issuer: "https://accounts.example.com",
Audience: "https://api.example.com",
TenantClaim: "org_id",
}))
Authorisation
- Gate sensitive operations with
auth.RequireRole. Don’t rely on the UI to hide them. - Use
db.Tenancyordb.ForceFilterfor row-level scoping. These run on the DB step so they apply to lists, reads, and writes uniformly — UI code cannot accidentally bypass them. - Strip privileged values with
validate.ForbiddenValuesfor role fields and similar — prevent a normal user from promoting themselves by including"role": "admin"in a payload.
Secrets and PII
- Hash passwords with
service.HashField, never store them raw. - Use
writeonlyon credential fields so they are accepted on input but never returned in responses. - Encrypt sensitive columns with
mfx:"encrypted"and a configuredKeyProvider. Pair with thekey:sub-option for per-domain keys. - Redact in responses with
response.RedactFieldwhen a column is visible to some callers and hidden from others. - Gate writes with
validate.FieldRolewhen a column is writable by some callers and not others (readonlyis all-or-nothing). Without it, a privileged field needs its own endpoint to keep it off the general PATCH.
Input
- Set
Config.QueryTimeoutso a slow query can’t tie up a connection indefinitely. - Cap body sizes with
body.MaxBodySizewhere you know the upper bound. The default 4 MB limit catches accidents, but a 10 KB endpoint should enforce 10 KB. - Strip unknown fields with
body.StripUnknownFieldsin environments where you want a strict contract — every accepted field appears on the model. - Validate beyond tags with
validate.RegexField,validate.UniqueField, andvalidate.CrossFieldValidate. The built-inmfx:rules cover the common cases; everything else belongs in middleware.
Output
- Set security headers globally via
response.AddHeader:Strict-Transport-Security,X-Content-Type-Options,Referrer-Policy. - List CORS origins explicitly with
response.CORSHeaders(origins...)— origins are required (there is no permissive wildcard default; it panics if you pass none), and"*"cannot be combined with credentials. - Cap rate-sensitive endpoints with
db.RateLimitso password resets and similar can’t be brute-forced.
Transport
- Terminate TLS at the load balancer or reverse proxy, not in the maniflex process. The framework is HTTP/1.1 + HTTP/2 ready.
- Set
Config.TrustProxyHeaders: trueonly when behind a trusted proxy. It is off by default: the client IP is the direct TCP peer, so a caller cannot forge it. When on, the IP is read fromX-Forwarded-For/X-Real-IP, which is only safe if the proxy overwrites (not appends) any inbound value the client sent. Every IP-keyed feature —db.RateLimit, idempotency scoping, and read-audit records — depends on this being correct; leaving it off while directly internet-facing keeps per-IP limits and audit logs honest. - Set
Config.PathPrefixto a non-default value if the proxy mounts the API at a custom path. Don’t rewrite paths inside the application.
Operations
- Use a JSON-emitting
sloghandler in production so logs are structured and ingestable by your aggregator. - Set
Config.ServiceName— every log line and audit record carries it. - Enable
Config.HealthCheckDBfor Kubernetes readiness probes; tuneConfig.HealthTimeoutshorter than the probe timeout. - Use
Config.PanicLoggerto route panics to a different sink than the rest of the framework logs, so they are easier to alert on.
Audit
- Register
db.AuditLogatmaniflex.Afterfor mutating operations. The records carry actor, model, operation, and a diff of the affected row. - Use
maniflex.ModelConfig{Versioned: true}on sensitive models. Every change writes a row to a sibling{model}_historytable.
Checklist
A reasonable production stack:
// Auth
server.Pipeline.Auth.Register(auth.JWTAuth(secret, jwtOpts))
server.Pipeline.Auth.Register(auth.RequireRole("admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpDelete))
// Body
server.Pipeline.Deserialize.Register(body.MaxBodySize(32<<10),
maniflex.ForModel("PasswordReset"))
server.Pipeline.Validate.Register(body.StripUnknownFields())
// DB
server.Pipeline.DB.Register(db.Tenancy("org_id", tenantFromAuth))
server.Pipeline.DB.Register(db.RateLimit(db.RateLimitConfig{
RequestsPerMinute: 10,
Key: keyByIP,
}), maniflex.ForModel("PasswordReset"))
server.Pipeline.DB.Register(db.AuditLog(auditSink),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
maniflex.AtPosition(maniflex.After))
// Response
server.Pipeline.Response.Register(response.CORSHeaders("https://app.example.com"))
server.Pipeline.Response.Register(
response.AddHeader("Strict-Transport-Security", "max-age=63072000"))
server.Pipeline.Response.Register(response.Logging(slog.Default()),
maniflex.AtPosition(maniflex.After))
PostgreSQL in Production
The maniflex/db/postgres adapter is the recommended backend for any deployment
beyond a single process. This page collects production-relevant details that
go beyond the Database Backends overview.
Opening the adapter
import "github.com/xaleel/maniflex/db/postgres"
// Single primary (no replica) — pass "" for the read DSN.
db, err := postgres.Open(
os.Getenv("DB_WRITE_URL"), // primary / write DSN (required)
os.Getenv("DB_READ_URL"), // replica / read DSN ("" → reuse the primary)
server.Registry(),
)
if err != nil {
log.Fatal(err)
}
server.SetDB(db)
Open takes the write DSN, the read DSN, and the registry. The write DSN is
required; pass "" for the read DSN to route reads at the primary. Both are
standard libpq connection strings or URLs
(postgres://user:pass@host:5432/dbname?sslmode=require). MustOpen is the
panic-on-error variant for package-level initialisation.
Tuning pools and session settings
Open applies production defaults. To override them, use OpenWithConfig,
which takes a separate PoolConfig for the write and read pools plus one
SessionConfig. Any zero-value field is replaced by the default Open uses, so
you set only what you want to change:
schema := "orders"
db, err := postgres.OpenWithConfig(
writeDSN, readDSN, server.Registry(),
postgres.PoolConfig{MaxOpenConns: 20}, // write pool
postgres.PoolConfig{MaxOpenConns: 60}, // read pool
postgres.SessionConfig{
StatementTimeout: 10 * time.Second,
ApplicationName: "orders-api",
SchemaName: &schema, // search_path; auto-created on connect if absent
},
)
Connection-pool tuning
The defaults suit a small service. OpenWithConfig exposes them as PoolConfig
fields, set independently for the write and read pools:
PoolConfig field | Default (write / read) | Considerations |
|---|---|---|
MaxOpenConns | 20 / 40 | keep the sum ≤ max_connections / number_of_app_instances |
MaxIdleConns | half of MaxOpenConns | enough open to absorb bursts, not so many you waste server slots |
ConnMaxLifetime | 30 min | rotate connections to pick up failover or DNS changes |
ConnMaxIdleTime | 5 min | close idle connections so PgBouncer can recycle |
If you front Postgres with PgBouncer in transaction-pooling mode:
- Set
MaxOpenConnson the client to roughly match the bouncer’sdefault_pool_size. - Avoid prepared statements that span transactions (the framework doesn’t use any; your raw queries should follow suit).
LISTEN/NOTIFYis not supported under transaction pooling — use the event-bus satellites instead.
Session settings
SessionConfig carries session-level parameters the adapter re-applies (SET …)
on every new physical connection — Postgres does not persist them across
reconnects, so they must be set per connection:
SessionConfig field | Default | Effect |
|---|---|---|
StatementTimeout | 30s | cancels any statement that runs longer (0 = server default) |
LockTimeout | 5s | aborts a statement that waits too long for a lock |
IdleInTransactionTimeout | 60s | aborts transactions left idle — guards against hung app code |
ApplicationName | maniflex | shown in pg_stat_activity and server logs |
TimeZone | UTC | session time zone for TIMESTAMPTZ rendering |
SchemaName | public | schema set as search_path (see below) |
Schema isolation (search_path)
By default the adapter operates in the public schema. Set
SessionConfig.SchemaName to scope every connection to a dedicated schema via
SET search_path — handy for multi-tenant deployments or co-locating several
apps in one database. The schema is created on connect when it does not yet
exist (CREATE SCHEMA IF NOT EXISTS), so AutoMigrate has somewhere to place
its tables; an existing schema is left untouched (a role with USAGE but not
CREATE still connects). The name must be a plain SQL identifier
([A-Za-z_][A-Za-z0-9_$]*); public is assumed to always exist and is never
re-created.
Read replicas
When a read DSN is supplied, OpList and OpRead operations are routed to the
read pool; everything else uses the write pool. Trade-offs:
- Reads inside an active write transaction route to the write pool, even when a read replica is configured — read-your-writes is preserved.
- Pure read endpoints get the replica’s spare capacity without any code change.
- The application sees the replica’s normal lag for non-transactional reads. If a workflow depends on read-your-writes outside a transaction, run it inside a write transaction so the read lands on the primary.
FOR UPDATE and pessimistic locking
ctx.LockForUpdate translates to SELECT … FOR UPDATE on Postgres. The lock
is held until the enclosing transaction commits or rolls back. Typical use:
row, err := ctx.LockForUpdate("StockBalance", stockID)
if err != nil {
return err
}
if row["quantity"].(int64) < 1 {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "no inventory")
return nil
}
// safe to subtract — concurrent writers are blocked
Combine with maniflex.WithTransaction (or manual BeginTx) so the lock has a
transaction to scope it.
Isolation levels
maniflex.WithTransaction(&maniflex.TxOptions{Isolation: sql.LevelSerializable}) opens
the request in SERIALIZABLE isolation. Postgres serialisation failures
produce 40001 errors. Note that NormalizeError maps only the constraint
codes 23505 / 23502 / 23503 to *maniflex.ErrConstraint; a 40001
serialisation failure is not normalised — it propagates as a generic error
and surfaces as a 500. If you need transparent retry on serialisation
failures, detect the 40001 SQLSTATE yourself (e.g. in an action handler or a
custom middleware) and retry the transaction.
Most APIs do fine with the default READ COMMITTED plus LockForUpdate on
the contested rows; reach for SERIALIZABLE when the contention pattern is
more complex than a single row.
AutoMigrate at scale
AutoMigrate is enabled by default. For larger production databases, prefer:
server := maniflex.New(maniflex.Config{DisableAutoMigrate: true, ...})
…and run schema changes through a dedicated migration tool (sqlc-migrate, golang-migrate, Atlas, etc.). The framework’s auto-migrator is conservative — it never drops columns and emits straightforward DDL — but coordinating schema changes across replicas, dropped indexes, and rolling deploys is the migration tool’s job.
If you keep AutoMigrate enabled, run the first instance to completion
before scaling out; later instances will see all-up-to-date schema and skip
the work.
TLS and connectivity
- Use
sslmode=require(or stricter) on both the write and read DSN for any production connection. The driver respects the URL parameter. - For Cloud SQL / RDS, the connection string is generated by the cloud console; copy it verbatim and store it as a secret.
- Resolve DNS lookups inside the process — don’t pre-resolve at process
start. The
ConnMaxLifetimesetting then picks up the new endpoint automatically during failover.
Observability
- The adapter exposes pool statistics via
sql.DB.Stats(); export them with theresponse.Metricsmiddleware or a separate collector. - Set
Config.QueryTimeoutto bound slow queries; offending requests return504 TIMEOUTrather than holding a connection open. - Postgres logs (
log_min_duration_statement) andpg_stat_statementsare the canonical way to identify slow queries; the framework does not duplicate that.
Example 3: Order Processing System
This example assembles every advanced topic into one application — actions, raw queries, a custom aggregate endpoint, a transactional outbox, and a background worker. The domain is small (orders and inventory) so the integration is visible.
Domain
type Product struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required,filterable,sortable"`
Price float64 `json:"price" mfx:"required,min:0"`
Stock int64 `json:"stock" mfx:"required,min:0,filterable"`
}
type Order struct {
maniflex.BaseModel
maniflex.WithDeletedAt
CustomerID string `json:"customer_id" mfx:"required,filterable,immutable"`
Total float64 `json:"total" mfx:"required,min:0,filterable,sortable"`
Status string `json:"status" mfx:"required,enum:pending|paid|shipped|cancelled,default:pending,filterable,sortable"`
Lines []OrderLine `json:"lines,omitempty"`
}
type OrderLine struct {
maniflex.BaseModel
OrderID string `json:"order_id" mfx:"required,filterable,immutable"`
ProductID string `json:"product_id" mfx:"required,filterable,immutable"`
Quantity int64 `json:"quantity" mfx:"required,min:1"`
UnitPrice float64 `json:"unit_price" mfx:"required,min:0"`
}
// Transactional outbox row — appended in the same transaction as the order.
type OutboxEvent struct {
maniflex.BaseModel
Kind string `json:"kind" mfx:"required,filterable"`
Payload map[string]any `json:"payload" mfx:"required"`
Status string `json:"status" mfx:"required,enum:pending|done|failed,default:pending,filterable"`
ErrorMsg string `json:"error_msg" mfx:"filterable"`
}
Action: place an order atomically
POST /orders would normally just insert a row. Real order placement needs:
locking the products, decrementing stock, creating the order and lines,
queueing payment — all in one transaction. An action endpoint
handles this explicitly:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: placeOrder,
Middleware: []maniflex.MiddlewareFunc{auth.JWTAuth(secret)},
})
func placeOrder(ctx *maniflex.ServerContext) error {
var req struct {
Lines []struct {
ProductID string `json:"product_id"`
Quantity int64 `json:"quantity"`
} `json:"lines"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
// Reserve stock under a row lock for each product.
var total float64
type line struct {
productID string
qty int64
unit float64
}
var lines []line
for _, l := range req.Lines {
p, err := ctx.LockForUpdate("Product", l.ProductID)
if err != nil {
return err
}
stock := p["stock"].(int64)
if stock < l.Quantity {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK",
fmt.Sprintf("product %s has %d in stock", l.ProductID, stock))
return nil
}
if _, err := ctx.GetModel("Product").Update(l.ProductID, map[string]any{
"stock": stock - l.Quantity,
}); err != nil {
return err
}
unit := p["price"].(float64)
total += unit * float64(l.Quantity)
lines = append(lines, line{l.ProductID, l.Quantity, unit})
}
order, err := ctx.GetModel("Order").Create(map[string]any{
"customer_id": ctx.Auth.UserID,
"total": total,
"status": "pending",
})
if err != nil {
return err
}
for _, l := range lines {
if _, err := ctx.GetModel("OrderLine").Create(map[string]any{
"order_id": order["id"],
"product_id": l.productID,
"quantity": l.qty,
"unit_price": l.unit,
}); err != nil {
return err
}
}
// Outbox row — picked up by the background worker after commit.
if _, err := ctx.GetModel("OutboxEvent").Create(map[string]any{
"kind": "charge-payment",
"payload": map[string]any{"order_id": order["id"], "amount": total},
"status": "pending",
}); err != nil {
return err
}
if err := tx.Commit(); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusCreated,
Data: order,
}
return nil
}
If any step fails, the deferred Rollback reverts the order, the lines, and
the stock decrement together.
Revenue report: a custom action
The dashboard wants GET /revenue to return revenue per day. There is no
SQL-backed “query model”; mount a custom action whose handler runs
the aggregate with ctx.RawQuery:
server.Action(maniflex.ActionConfig{
Method: "GET",
Path: "/revenue",
Handler: func(ctx *maniflex.ServerContext) error {
rows, err := ctx.RawQuery(`
SELECT date(created_at) AS day, SUM(total) AS total
FROM orders
WHERE status IN ('paid', 'shipped')
GROUP BY day
ORDER BY day DESC
LIMIT 30`)
if err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{StatusCode: http.StatusOK, Data: rows}
return nil
},
})
Clients call GET /revenue and get the last 30 days of revenue. For plain
counts/sums over a registered model, the built-in GET /{model}/aggregate
endpoint avoids hand-written SQL — see Raw Queries & Aggregates.
Background worker: process the outbox
A separate goroutine (or process) sweeps OutboxEvent, processes each event,
and updates its status. The worker uses the same registered models:
func runOutboxWorker(server *maniflex.Server) {
// Background workers have no ServerContext — build one with NewBackground,
// then use ctx.GetModel (or the typed maniflex.List/Read helpers).
bg := maniflex.NewBackground(context.Background(), server.DB(), server.Registry())
events := bg.GetModel("OutboxEvent")
for range time.Tick(2 * time.Second) {
rows, _ := events.List(&maniflex.QueryParams{
Filters: []*maniflex.FilterExpr{{
Field: "status", Operator: maniflex.OpEq, Value: "pending",
}},
Limit: 20,
})
for _, ev := range rows {
if err := process(ev); err != nil {
events.Update(ev["id"].(string), map[string]any{
"status": "failed",
"error_msg": err.Error(),
})
continue
}
events.Update(ev["id"].(string), map[string]any{"status": "done"})
}
}
}
The worker is part of the same binary in this example. In production, a
satellite from jobs/redis (see Events & Background Jobs)
replaces the polling loop with a durable queue and at-least-once delivery.
What this example tied together
- Actions for endpoints that don’t fit standard CRUD (
/orders/place). LockForUpdateto safely decrement stock under contention.ctx.BeginTxso the order, lines, and stock change commit atomically.- The transactional outbox pattern for crossing the boundary between the request transaction and external side effects.
- A custom action for the revenue report — a read endpoint built from raw
SQL (
ctx.RawQuery). - A background worker consuming a registered model as its work queue.
Each piece is documented on its own page in the Advanced section: actions, raw queries, batch-saga, and events-jobs.
1. Overview & Scaffolding
This is the start of a ten-part walkthrough. The end product is a small but realistic bookstore API — users sign in, browse books and reviews, place orders, and the system notifies them when an order ships. Each step introduces one capability of the framework; nothing is hand-waved.
Where the reference pages describe a feature in isolation, the tutorial shows how the features compose in a real application.
What you’ll build
bookstore — an HTTP API with:
| Endpoint family | Capability |
|---|---|
/api/users | sign-up, JWT auth, role-based access |
/api/books, /api/authors, /api/genres | the catalogue, with relations and full querying |
/api/reviews | per-book ratings with custom validation |
Book cover upload via multipart/form-data | a file field |
POST /api/orders/place | a transactional action with stock locking |
| Outbox + background worker | email a receipt after each order |
By the end of part 10 the same code base will be deployed to production with PostgreSQL, env-driven configuration, and a health probe.
Prerequisites
- Go 1.25 or newer.
- A text editor and a terminal. Nothing else; the development database is pure-Go SQLite, no CGo needed.
Project layout
The app follows the layer-based layout described in App Anatomy. It grows over the tutorial; this is the shape at the end:
bookstore/
├── go.mod
├── main.go # wiring: create, register, set DB, serve
├── config.go # maniflex.Config assembly
├── models/ # one file per model
│ ├── user.go
│ ├── book.go
│ ├── author.go
│ ├── genre.go
│ ├── review.go
│ ├── order.go
│ └── outbox.go
├── middleware/ # custom middleware
│ ├── auth.go
│ ├── validate.go
│ └── register.go
├── actions/ # custom endpoints
│ └── orders.go
├── jobs/ # background workers
│ └── outbox.go
└── static/
└── openapi.html # bundled API viewer
Bootstrap
Create the directory, initialise a module, and add the framework:
mkdir bookstore && cd bookstore
go mod init bookstore
go get github.com/xaleel/maniflex github.com/xaleel/maniflex/db/sqlite
main.go starts as the smallest maniflex app:
package main
import (
"log"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
)
func main() {
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
})
db, err := sqlite.Open("./bookstore.db", server.Registry())
if err != nil {
log.Fatal(err)
}
defer db.Close()
server.SetDB(db)
log.Fatal(server.Start())
}
Run it:
go run .
The server starts on :8080. Nothing is registered yet, so the only endpoint
that responds is /health — but GET /api/openapi.json already serves a
valid (empty) OpenAPI document.
Why the four-step shape
The framework’s lifecycle never deviates from the same four steps — create → register → set DB → serve. Everything we add in the next nine parts plugs into one of those steps:
- Create:
maniflex.Configgrows with logger, file storage, query timeout, and so on. - Register: more models, each carrying their
mfx:tags. - Set DB: SQLite for development, PostgreSQL by part 10.
- Serve: more pipeline middleware, but
Start()itself stays the same.
The structure of main.go will not change between this part and part 10. The
file simply grows new lines.
Next
In Part 2 — Users & Auth we add the User model, a
sign-up endpoint, and JWT-based authentication. By the end of part 2 every
write request will require a valid token.
2. Users & Auth
We start with the User model and the auth layer. By the end of this part,
the API has a sign-up endpoint, password hashing, JWT-based authentication on
all writes, and a role-based admin gate on user deletion.
The model
Create models/user.go:
package models
import "github.com/xaleel/maniflex"
type User struct {
maniflex.BaseModel
Email string `json:"email" mfx:"required,filterable,unique,immutable"`
Password string `json:"password" mfx:"required,writeonly,min:8"`
Name string `json:"name" mfx:"required,filterable,sortable"`
Role string `json:"role" mfx:"required,enum:admin|customer,default:customer,filterable"`
}
A few tag choices to notice:
emailisuniqueandimmutable— once a user signs up, the address is the account identity.passwordiswriteonlyso it is accepted on input but never appears in responses, andmin:8enforces a minimum length.roleis an enum with a safe default; we’ll gateadminwrites separately in middleware.
Register it from main.go:
import "bookstore/models"
server.MustRegister(models.User{})
That alone gives you POST /api/users (sign-up), GET /api/users/{id},
PATCH /api/users/{id}, DELETE /api/users/{id}, and GET /api/users. But
right now anyone can call any of them — we need to hash passwords on the way
in and gate the writes.
Hashing passwords
Add maniflex/middleware/service/bcrypt:
go get github.com/xaleel/maniflex/middleware/service/bcrypt
Then register the hashing middleware on the Service step, scoped to User
create and update:
import (
"github.com/xaleel/maniflex/middleware/service"
"github.com/xaleel/maniflex/middleware/service/bcrypt"
)
server.Pipeline.Service.Register(
service.HashField("password", bcrypt.Hasher()),
maniflex.ForModel("User"),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate),
)
The middleware reads the password field (ctx.Field), replaces it with the
bcrypt hash via ctx.SetField, and lets the DB step write the hash. Nothing else in the application
needs to know that the column is hashed.
JWT authentication
Pull in maniflex/middleware/auth:
go get github.com/xaleel/maniflex/middleware/auth
Register JWTAuth on the Auth step, scoped to writes — we’ll let reads stay
public for now. Auth scoping (ForModel / ForOperation) is
inclusion-only: a middleware runs only for the models and operations you
scope it to, so anything you don’t scope onto stays public.
import "github.com/xaleel/maniflex/middleware/auth"
// Protect updates and deletes on every model…
server.Pipeline.Auth.Register(
auth.JWTAuth("dev-secret", auth.JWTOptions{Issuer: "bookstore"}),
maniflex.ForOperation(maniflex.OpUpdate, maniflex.OpDelete),
)
// …and protect creates only where a session is required. "User" is deliberately
// left out, so POST /api/users (sign-up) needs no token.
server.Pipeline.Auth.Register(
auth.JWTAuth("dev-secret", auth.JWTOptions{Issuer: "bookstore"}),
maniflex.ForModel("Book"),
maniflex.ForOperation(maniflex.OpCreate),
)
JWTAuth verifies the Authorization: Bearer <token> header, parses the
claims, and populates ctx.Auth with the user ID and roles. Tokens fail with
401 UNAUTHORIZED; missing tokens fail the same way.
Sign-up (POST /api/users) is a create on User — a model we never scoped
auth onto — so it stays public with no extra middleware. There is no
AllowPublicWrite helper: public access always comes from not scoping the
authenticator onto an operation, because scoping is inclusion-only.
Role-gated deletes
Only admins should be able to delete users. auth.RequireRole does exactly
that:
server.Pipeline.Auth.Register(
auth.RequireRole("admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpDelete),
)
It runs after JWTAuth, so by the time it fires ctx.Auth.Roles is
populated. Non-admin users receive 403 FORBIDDEN.
Issuing tokens
JWTAuth only verifies tokens — it does not issue them. For development we
add a tiny token endpoint as a custom action:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/auth/login",
Handler: login,
})
func login(ctx *maniflex.ServerContext) error {
var req struct {
Email string `json:"email"`
Password string `json:"password"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
rows, err := ctx.RawQuery(
`SELECT id, password, role FROM users WHERE email = ?`, req.Email,
)
if err != nil || len(rows) == 0 {
ctx.Abort(http.StatusUnauthorized, "INVALID_CREDENTIALS", "bad email or password")
return nil
}
user := rows[0]
if !checkBcrypt(user["password"].(string), req.Password) {
ctx.Abort(http.StatusUnauthorized, "INVALID_CREDENTIALS", "bad email or password")
return nil
}
token := signJWT("dev-secret", user["id"].(string), []string{user["role"].(string)})
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusOK,
Data: map[string]any{"token": token},
}
return nil
}
signJWT and checkBcrypt are small helpers built on github.com/golang-jwt/jwt/v5
and maniflex/middleware/service/bcrypt. In production this endpoint would
issue a refresh token too — for now, a single bearer token is enough.
Trying it out
# Sign up
curl -X POST localhost:8080/api/users \
-H 'Content-Type: application/json' \
-d '{"email":"[email protected]","password":"hunter22!","name":"Alice"}'
# Log in
TOKEN=$(curl -s -X POST localhost:8080/api/auth/login \
-H 'Content-Type: application/json' \
-d '{"email":"[email protected]","password":"hunter22!"}' \
| jq -r .data.token)
# Authenticated read (lists are public, but writes need the token)
curl -X PATCH localhost:8080/api/users/<id> \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"name":"Alice A."}'
What we built
| Capability | How |
|---|---|
| Sign-up | POST /api/users stays public (auth not scoped onto it) |
| Password hashing | service.HashField("password", bcrypt.Hasher()) on the Service step |
| Bearer-token auth on writes | auth.JWTAuth on the Auth step |
| Admin-only delete | auth.RequireRole("admin") |
| Token issuance | /api/auth/login action |
Next
In Part 3 — Modeling Domain Entities & Relations we add
the catalogue: Author, Genre, Book, and Review, wired up with
BelongsTo, HasMany, and many-to-many relations.
3. Modeling Domain Entities & Relations
With users in place, we model the catalogue. A book belongs to one author and many genres, and accumulates many reviews — three relations, two flavours.
The catalogue
Create the models. Each one lives in its own file under models/.
// models/author.go
type Author struct {
maniflex.BaseModel
Name string `json:"name" mfx:"required,filterable,sortable"`
Bio string `json:"bio"`
Books []Book `json:"books,omitempty"` // HasMany
}
// models/genre.go
type Genre struct {
maniflex.BaseModel
Label string `json:"label" mfx:"required,filterable,sortable,unique"`
Books []Book `json:"books,omitempty" mfx:"through:BookGenre"`
}
// models/book.go
type Book struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
ISBN string `json:"isbn" mfx:"required,filterable,unique"`
Price float64 `json:"price" mfx:"required,min:0,filterable,sortable"`
Stock int64 `json:"stock" mfx:"required,min:0,filterable"`
PublishedAt string `json:"published_at" mfx:"filterable,sortable"`
AuthorID string `json:"author_id" mfx:"required,filterable"` // BelongsTo Author
Genres []Genre `json:"genres,omitempty" mfx:"through:BookGenre"` // ManyToMany
Reviews []Review `json:"reviews,omitempty"` // HasMany
}
// models/book_genre.go — the junction model for Book ↔ Genre.
type BookGenre struct {
maniflex.BaseModel
BookID string `json:"book_id" mfx:"required,filterable,immutable"`
GenreID string `json:"genre_id" mfx:"required,filterable,immutable"`
}
// models/review.go
type Review struct {
maniflex.BaseModel
maniflex.WithDeletedAt
BookID string `json:"book_id" mfx:"required,filterable,immutable"` // BelongsTo Book
UserID string `json:"user_id" mfx:"required,filterable,immutable"` // BelongsTo User
Rating int `json:"rating" mfx:"required,min:1,max:5,filterable,sortable"`
Body string `json:"body" mfx:"required"`
}
Three relation styles in one place:
- BelongsTo (convention) —
Book.AuthorID→Author,Review.BookID→Book,Review.UserID→User. No tags required; the framework reads theIDsuffix. - HasMany —
Author.Books,Book.Reviews. A slice of the related struct, not a column on this table. - ManyToMany —
Book.Genres↔Genre.Booksthrough the explicitBookGenrejunction model. Thethrough:tag names the junction; both sides declare it.
Registering
All five models go to MustRegister:
server.MustRegister(
models.User{},
models.Author{},
models.Genre{},
models.Book{},
models.BookGenre{},
models.Review{},
)
AutoMigrate creates the tables. The junction book_genres carries
book_id and genre_id; the framework wires the Book ↔ Genre relation
from the through: tag.
Trying the relations
Create an author, a genre, and a book:
AUTH=$(curl -s -X POST localhost:8080/api/authors -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"name":"Ursula K. Le Guin"}' | jq -r .data.id)
SCIFI=$(curl -s -X POST localhost:8080/api/genres -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' -d '{"label":"Science Fiction"}' | jq -r .data.id)
BOOK=$(curl -s -X POST localhost:8080/api/books -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"title\":\"The Dispossessed\",\"isbn\":\"9780061054884\",\"price\":12.99,\"stock\":10,\"author_id\":\"$AUTH\"}" \
| jq -r .data.id)
# Tag it as sci-fi via the junction model.
curl -X POST localhost:8080/api/book_genres -H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"book_id\":\"$BOOK\",\"genre_id\":\"$SCIFI\"}"
Now include the related rows in a read:
curl "localhost:8080/api/books/$BOOK?include=author,genres,reviews"
The response carries author (a single object), genres (an array), and
reviews (an empty array for now). Each include is a separate query against
the related table.
Filtering through relations
Filters can traverse relations using dot notation. The related field must be
filterable:
# All books written by anyone whose name starts with "Ursula"
curl "localhost:8080/api/books?filter=author.name:ilike:Ursula%25"
# All books in the "Science Fiction" genre
curl "localhost:8080/api/books?filter=genres.label:eq:Science+Fiction&include=genres"
Filtering does not require an include; including merely returns the related
rows. You can join one and not the other freely.
Cascading deletes
A deleted author should not orphan books. Update the Book.AuthorID tag to
declare a cascade:
AuthorID string `json:"author_id" mfx:"required,filterable,relation:Author;onDelete:cascade"`
Author Author `json:"author,omitempty"`
The companion Author field is now needed because the explicit relation:
tag must name a companion field of the target type. We also gain a slightly
better OpenAPI: the spec carries the relation explicitly.
onDelete:setNull and onDelete:restrict are the alternatives — see
Relations.
What we built
| Concept | Where |
|---|---|
| BelongsTo (convention) | Book.AuthorID, Review.BookID, Review.UserID |
| HasMany | Author.Books, Book.Reviews |
| ManyToMany | Book.Genres ↔ Genre.Books via BookGenre |
| Explicit relation with cascade | Book.AuthorID after the cascade edit |
| Soft delete | maniflex.WithDeletedAt on Review |
| Filtering through relations | ?filter=author.name:ilike:Ursula% |
Next
In Part 4 — Validation & Business Rules we tighten the rules: ISBNs follow a specific format, a user may not review the same book twice, and reviews on out-of-stock books are blocked.
4. Validation & Business Rules
The mfx: tag rules from Part 3 cover the common case — required fields,
numeric ranges, enums, uniqueness hints. Anything that goes beyond a single
field belongs in Validate-step middleware. This part adds three rules:
- ISBNs must be 13 digits with hyphens optional.
- A user may not review the same book twice.
- A user must have bought a book before reviewing it.
Field-format validation
validate.RegexField is enough for the ISBN check:
import "github.com/xaleel/maniflex/middleware/validate"
server.Pipeline.Validate.Register(
validate.RegexField("isbn", `^(?:97[89])?\d{10}$`),
maniflex.ForModel("Book"),
)
The middleware runs after the mfx: tag rules. A malformed ISBN aborts the
request with 422 VALIDATION_ERROR and the field in details.
We strip hyphens before validating so the client can send the human-readable form. A small Service-step middleware does the rewrite:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if raw, ok := ctx.Field("isbn"); ok {
if v, ok := raw.(string); ok {
ctx.SetField("isbn", strings.ReplaceAll(v, "-", ""))
}
}
return next()
}, maniflex.ForModel("Book"), maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate))
Order matters: Validate runs before Service in the pipeline. We’re rewriting
after validation has confirmed the cleaned-up format would pass. To
validate the cleaned value, swap the registration so the cleanup is on
Validate at maniflex.Before (the default).
“One review per book per user”
Two flavours of uniqueness:
- Schema uniqueness (
mfx:"unique") — adds aUNIQUEconstraint on a single column. Good for an email address. - Cross-column uniqueness — needs custom validation, because no single column is unique.
For reviews we need both book_id and user_id to be unique together. A
small middleware that consults the database:
server.Pipeline.Validate.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
rows, err := ctx.RawQuery(
`SELECT id FROM reviews
WHERE book_id = ? AND user_id = ? AND deleted_at IS NULL`,
bookID, ctx.Auth.UserID,
)
if err != nil {
return err
}
if len(rows) > 0 {
ctx.Abort(http.StatusConflict, "ALREADY_REVIEWED",
"you have already reviewed this book")
return nil
}
return next()
}, maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate))
Two things to notice:
- We use
ctx.RawQueryrather thanctx.GetModel(...).Listbecause we need a count, not the rows. Either works. - The
user_idwe check isctx.Auth.UserID, not the body’suser_id. In the next section we’ll force the body field to match.
“Body owner must match authenticated user”
Letting a client supply user_id is asking for impersonation. service.OwnerScope
from the catalogue forces the field on every create:
import "github.com/xaleel/maniflex/middleware/service"
server.Pipeline.Service.Register(
service.OwnerScope("user_id"),
maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate),
)
OwnerScope reads ctx.Auth.UserID and sets it on the body via ctx.SetField,
overwriting whatever the client sent. A client who omits the field gets it
filled in; a client who sets it to someone else’s ID has the value
overwritten silently.
This is a good place to apply Forbidden values
on role for User too — defence in depth against a privilege-escalation
payload:
server.Pipeline.Validate.Register(
validate.ForbiddenValues("role", "admin"),
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate),
)
A normal sign-up cannot self-promote to admin; an admin still can, because
they go through PATCH not POST, and the rule is scoped to create only.
Cross-field rules
The third rule — review only books you’ve bought — depends on a model
(Order) we haven’t built yet. We come back to it in Part 7 once orders
exist, using the same Validate.Register shape with a join query:
server.Pipeline.Validate.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
rows, _ := ctx.RawQuery(
`SELECT 1
FROM order_lines ol
JOIN orders o ON o.id = ol.order_id
WHERE o.customer_id = ?
AND ol.book_id = ?
AND o.status IN ('paid','shipped')`,
ctx.Auth.UserID, bookID,
)
if len(rows) == 0 {
ctx.Abort(http.StatusForbidden, "PURCHASE_REQUIRED",
"you may only review books you have bought")
return nil
}
return next()
}, maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate))
We leave this in the codebase as a stub for now and complete it in Part 7.
Where each rule lives
| Rule | Where | Why |
|---|---|---|
| ISBN format | Validate (catalogue) | format check on one field |
| One-review-per-book | Validate (custom) | queries another row |
user_id belongs to caller | Service (catalogue) | mutates body |
role cannot be admin on sign-up | Validate (catalogue) | rejects body value |
| Must have purchased | Validate (custom, deferred) | cross-model query |
The general rule: field-level rules go in Validate; rules that mutate the body go in Service; rules that need the row to exist go in After-DB.
Next
In Part 5 — File Uploads we add a cover image to each book, served from local storage during development and ready to swap for S3 in production.
5. File Uploads
A book needs a cover image. In this part we add a file field to the Book
model, configure local storage for development, and learn how the same code
handles a swap to S3 in production.
Adding the file field
Edit models/book.go:
type Book struct {
maniflex.BaseModel
Title string `json:"title" mfx:"required,filterable,sortable"`
ISBN string `json:"isbn" mfx:"required,filterable,unique"`
Price float64 `json:"price" mfx:"required,min:0,filterable,sortable"`
Stock int64 `json:"stock" mfx:"required,min:0,filterable"`
PublishedAt string `json:"published_at" mfx:"filterable,sortable"`
AuthorID string `json:"author_id" mfx:"required,filterable,relation:Author;onDelete:cascade"`
Author Author `json:"author,omitempty"`
Cover string `json:"cover" mfx:"file,max_size:2MB,accept:image/png|image/jpeg"`
Genres []Genre `json:"genres,omitempty" mfx:"through:BookGenre"`
Reviews []Review `json:"reviews,omitempty"`
}
The cover field is a string in Go and a string in the database, but the
mfx:"file" tag opts the model into multipart uploads. The column stores the
storage key — a path under whichever backend you have configured.
max_size and accept are enforced in the framework, before the upload
reaches storage. An oversize 5 MB JPEG is rejected with
413 FILE_TOO_LARGE, and a disallowed application/pdf with
415 FILE_TYPE_NOT_ALLOWED — neither is ever written.
Configuring storage
For development we use local disk. The maniflex/storage package ships a ready
implementation:
import "github.com/xaleel/maniflex/storage"
fs, err := storage.NewLocalStorage("./uploads")
if err != nil {
log.Fatal(err)
}
server := maniflex.New(maniflex.Config{
Port: 8080,
PathPrefix: "/api",
FileStorage: fs,
})
./uploads is created if it doesn’t exist. Every uploaded file lands under
uploads/<uuid>/<sanitised-filename> so collisions are impossible.
Uploading a cover
There are two ways to attach a cover, both supported out of the box.
1. Multipart upload alongside create
The client sends multipart/form-data with one part per field:
curl -X POST localhost:8080/api/books \
-H "Authorization: Bearer $TOKEN" \
-F 'title=The Dispossessed' \
-F 'isbn=9780061054884' \
-F 'price=12.99' \
-F 'stock=10' \
-F "author_id=$AUTH" \
-F 'cover=@./covers/dispossessed.jpg;type=image/jpeg'
The framework parses the multipart envelope, streams cover into the
storage backend, writes the resulting key into the column, and persists the
row. The response is the usual JSON envelope:
{
"data": {
"id": "...",
"title": "The Dispossessed",
"cover": "uploads/3f2b.../dispossessed.jpg",
...
}
}
2. Two-step upload + reference
For large files or out-of-band uploads, hit the standalone
/files endpoint first:
KEY=$(curl -s -X POST localhost:8080/files \
-H "Authorization: Bearer $TOKEN" \
-F 'file=@./covers/dispossessed.jpg;type=image/jpeg' \
| jq -r .data.key)
curl -X POST localhost:8080/api/books \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"title\":\"The Dispossessed\",\"isbn\":\"9780061054884\",\"price\":12.99,\"stock\":10,\"author_id\":\"$AUTH\",\"cover\":\"$KEY\"}"
The file field accepts a plain string in JSON — the storage key returned
by /files. The framework recognises that the value is already a key (not a
new upload) and stores it as-is.
Downloading a cover
Storage keys are served at /files/{key...}:
curl 'localhost:8080/files/uploads/3f2b.../dispossessed.jpg' --output cover.jpg
The handler sets Content-Type, Content-Disposition: inline, and
Content-Length from the metadata stored alongside the file.
For a permission layer in front of downloads — say, only registered users can fetch covers — add Auth middleware to the file route just as you would for any model route.
Automatic cleanup
The framework tracks the row that owns each key. A file is deleted from storage when:
- the owning row is hard-deleted, or
- the field is overwritten by a
PATCHthat supplies a new file or key.
Book does not embed WithDeletedAt, so a delete is a hard-delete and the
cover goes away too. If you want covers to outlive book deletions (for an
audit trail), tag the field with auto_delete:false:
Cover string `json:"cover" mfx:"file,max_size:2MB,accept:image/*,auto_delete:false"`
Swapping in S3
FileStorage is a four-method interface — Store, Retrieve, Delete,
Exists. A drop-in S3 implementation looks like:
type S3Storage struct{ client *s3.Client; bucket string }
func (s *S3Storage) Store(ctx context.Context, key string, r io.Reader, meta maniflex.FileMeta) error {
_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
Bucket: &s.bucket,
Key: &key,
Body: r,
ContentType: &meta.ContentType,
})
return err
}
// Retrieve, Delete, Exists similarly.
Swap storage.NewLocalStorage(...) for the new type in main.go and nothing
else changes. The same model code, the same endpoints, the same multipart
parser. The model never knows.
What we built
| Capability | How |
|---|---|
| File field on Book | mfx:"file,max_size:...,accept:..." |
| Local storage backend | storage.NewLocalStorage("./uploads") |
| Multipart upload | The framework auto-detects multipart/form-data on create/update |
| Pre-uploaded key reference | Plain string in the JSON body |
| Standalone upload | POST /files, returns a key |
| Backend-agnostic | maniflex.FileStorage interface — swap to S3 with no model change |
Next
In Part 6 — Filtering, Sorting & Pagination we build a catalogue browser: lookup books by title, sort by price or publication date, paginate the results, and combine includes with filters.
6. Filtering, Sorting & Pagination
The catalogue is in place. This part stitches together the query parameters
exposed by every list endpoint — filter, sort, include, page,
limit — to build a real browse experience.
Recap: opt-in fields
Every queryable field carries the relevant mfx: tag. Book’s fields are
already tagged from Part 3:
| Field | Tags |
|---|---|
title | filterable,sortable |
isbn | filterable,unique |
price | filterable,sortable |
stock | filterable |
published_at | filterable,sortable |
author_id | filterable |
Untagged fields are deliberately invisible to clients — a query string that
references them is rejected with 400 INVALID_QUERY.
Filter operators
All the operators on one model:
# Title contains "wind" (case-insensitive)
curl 'localhost:8080/api/books?filter=title:ilike:%25wind%25'
# Priced between $10 and $20
curl 'localhost:8080/api/books?filter=price:gte:10&filter=price:lte:20'
# Out of stock
curl 'localhost:8080/api/books?filter=stock:eq:0'
# In any of three genres
curl 'localhost:8080/api/books?filter=genres.label:in:Fantasy,Sci-Fi,Mystery'
# Published after a date, returned newest first
curl 'localhost:8080/api/books?filter=published_at:gte:2020-01-01&sort=published_at:desc'
Multiple filters compose with AND. The framework parses each filter once in
the Deserialize step into ctx.Query.Filters, then the DB step translates
the slice into a WHERE clause.
Relation filters
?filter=genres.label:in:... filters through the many-to-many junction.
Dot-notation works on any relation whose target field is filterable:
# Books by an author whose name contains "Le Guin"
curl 'localhost:8080/api/books?filter=author.name:ilike:%25Le+Guin%25&include=author'
# Reviews left on books with a specific ISBN
curl 'localhost:8080/api/reviews?filter=book.isbn:eq:9780061054884&include=book'
The include is independent of the filter — you can filter on a relation without returning it, and vice versa.
Sorting
?sort=field:direction for one column, repeat for tie-breakers:
# Cheapest first, oldest among ties
curl 'localhost:8080/api/books?sort=price:asc&sort=published_at:asc'
Only sortable fields work. BaseModel’s created_at and updated_at are
sortable by default.
Pagination
The defaults — page 1, 20 per page — work everywhere. Override per request:
curl 'localhost:8080/api/books?page=2&limit=50'
limit is clamped at 200; oversize requests are silently reduced. List
responses carry pagination metadata in meta:
{
"data": [ ... ],
"meta": { "total": 137, "page": 2, "limit": 50, "pages": 3 }
}
For models where the rows are expensive to render — full audit logs,
analytics tables — register db.Paginate from the catalogue to lower the
ceiling per model:
import "github.com/xaleel/maniflex/middleware/db"
server.Pipeline.DB.Register(db.Paginate(50), maniflex.ForModel("AuditLog"))
Includes
?include=relation1,relation2 populates nested objects in the response.
Includes are separate queries — they do not multiply rows or affect
pagination of the primary list:
curl 'localhost:8080/api/books/<id>?include=author,genres,reviews'
The relation keys come from the model declarations — see
Relations for how they are derived. For a BelongsTo the
result is a single nested object; for HasMany and ManyToMany, an array.
Combining everything
A realistic “browse” call:
curl 'localhost:8080/api/books?filter=genres.label:eq:Science+Fiction
&filter=stock:gt:0
&filter=price:lte:20
&sort=published_at:desc
&include=author,genres
&page=1
&limit=12'
The framework executes this as:
- Parse query →
ctx.Query.Filters,Sorts,Includes,Page,Limit. - Run the main
SELECTwith the WHERE + ORDER BY + LIMIT/OFFSET. - Issue follow-up queries for each include, batched by foreign key.
- Compose the JSON envelope.
Hardcoding tenant scope
In some applications a request from one customer should never see another
customer’s rows. We don’t have multi-tenancy in the bookstore, but the
mechanism is worth knowing. db.Tenancy enforces row-level scoping
unconditionally:
server.Pipeline.DB.Register(
db.Tenancy("organization_id", func(ctx *maniflex.ServerContext) string {
return ctx.Auth.TenantID
}),
)
Once registered, every list, read, update, and delete is silently filtered to
organization_id = ctx.Auth.TenantID. The client cannot override or escape
it.
Custom filters in middleware
ctx.Query.Filters is a slice — middleware can append to it before the DB
step runs. We’ll use this in Part 7 when we want logged-in customers to see
only their own orders:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
ctx.Query.Filters = append(ctx.Query.Filters, &maniflex.FilterExpr{
Field: "customer_id",
Operator: maniflex.OpEq,
Value: ctx.Auth.UserID,
})
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpList))
The pattern is the same as the bigger Tenancy middleware — write a filter,
let the DB step honour it.
Next
In Part 7 — Custom Endpoints & Actions we add order placement: a transactional action that locks stock, creates the order and its lines, and writes an outbox row that Part 8’s background worker will consume.
7. Custom Endpoints & Actions
Customers need to place orders. A simple POST /api/orders would just
insert one row, but real order placement also has to lock stock, create
line items, and queue a downstream notification — all atomically. This is
the textbook use case for a custom action.
The models
Two new entities. Both opt into soft-delete so an audit trail survives.
// models/order.go
type Order struct {
maniflex.BaseModel
maniflex.WithDeletedAt
CustomerID string `json:"customer_id" mfx:"required,filterable,immutable"`
Total float64 `json:"total" mfx:"required,min:0,filterable,sortable"`
Status string `json:"status" mfx:"required,enum:pending|paid|shipped|cancelled,default:pending,filterable,sortable"`
Lines []OrderLine `json:"lines,omitempty"`
}
// models/order_line.go
type OrderLine struct {
maniflex.BaseModel
OrderID string `json:"order_id" mfx:"required,filterable,immutable"`
BookID string `json:"book_id" mfx:"required,filterable,immutable"`
Quantity int64 `json:"quantity" mfx:"required,min:1"`
UnitPrice float64 `json:"unit_price" mfx:"required,min:0"`
}
// models/outbox.go — Part 8 will consume rows from here.
type OutboxEvent struct {
maniflex.BaseModel
Kind string `json:"kind" mfx:"required,filterable"`
Payload map[string]any `json:"payload" mfx:"required"`
Status string `json:"status" mfx:"required,enum:pending|done|failed,default:pending,filterable"`
ErrorMsg string `json:"error_msg"`
}
Register them. We also tenancy-scope reads of Order to the calling
customer so one user cannot list another’s orders:
server.MustRegister(models.Order{}, models.OrderLine{}, models.OutboxEvent{})
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
ctx.Query.Filters = append(ctx.Query.Filters, &maniflex.FilterExpr{
Field: "customer_id", Operator: maniflex.OpEq, Value: ctx.Auth.UserID,
})
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpList))
Why an action
The standard POST /api/orders would insert one Order row and stop. We
need:
- Lock the books so concurrent buyers don’t oversell stock.
- Decrement stock on every line.
- Create the order.
- Create one
OrderLineper book. - Append an outbox row describing the order, in the same transaction.
A single transaction must cover all five. The Service step on POST /orders
sees only the order body — the lines come from the client. We could write
five middleware functions, but a custom action keeps
the transaction obvious and the trimmed pipeline lighter:
Auth → action handler → Response
Deserialize, Validate, Service, and DB are skipped. Our handler does
its own parsing and database work.
The handler
actions/orders.go:
package actions
func PlaceOrder(ctx *maniflex.ServerContext) error {
var req struct {
Lines []struct {
BookID string `json:"book_id"`
Quantity int64 `json:"quantity"`
} `json:"lines"`
}
if err := ctx.BindJSON(&req); err != nil {
return nil
}
if len(req.Lines) == 0 {
ctx.Abort(http.StatusBadRequest, "EMPTY_ORDER", "an order must contain at least one line")
return nil
}
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
ctx.Tx = tx
// 1+2: lock each book row and decrement stock.
type planned struct {
bookID string
quantity int64
unitPrice float64
}
var plan []planned
var total float64
for _, l := range req.Lines {
book, err := ctx.LockForUpdate("Book", l.BookID)
if err != nil {
ctx.Abort(http.StatusNotFound, "BOOK_NOT_FOUND",
fmt.Sprintf("book %s does not exist", l.BookID))
return nil
}
stock := book["stock"].(int64)
if stock < l.Quantity {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK",
fmt.Sprintf("book %s has %d in stock", l.BookID, stock))
return nil
}
if _, err := ctx.GetModel("Book").Update(l.BookID, map[string]any{
"stock": stock - l.Quantity,
}); err != nil {
return err
}
price := book["price"].(float64)
total += price * float64(l.Quantity)
plan = append(plan, planned{l.BookID, l.Quantity, price})
}
// 3: the Order row.
order, err := ctx.GetModel("Order").Create(map[string]any{
"customer_id": ctx.Auth.UserID,
"total": total,
"status": "pending",
})
if err != nil {
return err
}
// 4: one OrderLine per book.
for _, p := range plan {
if _, err := ctx.GetModel("OrderLine").Create(map[string]any{
"order_id": order["id"],
"book_id": p.bookID,
"quantity": p.quantity,
"unit_price": p.unitPrice,
}); err != nil {
return err
}
}
// 5: outbox row — picked up by the worker in Part 8.
if _, err := ctx.GetModel("OutboxEvent").Create(map[string]any{
"kind": "order-placed",
"payload": map[string]any{
"order_id": order["id"],
"customer_id": ctx.Auth.UserID,
"total": total,
},
"status": "pending",
}); err != nil {
return err
}
if err := tx.Commit(); err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusCreated,
Data: order,
}
return nil
}
Three things worth pointing out:
ctx.LockForUpdateacquires a row-level write lock that lasts until the transaction ends. A concurrent buyer hitting the same book waits at that line until we commit or roll back.- All five inserts share
ctx.Tx.ctx.GetModel(...).Createroutes through the transaction automatically — there is no separate “transactional client” to thread. defer tx.Rollback()is safe after a successfulCommit— rollback becomes a no-op once the transaction has been finalised.
Registering the action
In main.go:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: actions.PlaceOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth("dev-secret"), // identity → ctx.Auth
},
})
auth.JWTAuth is the same middleware registered globally on Auth in Part 2,
but action middleware runs only for the action itself — handy when the action
needs different auth from the generated routes.
Trying it
curl -X POST localhost:8080/api/orders/place \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d "{\"lines\":[{\"book_id\":\"$BOOK\",\"quantity\":2}]}"
Response:
{
"data": {
"id": "abc123…",
"customer_id": "user-alice",
"total": 25.98,
"status": "pending",
...
}
}
Re-run it until the book runs out, and the next request gets a clean
409 OUT_OF_STOCK instead of a partial write.
Finishing the “must have bought” review check
Part 4 left a stub: only customers who have bought a book may review it. The
join query needs order_lines and orders — which we now have:
server.Pipeline.Validate.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
rows, _ := ctx.RawQuery(
`SELECT 1
FROM order_lines ol
JOIN orders o ON o.id = ol.order_id
WHERE o.customer_id = ?
AND ol.book_id = ?
AND o.status IN ('paid','shipped')`,
ctx.Auth.UserID, bookID,
)
if len(rows) == 0 {
ctx.Abort(http.StatusForbidden, "PURCHASE_REQUIRED",
"you may only review books you have bought")
return nil
}
return next()
}, maniflex.ForModel("Review"), maniflex.ForOperation(maniflex.OpCreate))
Next
In Part 8 — Events & Background Jobs we build the background worker that consumes outbox rows and emails order receipts.
8. Events & Background Jobs
Part 7 added a custom action for placing orders. This part defers the post-order work — sending a receipt email — to a background job so that a slow or failing mailer never blocks the purchase response or rolls back the order transaction.
Why a background worker
Running side-effects inside the request transaction creates two problems:
- If the email fails, the whole order rolls back — a transient mail outage breaks the purchase flow entirely.
- If the order rolls back after a successful email, the email can’t be unsent.
Decoupling fixes both: the transaction writes a small job description; a worker outside the transaction carries it out. If the worker crashes mid-task the job is retried automatically.
Wiring up the job queue
The jobs/ package family provides a durable queue with retries and REST-based
status polling. For the bookstore we use jobs/sql, which enqueues inside the
same database transaction as the business write.
Install the queue alongside the server setup in main.go:
import (
"github.com/xaleel/maniflex"
jobsmaniflex "github.com/xaleel/maniflex/jobs/maniflex"
"github.com/xaleel/maniflex/jobs"
jobssql "github.com/xaleel/maniflex/jobs/sql"
)
server := maniflex.New(maniflex.Config{ /* ... */ })
server.MustRegister(Order{}, User{} /* ... */)
db, _ := sqlite.Open("./app.db", server.Registry())
server.SetDB(db)
// jobs/sql takes a database/sql handle, not the maniflex adapter; point it at the
// same database file so jobs live alongside your data. (The "sqlite" driver is
// registered by importing the db/sqlite package above.)
jobsDB, _ := sql.Open("sqlite", "./app.db")
queue := jobssql.New(jobsDB)
jobssql.Migrate(ctx, jobsDB, "sqlite") // "postgres" on PG
// Mount registers the StatusModel and returns a wrapped queue + sink.
// After this, GET /api/job_statuses/:id is available automatically.
sink, queue, err := jobsmaniflex.Mount(server, queue)
if err != nil { log.Fatal(err) }
// Wire up the worker.
w, _ := jobs.NewWorker(jobs.WorkerConfig{
Source: queue.(jobs.Source),
Status: sink,
Handlers: map[string]jobs.Handler{
"send_receipt": sendReceiptHandler(mailer),
},
})
go w.Run(ctx)
log.Fatal(server.Start())
Enqueueing from the order action
Modify the order placement action from Part 7 to enqueue a job instead of calling the mailer directly:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders",
Handler: func(ctx *maniflex.ServerContext) error {
// ... validate, insert order, etc. ...
jobID, err := queue.Enqueue(ctx.Ctx, jobs.Job{
Type: "send_receipt",
ActorID: ctx.Auth.UserID,
Payload: mustJSON(map[string]any{"order_id": orderID}),
})
if err != nil {
return err
}
ctx.Response = &maniflex.APIResponse{
StatusCode: http.StatusAccepted,
Data: map[string]any{
"order_id": orderID,
"job_id": jobID, // clients can poll /api/job_statuses/:job_id
},
}
return nil
},
})
The wrapped queue creates an enqueued status row before returning, so the
client can poll immediately — no race between enqueue and the first GET.
The handler
func sendReceiptHandler(mailer Mailer) jobs.Handler {
return func(ctx context.Context, j jobs.Job) (jobs.Result, error) {
var p struct {
OrderID string `json:"order_id"`
}
if err := json.Unmarshal(j.Payload, &p); err != nil {
return jobs.Result{}, err
}
return jobs.Result{}, mailer.SendReceipt(ctx, p.OrderID)
}
}
Handlers return (jobs.Result, error). On error the worker retries with
exponential backoff (default up to 3 attempts). After all retries the job is
marked dead and the status row records the final error message.
Polling for completion
The client receives job_id in the response and polls until done:
POST /api/orders
← 202 {"data": {"order_id": "xyz", "job_id": "01JABC..."}}
GET /api/job_statuses/01JABC...
← 200 {"data": {"status": "enqueued", ...}}
GET /api/job_statuses/01JABC... (retry after a tick)
← 200 {"data": {"status": "succeeded", "completed_at": "2025-01-15T09:01:02Z"}}
No extra endpoint or custom table — the StatusModel is wired up automatically
by Mount.
Emitting events from the pipeline
For lighter-weight fan-out — “notify other services every time an Order is
created” — the events.Emit middleware is a simpler fit than the job queue:
import (
"github.com/xaleel/maniflex/events"
"github.com/xaleel/maniflex/events/redis"
)
bus := redis.New(redisClient, "myapp") // prefix namespaces the Redis stream keys
server.Pipeline.DB.Register(
events.Emit(bus),
maniflex.ForModel("Order"),
maniflex.AtPosition(maniflex.After),
)
Emit publishes order.created (and order.updated, order.deleted) to the
bus on the DB-After step — only when the write succeeded. Subscribers in the
same or other processes consume events independently. For WebSocket fan-out to
connected clients, wire a realtime.Hub to the bus (see
Realtime / WebSockets).
Webhooks
events.Webhook delivers events to external URLs with an HMAC signature —
useful for one-off partner integrations. Unlike events.Emit, it is an event-bus
subscriber, not pipeline middleware, so wire it with bus.Subscribe:
bus.Subscribe(ctx, events.Subscription{
Patterns: []string{"order.*"},
Handler: events.Webhook(events.WebhookConfig{
URL: "https://partner.example.com/orders",
Secret: os.Getenv("WEBHOOK_SECRET"),
}),
})
What we built
| Capability | How |
|---|---|
| Decoupled post-order email | jobs.Queue — enqueue in action, process in worker |
| Status polling | jobs/maniflex.Mount → GET /api/job_statuses/:id |
| Automatic retries | jobs.Job.MaxRetry + exponential backoff |
| Transactional enqueue | jobs/sql inserts the job row in the same DB transaction |
| Domain event fan-out | events.Emit on DB-After → event bus subscribers |
| External webhook delivery | events.Webhook as a bus subscriber |
Next
In Part 9 — Testing the API we test the whole app end to end, including the job worker and the polling flow.
9. Testing the API
A useful test suite for a maniflex app exercises the HTTP layer, not just the
database. The framework is built on net/http, so the standard
httptest.Server is enough — start an in-memory SQLite, register everything,
hit the routes, assert on responses.
A test harness
tests/setup.go:
package tests
import (
"context"
"log"
"net/http/httptest"
"github.com/xaleel/maniflex"
"github.com/xaleel/maniflex/db/sqlite"
"bookstore/middleware"
"bookstore/models"
)
// newTestServer returns a running httptest.Server backed by an in-memory
// SQLite. Each test gets a fresh database.
func newTestServer(t *testing.T) (*httptest.Server, *maniflex.Server) {
t.Helper()
server := maniflex.New(maniflex.Config{
Port: 0,
PathPrefix: "/api",
})
server.MustRegister(
models.User{}, models.Author{}, models.Genre{},
models.Book{}, models.BookGenre{}, models.Review{},
models.Order{}, models.OrderLine{}, models.OutboxEvent{},
)
db, err := sqlite.Open(":memory:", server.Registry())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { db.Close() })
server.SetDB(db)
// Handler() does NOT migrate — only Start() does, and tests mount Handler()
// directly. MigrateOnly runs the migration and honours AutoMigrate.
if err := server.MigrateOnly(context.Background()); err != nil {
t.Fatal(err)
}
middleware.Register(server)
ts := httptest.NewServer(server.Handler())
t.Cleanup(ts.Close)
return ts, server
}
Three notes:
:memory:opens an in-memory SQLite database. There is no file to clean up; closing the connection discards it.server.MigrateOnly(...)creates the tables.server.Handler()builds the router but does not migrate — onlyStart()does that. When you mountHandler()yourself (tests, embedding), migrate explicitly first or every request fails against missing tables.server.Handler()returns the chi router —httptest.NewServerwraps it and serves requests in-process.
A small JSON helper keeps tests readable:
func do(t *testing.T, ts *httptest.Server, method, path, token string, body any) (int, map[string]any) {
t.Helper()
var rdr io.Reader
if body != nil {
b, _ := json.Marshal(body)
rdr = bytes.NewReader(b)
}
req, _ := http.NewRequest(method, ts.URL+path, rdr)
if token != "" {
req.Header.Set("Authorization", "Bearer "+token)
}
req.Header.Set("Content-Type", "application/json")
resp, err := ts.Client().Do(req)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
var out map[string]any
json.NewDecoder(resp.Body).Decode(&out)
return resp.StatusCode, out
}
Happy-path sign-up + login
func TestSignupAndLogin(t *testing.T) {
ts, _ := newTestServer(t)
code, body := do(t, ts, "POST", "/api/users", "", map[string]any{
"email": "[email protected]",
"password": "hunter22!",
"name": "Alice",
})
if code != 201 {
t.Fatalf("signup: %d %v", code, body)
}
code, body = do(t, ts, "POST", "/api/auth/login", "", map[string]any{
"email": "[email protected]",
"password": "hunter22!",
})
if code != 200 || body["data"].(map[string]any)["token"] == "" {
t.Fatalf("login: %d %v", code, body)
}
}
A new in-memory database for each t.Run keeps the tests isolated; nothing
to truncate, nothing to seed beyond the test’s own writes.
Validation failures
The exact response shape matters because clients depend on it. Pin it down:
func TestInvalidEmail(t *testing.T) {
ts, _ := newTestServer(t)
code, body := do(t, ts, "POST", "/api/users", "", map[string]any{
"password": "hunter22!",
"name": "Alice",
// email missing
})
if code != 422 {
t.Fatalf("got %d, want 422", code)
}
if body["error"].(map[string]any)["code"] != "VALIDATION_ERROR" {
t.Fatalf("code = %v", body["error"])
}
}
Stock contention
The order-placement transaction is the most interesting code path. The test starts two goroutines that race for the last unit:
func TestStockContention(t *testing.T) {
ts, _ := newTestServer(t)
tok, bookID := seedOneBookOneCustomer(t, ts, 1) // stock = 1
var wg sync.WaitGroup
results := make([]int, 2)
for i := range results {
wg.Add(1)
go func(i int) {
defer wg.Done()
results[i], _ = do(t, ts, "POST", "/api/orders/place", tok, map[string]any{
"lines": []map[string]any{{"book_id": bookID, "quantity": 1}},
})
}(i)
}
wg.Wait()
// Exactly one 201 Created, exactly one 409 Conflict.
sort.Ints(results)
if results[0] != 201 || results[1] != 409 {
t.Fatalf("expected one 201 and one 409, got %v", results)
}
}
LockForUpdate ensures only one of the two transactions wins; the other
sees the decremented stock and aborts with OUT_OF_STOCK.
Worker tests
The background worker is plain Go. Inject a stub mailer and assert on it:
func TestOutboxWorker(t *testing.T) {
ts, server := newTestServer(t)
tok, bookID := seedOneBookOneCustomer(t, ts, 5)
// Place an order — should write an outbox row.
do(t, ts, "POST", "/api/orders/place", tok, map[string]any{
"lines": []map[string]any{{"book_id": bookID, "quantity": 1}},
})
stub := &captureMailer{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go jobs.RunOutboxOnce(ctx, server, stub) // single iteration
// Give the worker a moment.
if !waitFor(time.Second, func() bool { return len(stub.Sent) == 1 }) {
t.Fatalf("expected 1 email, got %d", len(stub.Sent))
}
if stub.Sent[0].Subject != "Thank you for your order" {
t.Fatalf("wrong subject: %q", stub.Sent[0].Subject)
}
}
Run worker logic in a one-shot variant (RunOutboxOnce) for testability —
or accept a context.Context and cancel it after the assertions pass. Both
patterns avoid time.Sleep in tests.
The framework’s own test suite
The framework ships an end-to-end suite under tests/e2e/. It is the
canonical reference for what a thorough maniflex test looks like — every step
of the pipeline, every adapter, every middleware option. Run it with:
go test ./tests/e2e/...
…and look at the test files for patterns you can lift into your own suite.
Coverage strategy
For a typical bookstore-shaped app, a useful test split:
- Per model: happy-path create + read + update + delete.
- Per
mfx:tag rule: at least one negative test (required,enum,min/max,unique). - Per custom middleware: at least one happy-path and one rejection.
- Per action: happy path, a representative failure, and a contention test.
- The outbox worker: receives an event, processes it, marks it done.
That covers the surface area without exploding into combinatorial tests of every filter operator and every relation include — those are exercised by the framework’s own e2e suite, which you depend on transitively.
Next
In Part 10 — Deploying to Production we swap SQLite for PostgreSQL, drive configuration from environment variables, enable the health probe, and produce a single binary suitable for a container image.
10. Deploying to Production
The bookstore runs cleanly on SQLite + go run . for development. Production
needs three additional things: a real database, configuration from the
environment, and a sensible operational contract — health probes,
structured logs, graceful shutdown. Code changes are minimal; the framework
was already built for this.
Swapping SQLite for PostgreSQL
Add the satellite:
go get github.com/xaleel/maniflex/db/postgres
Change one import in main.go:
- import "github.com/xaleel/maniflex/db/sqlite"
+ import "github.com/xaleel/maniflex/db/postgres"
…and the adapter open call:
// Open(writeDSN, readDSN, registry) is positional. For pool/session tuning use
// OpenWithConfig(writeDSN, readDSN, registry, writePool, readPool, session).
db, err := postgres.OpenWithConfig(
os.Getenv("DB_WRITE_URL"),
os.Getenv("DB_READ_URL"), // optional; "" routes reads to the primary
server.Registry(),
postgres.PoolConfig{MaxOpenConns: 25, MaxIdleConns: 5, ConnMaxLifetime: 30 * time.Minute}, // write pool
postgres.PoolConfig{MaxOpenConns: 25, MaxIdleConns: 5, ConnMaxLifetime: 30 * time.Minute}, // read pool
postgres.SessionConfig{ApplicationName: "bookstore"},
)
Models, middleware, actions, and tests all carry over unchanged. The shared
db/sqlcore adapter means SQL emitted by AutoMigrate is portable between
the two backends. See PostgreSQL in Production for
pool tuning, read replicas, and migration choices.
Configuration from environment
A single Config populated from os.Getenv:
// config.go
func loadConfig() maniflex.Config {
return maniflex.Config{
Port: envInt("PORT", 8080),
PathPrefix: envStr("PATH_PREFIX", "/api"),
ServiceName: envStr("SERVICE_NAME", "bookstore"),
DisableAutoMigrate: envStr("AUTO_MIGRATE", "true") != "true",
QueryTimeout: envDuration("QUERY_TIMEOUT", 30*time.Second),
ShutdownTimeout: envDuration("SHUTDOWN_TIMEOUT", 30*time.Second),
HealthCheckDB: true,
HealthTimeout: 3 * time.Second,
Logger: slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
})),
}
}
maniflex also ships a helper, maniflex.ConfigFromEnv(prefix), that reads a
conventional set of environment variables (PORT, DB_WRITE_URL,
QUERY_TIMEOUT_MS, … — see Configuration).
It returns an error if one of them is set to something it cannot read, so a
mistyped PORT stops the deploy instead of booting on 8080. Pick whichever style
fits your team — both produce the same maniflex.Config.
Production-safe migrations
Migration runs by default (convenient in development). In production, the prevailing pattern is:
- Disable
AutoMigrateon every instance. - Run schema changes through a dedicated migration tool
(
golang-migrate, Atlas, sqlc-migrate) executed as a separate one-shot step in your deploy pipeline. - Roll out the new application code afterwards.
AUTO_MIGRATE=false
The framework’s auto-migrator never drops columns, but it isn’t aware of your release strategy — splitting “deploy” and “migrate” into two steps lets you stage them deliberately.
Health probes
HealthCheckDB: true enables a real probe — GET /health calls
db.Ping() with a HealthTimeout budget. Kubernetes:
livenessProbe:
httpGet:
path: /health
port: 8080
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /health
port: 8080
periodSeconds: 5
timeoutSeconds: 5
Set HealthTimeout (default 3s) shorter than the probe’s timeoutSeconds
so the handler can return a clean 503 before the probe gives up.
terminationGracePeriodSeconds on the pod should be longer than
Config.ShutdownTimeout, otherwise Kubernetes will SIGKILL the process
before in-flight requests have finished. With the defaults (30s shutdown),
60s grace is a comfortable buffer.
Logging and tracing
The JSON handler above turns every line into a structured record that an
aggregator can index. ctx.Logger() automatically adds request_id and
trace_id per request, so a single trace can be reconstructed end-to-end.
Set Config.ServiceName so every log line and every audit record carries
the service identifier — invaluable when a single aggregator collects logs
from several services.
For a debugging spike, enable pipeline tracing:
TRACE=1
if envStr("TRACE", "") != "" {
cfg.Trace = maniflex.PipelineTrace{Enabled: true, Skips: true}
}
Steps, Timings, and Aborts produce DEBUG-level records that show
every middleware enter/exit and the file:line of every Abort call.
Disable in normal operation — they are high-volume.
The Dockerfile
A typical Dockerfile for the binary:
FROM golang:1.25-alpine AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -o /out/bookstore ./
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/bookstore /bookstore
COPY static /static
EXPOSE 8080
USER 65532:65532
ENTRYPOINT ["/bookstore"]
CGO_ENABLED=0 works because maniflex/db/postgres uses lib/pq (pure Go) and
nothing else in the framework requires a C toolchain.
static/ is copied so the Scalar OpenAPI viewer is reachable at
/static/openapi.html — provided the app opts into static serving with
maniflex.Config{StaticDir: "static"} (serving is opt-in; an unset StaticDir
publishes nothing).
Production checklist
| Setting | |
|---|---|
| Database | maniflex/db/postgres with WriteURL and optional ReadURL |
| Migrations | DisableAutoMigrate: true + external migration tool |
| Logger | JSON handler |
Config.ServiceName | the service name |
Config.QueryTimeout | bounded (e.g. 30s) |
Config.ShutdownTimeout | matches the slowest legitimate request |
Config.HealthCheckDB | true |
K8s terminationGracePeriodSeconds | larger than ShutdownTimeout |
| TLS | terminated at the load balancer |
| Auth | auth.JWTAuth with an asymmetric key from your IdP |
| Rate limits | db.RateLimit on password-reset / sign-up / login |
| Audit log | db.AuditLog on OpCreate / OpUpdate / OpDelete |
| File storage | swap LocalStorage for S3 / R2 / GCS |
| Outbox worker | run alongside the API, or as a separate deployment |
Where to go from here
The tutorial finishes here. From this point, the reference pages cover everything in more depth, and the code base is small enough to grow in any direction:
- More middleware from the Middleware Catalogue.
- Customisation via Writing Middleware.
- Advanced workflows in Custom Endpoints, Raw Queries & Query Models, and Batch Operations & Sagas.
- Hardening with Auth & Security Hardening.
The shape of main.go has not changed in ten parts. Add models, add
middleware, add actions — the wiring is the same.
Startup Validation & Strict Mode
maniflex validates your configuration once, at startup, and reports everything
wrong with it in a single message. Most problems are fatal on their own;
Config.Strict adds the handful that are merely suspect.
The principle: a directive that parses but cannot be honoured is an error, not a silent no-op. A misspelt tag that is quietly ignored leaves the protection you wrote absent, and nothing at runtime says so.
Always fatal
These fail whether or not strict mode is on. Each one used to warn and then discard what you asked for.
| Problem | Why it cannot be a warning |
|---|---|
A ModelConfig at argument position 0 | It has no model to attach to, so the whole config was dropped — a ModelConfig{Headless: true} silently did not apply and the model mounted routes you thought were suppressed. |
Two ModelConfigs in a row | The second has no fresh model to bind to; same silent drop. |
An invalid mfx:"scheduled" tag | The field was dropped from the sweep, so the scheduled transition you configured simply never ran. |
mfx:"relation" on a field not ending in ID | The target model is derived by stripping that suffix. Without one it is inferred from the whole field name — almost never a real model. Write mfx:"relation:Target". |
| A middleware registered on a step its operations never reach | It is frozen into the chain and never runs. When it is an authorisation check, that is a silent hole. |
A RequiresField declaration no model can satisfy | The gate watches for a field name nothing has — see below. |
The first three are registration errors, returned from Register (so
MustRegister panics). The last two need the complete registry — a relation’s
target may be registered after the model pointing at it — so they are raised
when the router is built.
Declaring the fields a middleware gates
A middleware that gates a field by name has a nasty failure mode: misspell the name and the gate watches for a body key nothing sends, while the real field keeps its name and nothing gates it. It fails open, silently.
Nothing at runtime can detect this. From inside a request, “this model has no such field” looks identical to a gate deliberately registered across models where only some carry it. Only the registration knows which it is — so that is where you say so:
server.Pipeline.Validate.Register(
validate.RestrictField("document_quota_bytes", isSuperuser),
maniflex.ForModel("User"),
maniflex.RequiresField("document_quota_bytes"), // ← checked at startup
)
Writing the name twice is not the tautology it looks like: the declared name is checked against the model, not against the middleware’s argument, so a misspelling is caught either way.
The ForModel scope makes the check exact:
- With
ForModel— every named model must have every declared field. The gate was aimed at those models specifically, so a model that lacks the field is a mistake. - Without it — at least one registered model must have the field. A gate no model can trigger cannot be doing anything.
Use it for any middleware that gates or reads a field by name, not just
validate.RestrictField — response.RedactField and hand-written gates have
the same failure mode.
It is opt-in: a registration that declares nothing is not checked, so existing
code is unaffected. validate.RestrictField keeps its first-request warning as
a fallback for undeclared registrations, but that warning cannot fire for an
endpoint nobody exercises, and cannot tell a typo from a deliberate broad
registration. Declare the field.
Gated by Config.Strict
maniflex.New(maniflex.Config{Strict: true})
These stay warnings by default because each has a legitimate reading:
| Problem | The legitimate reading |
|---|---|
mfx:"relation" whose target model is not registered | The field may be a plain foreign id that wants no relation tag. The FK column works either way. |
The standalone /files endpoints mounted with no auth middleware | A deliberately public upload endpoint is conceivable, if rarely wise. |
Config.StaticDir names a directory that does not exist | Static serving degrades to 404s. Failing the boot would let a missing frontend asset bundle take down a working API. |
Turn it on in CI and staging, where a boot failure costs a re-run rather than an outage. Leave it off in production if you would rather serve a degraded API than none.
One report, not one restart at a time
Every problem found is listed together:
maniflex: 3 startup problems:
1. [relation] Invoice.Owner is tagged mfx:"relation" but its name does not
end in "ID", so the target model was inferred from the whole field name
as "Owner" — write mfx:"relation:Target" to name the target explicitly
2. [middleware] middleware "audit" is registered on the db step for
operation(s) [search], all of which skip that step — it will never run;
register it on a step those operations run, or widen ForOperation
3. [static] Config.StaticDir "./pubic" does not exist, so nothing would be
served under /static (Config.Strict)
Issues that are only fatal because strict mode is on are marked (Config.Strict),
so you never hunt for a bug in configuration that is legal by default.
When validation runs
Startup validation happens before migrations and before services start:
Start()
├── validate + build router ← fails here
├── migrate
├── start services
└── open listener
A configuration error therefore costs nothing to find — it cannot leave a half-migrated schema behind.
Start and StartWithContext return the error. Handler() panics,
because it has no error return; if you mount maniflex inside another router,
that is the form you will see.
Migration failures
Startup validation checks configuration. AutoMigrate separately refuses to
proceed when it cannot build a unique index — the model declares a
constraint and the database would not be enforcing it, so every write that
should have been refused would be accepted silently. This is not gated by
Config.Strict: a constraint that does not exist is wrong in every environment.
The usual cause is data that already violates it, and the error says so, naming the table, the index and the columns. De-duplicate and start again.
A failed plain index stays a warning: it costs a table scan, not a guarantee.
What stays a warning
Some warnings describe a legitimate operational state and are not promoted even under strict mode:
- A column in the database that is not on the model. Normal during a staged field removal; failing here would break rolling deploys.
- A column whose type differs from the model’s. Promoting this would brick startup against an already-drifted production database.
- Two scheduled fields targeting the same column where the later omits
from=. A real hazard, but a legal design.
Glossary
Every framework term used in these docs, in one place. Links point to the page where each term is defined in depth.
A
Action. A custom endpoint registered with server.Action(...). Runs a
trimmed pipeline (Auth → action middleware → handler → Response); the
Deserialize, Validate, Service, and DB steps are skipped. See
Custom Endpoints.
Adapter. An implementation of the maniflex.DBAdapter interface. Two ship:
db/sqlite and db/postgres. Custom backends implement the same
interface. See Database Backends.
After (position). A middleware position that places the function after
the step’s default handler. The middleware sees the result of the default
(ctx.DBResult, ctx.Response). See Writing Middleware.
AuditRecord. The structured record produced by db.AuditLog — fields
include Timestamp, Model, Operation, ResourceID, Actor,
TenantID, RequestID, TraceID, ServiceName, and optionally a
per-field Changes diff. See Audit Logging.
AuthInfo. The struct populated by Auth middleware and stored on
ctx.Auth. Carries UserID, Roles, Claims, TenantID,
IdentityType, Scopes, SessionID, AuthMethod. See
ServerContext.
AutoMigrate. The startup phase that creates and alters tables to match
registered models. Runs by default; disable with Config.DisableAutoMigrate. See
Database Backends.
B
BaseModel. The struct every registered model must embed. Contributes
id (UUID, framework-assigned), created_at, and updated_at. See
Models & BaseModel.
Batch. Multiple inserts or updates inside a single request, usually through a custom action that opens one transaction. See Batch Operations & Sagas.
Before (position). The default middleware position; the function runs before the step’s default handler. See Writing Middleware.
BelongsTo. A relation where this model carries the foreign key. Declared
either by convention (UserID field → User) or explicitly with
mfx:"relation:Name" plus a companion field of the target type. See
Relations.
C
Cache (CacheStore). The generic key/value cache interface used by
several middlewares (idempotency, rate limit, response cache). maniflex
ships MemoryCache; satellite modules provide Redis-backed
implementations. See cache.go.
Catalogue. The set of ready-made middleware shipping under
maniflex/middleware/. See Middleware Catalogue.
ServerContext. The single per-request struct threaded through every
pipeline step. Fields include Request, Writer, Ctx, Model,
Operation, ResourceID, RawBody, ParsedBody, Query, DBResult,
Response, Auth, Tx. See ServerContext.
Companion field. On an explicit BelongsTo relation, the struct field
of the target type that accompanies the FK column. Required by
mfx:"relation:Name"; the field is named Name and typed as the target
model. See Relations.
Config (maniflex.Config). The single struct passed to maniflex.New. Every
field has a sensible default. See Configuration.
D
DBAdapter. The interface implemented by every database backend.
Methods: FindByID, FindMany, Create, Update, Delete,
FindByIDForUpdate, AutoMigrate, BeginTx, Raw, Close. Ping is
not on DBAdapter — it is the separate, optional Pinger interface.
See Database Backends.
Diff (versioning). The per-field {old, new} map written into the
diff column of a history row. Hidden, writeonly, and encrypted fields
are excluded. See Versioning & History.
E
Embed. A BaseModel, WithDeletedAt, or WithIsDeleted value
included in a model struct as an anonymous field. Embeds contribute
columns and turn on framework behaviour. See Models & BaseModel
and Soft Delete.
Envelope (storage). The binary blob produced by KeyProvider.Encrypt,
stored in the database as enc:<base64(envelope)>. The envelope embeds
its keyID so Decrypt can route to the right key. See
Encryption at Rest.
Envelope (response). The JSON shape {"data": ...} (success) or
{"error": {...}} (failure). Customisable via response.Envelope. See
Response Envelope.
F
FieldMeta. The framework’s per-field metadata, derived from a struct
field’s json / db / mfx tags by parseFieldTags. Stored on
ModelMeta.Fields. See Models & BaseModel.
FileStorage. The interface for file backends. maniflex/storage ships
LocalStorage; custom implementations cover S3, R2, GCS. See
File Fields & Uploads.
Filter / FilterExpr. A parsed ?filter= expression. A slice of
these on ctx.Query.Filters becomes the SQL WHERE clause. See
Querying.
G
Generate (OpenAPI step). The middle step of the OpenAPI pipeline that
builds the *OpenAPISpec from the registry. After-position middleware
customises the spec. See OpenAPI Spec.
H
Handler (action). The func(ctx *maniflex.ServerContext) error registered
with server.Action(...) as an action’s body. See
Custom Endpoints.
HasMany. A relation declared as a slice field of the related struct. No column on this table; the related table carries the FK. See Relations.
HMAC column. A {field}_hmac companion column auto-created for
mfx:"encrypted,unique" fields, storing a keyed digest so uniqueness can
be enforced without comparing ciphertexts. See
Encryption at Rest.
History table. The {model}_history sibling table created for a
maniflex.ModelConfig{Versioned: true} model. Receives one row per write to
the source model. See Versioning & History.
I
Idempotency-Key. The HTTP header consumed by
middleware/idempotency to deduplicate retries. The first request runs;
subsequent requests with the same key replay the cached response. See
Idempotency.
IdentityType. The AuthInfo field classifying the principal — Human,
ServiceAccount, or Anonymous. See ServerContext.
Immutable (tag). A mfx: directive that accepts a value on create but
strips it on update. See Field Tags Reference.
Include. The ?include=relationKey,... query parameter that populates
related rows inline in the response. See Relations and
Querying.
Index (IndexSpec). A declared database index, created during
AutoMigrate. Declared in ModelConfig.Indices or auto-generated for
mfx:"scheduled" columns. See model.go.
J
Junction (model). The third model in a many-to-many relation, carrying
the two FKs. Named via mfx:"through:JunctionModel" on both sides. See
Relations.
K
KeyProvider. The interface that the encryption subsystem uses to
encrypt, decrypt, and HMAC field values. Implementations: EnvKeyProvider,
VaultKeyProvider. See Encryption at Rest.
L
LockForUpdate. A *ServerContext method that acquires a row-level
write lock inside an active transaction. SELECT ... FOR UPDATE on
Postgres; transaction-level lock on SQLite. See Transactions.
M
MiddlewareFunc. The signature every pipeline middleware must
satisfy: func(ctx *maniflex.ServerContext, next func() error) error. See
Writing Middleware.
ModelAccessor. The CRUD helper returned by ctx.GetModel(name).
Exposes List, Read, Create, Update, Delete for any registered
model, routed through ctx.Tx when set. See ServerContext.
ModelConfig. The per-model options passed alongside a struct in
MustRegister. Fields: TableName, SoftDelete, Middleware,
Versioned, VersionedDiffOnly, Indices. See Models & BaseModel.
ModelMeta. The framework’s runtime description of a registered
model. Carries Name, TableName, Fields, Relations, SoftDelete,
Config, Indices, and resolved scheduled specs.
O
OnDelete. The referential action attached to a foreign key —
cascade, setNull, restrict, or unset. See Relations.
Operation (maniflex.Operation). The CRUD verb identifying a request:
OpList, OpRead, OpCreate, OpUpdate, OpDelete, OpOptions, OpAction.
A HEAD request runs as the GET it mirrors (OpRead / OpList). See
Pipeline Overview.
Outbox. The transactional outbox pattern: a row written in the same transaction as the primary write, consumed by a background worker for external side effects (emails, webhooks, events). See Batch Operations & Sagas.
P
Pipeline. The six-step request pipeline (Auth → Deserialize → Validate → Service → DB → Response) and its sibling OpenAPI pipeline.
See Pipeline Overview.
Position. Where in a step’s chain a middleware sits — Before
(default), After, or Replace. See Writing Middleware.
Q
Query model. ctx.QueryModel (a *ServerContext method) is a read
helper that fetches any registered model through the standard List path,
returning filtered, sorted, paginated results without a full request. See
Raw Queries & Query Models.
QueryParams. The parsed ?page=&limit=&filter=&sort=&include= of
a request, stored on ctx.Query. See Querying.
R
Registry. The in-memory map of every registered *ModelMeta. Built
by MustRegister, consumed by the adapter and the router. See
Architecture.
Relation. A connection between two models — BelongsTo, HasMany,
or ManyToMany. See Relations.
Replace (position). A middleware position that substitutes the step’s
default handler entirely. The last matching Replace middleware wins.
See Writing Middleware.
S
Saga. A multi-step workflow composed of forward steps and compensating undos, usually implemented via a transactional outbox and a worker. See Batch Operations & Sagas.
Scheduled (tag / runner). mfx:"scheduled;..." declares a
time-driven transition on a *time.Time column. The
scheduled.Runner sweeps the rows and applies the transition. See
Scheduled Fields.
Service name. Config.ServiceName. Identifies the service in logs,
audit records, and the X-Service-Name response header. See
Configuration.
Soft delete. Marking a row as deleted instead of removing it. Opt in
via maniflex.WithDeletedAt (timestamp) or maniflex.WithIsDeleted (boolean). See
Soft Delete.
Step (StepRegistry). One of the six pipeline steps. Exposes
Register(fn, opts...) to attach middleware. See
Pipeline Overview.
T
Tag (mfx: directive). A comma-separated list in a struct field’s
mfx tag declaring per-field behaviour. See
Field Tags Reference.
TenantID. The AuthInfo field that scopes a principal to one tenant.
Populated by JWT auth (TenantClaim) or by custom Auth middleware. See
Auth Middleware.
Through (tag). mfx:"through:JunctionModel" declares a many-to-many
relation via a named junction model. See Relations.
Trace (PipelineTrace). Debug-level pipeline tracing controlled by
Config.Trace. Sub-flags: Steps, Timings, Aborts, Bodies,
Skips. See Configuration.
Tx. A transaction handle returned by ctx.BeginTx (or the underlying
adapter). Stored on ctx.Tx; the default DB step routes through it
automatically. See Transactions.
V
Versioned (config). ModelConfig.Versioned = true writes a row to
the sibling {model}_history table on every change to the source model.
See Versioning & History.
W
WithTransaction. The catalogue middleware that wraps the DB step in
a transaction, committing on success and rolling back on error. See
Transactions.
writeonly. A tag directive that accepts the field on input but
strips it from responses. Standard choice for passwords. See
Field Tags Reference.
FAQ & Troubleshooting
Common questions and the pitfalls that catch new users. Each entry links to the page that covers the underlying concept in depth.
Registration & startup
“I registered a model after SetDB and nothing happens.”
The adapter is built from the registry; once it’s open, later registrations don’t reach it. Register every model before opening the database:
server.MustRegister(User{}, Order{}, Invoice{}) // 1. registry populated
db, _ := sqlite.Open("./app.db", server.Registry()) // 2. adapter built
server.SetDB(db) // 3. adapter wired in
See Architecture and Models & BaseModel.
“AutoMigrate didn’t add my new column.”
AutoMigrate adds missing columns but never drops or alters existing ones.
If your struct says string and the table column is INTEGER, the
migrator leaves it alone and logs a drift warning. Inspect the warning
log, then either fix the struct or run a manual ALTER TABLE.
“Why does the framework panic when a struct doesn’t embed BaseModel?”
Because every model needs an id and timestamp columns, and BaseModel
provides them. The check is deliberate; embedding BaseModel is one
line. See Models & BaseModel.
“Can I rename the id column?”
No. The framework hard-codes id as the primary-key column name across
the adapter, the relation resolver, and the OpenAPI generator. Pick a
table prefix or rename the table instead.
Pipeline & middleware
“My middleware doesn’t fire — I scoped it to OpAction.”
OpAction requests run a trimmed pipeline: Auth → action middleware → handler → Response. Middleware registered on Validate, Service, or DB
with ForOperation(OpAction) is never reached. Move per-action logic
into the action’s own middleware list:
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/place",
Handler: placeOrder,
Middleware: []maniflex.MiddlewareFunc{auth.JWTAuth(secret), checkStock},
})
See Custom Endpoints.
“ctx.Abort was called but the database still got written to.”
Probably called next() after Abort. Abort only populates
ctx.Response; it does not stop the chain. Return without next() and
the chain unwinds. See the “Calling next() after Abort” section of
ServerContext.
“Why doesn’t After-position middleware see my error?”
It does — through ctx.Response:
func afterDB(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { return err }
if ctx.Response != nil && ctx.Response.StatusCode >= 400 {
return nil // skip the side effect
}
record(ctx)
return nil
}
A non-nil error from next() and a 4xx/5xx ctx.Response are distinct
signals — both mean “the default step refused to succeed.”
“I want the same middleware on every model except one.”
Register it without ForModel, then register a passthrough that
short-circuits for the excluded model — or guard inside the middleware:
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if ctx.Model.Name == "PublicResource" {
return next()
}
// ... usual work ...
return next()
})
Transactions
“LockForUpdate returned an error: ‘requires an active transaction’.”
Pessimistic locks only make sense inside a transaction. Wrap the call:
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil { return err }
ctx.Tx = tx
defer tx.Rollback()
row, err := ctx.LockForUpdate("StockBalance", id)
// ...
return tx.Commit()
…or register maniflex.WithTransaction(nil) on the Service step so the
transaction is already open by the time the lock fires.
“WithTransaction registered twice on SQLite — got a ‘tx already active’ error.”
SQLite does not support nested transactions. WithTransaction is
idempotent — registering it twice is fine — but calling ctx.BeginTx
manually inside an already-open SQLite transaction will fail. Reuse
ctx.Tx instead of starting a new one.
“My transaction committed even though I returned an error.”
Three things to check:
- The error was returned from
next(), not swallowed inside the middleware. - The middleware did not call
next()and return its own error — the framework treats those independently. ctx.Response.StatusCodeis>= 400ornext()returned non-nil. Both abort the commit.
Querying
“?filter=foo:eq:bar returns 400 INVALID_QUERY.”
The field isn’t filterable. Add the tag:
Foo string `json:"foo" mfx:"filterable"`
The check is intentional — exposing every column to client filters lets clients build arbitrary indexes against you. Opt in per field.
“Nested filter ?filter=author.name:eq:X doesn’t work.”
Two conditions:
- The relation must be declared on the current model (
AuthorIDfor convention,relation:Authorfor explicit). - The target field must itself be
filterableon the related model.
“I want a default sort order.”
There isn’t a built-in “default sort” tag. Register a Deserialize
middleware that appends to ctx.Query.Sorts when the client doesn’t
supply one:
server.Pipeline.Deserialize.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { return err }
if len(ctx.Query.Sorts) == 0 {
ctx.Query.Sorts = append(ctx.Query.Sorts, maniflex.SortExpr{
Field: "created_at", Direction: maniflex.SortDesc,
})
}
return nil
}, maniflex.ForModel("Article"), maniflex.ForOperation(maniflex.OpList), maniflex.AtPosition(maniflex.After))
“Soft-deleted rows show up in my admin tool list.”
They don’t — the framework filters them out everywhere. To see them, filter on the marker explicitly:
curl '...?filter=deleted_at:not_null'
See Soft Delete.
Files & encryption
“My multipart upload returns 501 NO_STORAGE.”
Config.FilesConfig.Storage is nil. Configure a backend:
fs, _ := storage.NewLocalStorage("./uploads")
server := maniflex.New(maniflex.Config{FileStorage: fs, ...})
“Encrypted field rejected my write: ENCRYPTION_NOT_CONFIGURED.”
Same shape — Config.KeyProvider is nil. Configure one (e.g.
encryption.EnvKeyProvider) before any model with mfx:"encrypted" is
exercised. See Encryption at Rest.
“I added mfx:"unique" to an encrypted field and got a duplicate-key error on legit data.”
The framework stores a keyed HMAC of the plaintext in a {field}_hmac
companion column. Two plaintexts that hash to the same digest is a
collision — astronomically unlikely with SHA-based HMAC. More likely you
re-encrypted under a new key and the old HMACs are still in the table;
re-run maniflex.RotateEncryptionKey and the digests are refreshed.
Auth
“I added auth.JWTAuth and sign-up stopped working.”
Sign-up is a write — a global auth.JWTAuth rejects it because no token
exists yet. There is no AllowPublicWrite helper; instead scope the
authenticator so it never runs on the sign-up path. Auth scoping
(ForModel / ForOperation) is inclusion-only: a middleware runs only
for the models and operations you scope it to, so anything you leave out of
every auth registration stays public.
// Protect updates and deletes everywhere…
server.Pipeline.Auth.Register(
auth.JWTAuth(secret),
maniflex.ForOperation(maniflex.OpUpdate, maniflex.OpDelete),
)
// …and protect creates only on the models that need a session. "User" is
// deliberately not listed, so POST /users (sign-up) is covered by no auth
// registration and stays public.
server.Pipeline.Auth.Register(
auth.JWTAuth(secret),
maniflex.ForModel("Post", "Comment"),
maniflex.ForOperation(maniflex.OpCreate),
)
The same inclusion-only rule applies to public reads: auth.AllowPublicRead
is a passthrough that must run before an aborting authenticator, but a
global JWTAuth aborts first — so prefer scoping JWTAuth away from the
operations you want public rather than relying on a later passthrough.
“JWT keeps returning 401 — the token validates manually.”
Three usual suspects:
- Algorithm mismatch —
HS256token verified against anRS256public key (or vice versa). SetJWTOptions.PublicKeyfor asymmetric keys. Issuer/Audiencemismatch — the framework rejects tokens whoseiss/auddoesn’t match the configured value.- Clock skew — the token’s
nbfis in the future orexpis in the past on the server clock. Sync NTP.
Database
“Why does Tenancy middleware not apply to my reads?”
It does, including reads — Tenancy filters every operation, including
list and read. If you’re seeing rows from another tenant, check that the
middleware is registered without a ForOperation filter and that
ctx.Auth.TenantID is being populated by the upstream Auth middleware.
“My Postgres reads have stale data after a write.”
Read replicas have replication lag. The framework routes reads to the
replica only outside an active transaction; reads inside a
WithTransaction-managed request go to the primary. For
read-your-writes outside a transaction, run the query through
ctx.RawQuery against an explicit primary connection, or shorten the
client’s expected window.
“I dropped a column from my struct — AutoMigrate didn’t remove it.”
By design. The migrator never drops; it logs a drift warning so you see
the column still exists. Remove it with an explicit ALTER TABLE … DROP COLUMN
during a maintenance window.
“I changed a field’s type — the column still has the old one.”
Also by design. AutoMigrate adds columns; it never rewrites one, because
converting a column’s type can lose data and locks the table while it runs. It
logs a drift warning naming the table, the column, the type the database has and
the type the model wants, at every startup. Convert the column with an explicit
versioned migration (ALTER TABLE … ALTER COLUMN, plus whatever backfill the
new type needs).
“504 TIMEOUT on a query that used to work.”
Config.QueryTimeout fired. The deadline is per request, applied to
ctx.Ctx. Either:
- Raise the timeout (
30sis a common ceiling). - Speed up the query (missing index, expensive include, large LIMIT).
- Add a
db.Paginatecap on the offending list endpoint.
A 504 always means a server-side deadline expired. A client that gives up and
disconnects mid-request is logged as 499 instead, so the two do not share a
line in your metrics.
OpenAPI
“/openapi.json is empty.”
You didn’t register any models, or you call server.Pipeline.OpenAPI.*
before MustRegister. The generator reads the registry at request time,
not registration time — but if the registry is empty when a client hits
it, the spec is empty too.
“I customised the spec but my changes don’t appear.”
Register the customisation at maniflex.After position on the Generate step,
not Before. Before runs against an empty spec; After mutates the
just-generated document.
server.Pipeline.OpenAPI.Generate.Register(
openapi.SetTitle("My API"),
maniflex.After, // <-- not the default Before
)
Production
“My pod terminated mid-request on deploy.”
Kubernetes sent SIGTERM, gave you terminationGracePeriodSeconds,
then SIGKILL’d the process. Config.ShutdownTimeout defaults to 30s;
set the pod’s grace period larger (60s is comfortable) so the graceful
path has time to complete. See Graceful Shutdown.
“Health probe returns 503 intermittently.”
If Config.HealthCheckDB is true, the probe pings the database. Tune:
Config.HealthTimeout— should be shorter than the probe’stimeoutSeconds.- The database — if the pool is exhausted,
db.Ping()waits for a connection.
“Logs are noisy with debug records.”
Config.Trace.Enabled = true was left on, or the Logger accepts DEBUG.
Set the handler’s level to INFO in production. Trace flags are opt-in
specifically because they’re high-volume.
Library design questions
“Why reflection instead of codegen?”
To avoid the regeneration step. A mfx: tag change is in effect on the
next process start; nothing to rebuild. The cost is one reflection pass
per model at boot — usually under a millisecond per model. See
Architecture.
“Can I use maniflex with GraphQL?”
The generated routes are REST. You can put GraphQL in front (gqlgen,
graphql-go) and have resolvers call ctx.GetModel(...).List / .Read
/ etc. — the model accessor doesn’t care which HTTP layer is on top.
“Can I use multiple databases?”
One adapter per *maniflex.Server. For multi-database setups, run two servers
(possibly in the same process) and route between them at the application
layer, or use raw queries from custom actions that target the secondary
database directly.
“Is there a CLI?”
Not yet. The framework is intentionally library-only — no project-scaffolding command, no migration runner. Use the standard Go toolchain and an external migration tool. The App Anatomy page describes the recommended project layout.
Where to find more
If your question isn’t here, three places to look next:
- The page for the concept involved — every link in this FAQ points to it.
- The framework’s own e2e tests under
tests/e2e/— they exercise every edge case the docs describe. - The source. The
maniflexpackage is small; reading the implementation of a step or a middleware is often faster than guessing.
AI Agents
A condensed, self-contained reference for AI coding agents working on maniflex
projects. Copy the block below into CLAUDE.md, AGENTS.md, or an equivalent
context file. Everything an agent needs to write correct maniflex code is in it;
no prose links to chase.
# maniflex — Reference for AI coding agents
Reflection-driven Go REST framework. Annotated structs in → full REST API out:
filtering, pagination, relations, soft-delete, file uploads, OpenAPI 3.1 spec.
No codegen.
## Modules
- `maniflex` — core (chi + uuid only).
- `maniflex/db/sqlite` — pure-Go SQLite (modernc.org/sqlite). No CGo.
- `maniflex/db/postgres` — lib/pq.
- `maniflex/middleware/{auth,body,validate,service,db,response,openapi,idempotency}` — catalogue.
- `maniflex/middleware/service/bcrypt` — password hashing.
- `maniflex/middleware/db/redis` — Redis rate-limit backend (`db.RateLimitConfig.Backend`).
- `maniflex/events/{kafka,nats,rabbitmq,redis}` — event publishers.
- `maniflex/jobs/redis` — background job queue.
- `maniflex/scheduled` — runner for `mfx:"scheduled"` fields.
- `maniflex/storage` — local file storage; ships `LocalStorage`.
- `maniflex/pkg/encryption` — `EnvKeyProvider`, `VaultKeyProvider`.
Each is its own module. Import only what you use.
## The four-step lifecycle (fixed order)
```go
server := maniflex.New(maniflex.Config{Port: 8080, PathPrefix: "/api"})
server.MustRegister(User{}, Post{}) // 1. populate registry
db, _ := sqlite.Open("./app.db", server.Registry()) // 2. adapter reads registry
server.SetDB(db) // 3. inject adapter
log.Fatal(server.Start()) // 4. serve
```
Models registered after `SetDB` do not reach the adapter. The adapter is built
from the registry at `Open` time.
## Models
Every model embeds `maniflex.BaseModel`:
```go
type Post struct {
maniflex.BaseModel // adds id (UUID), created_at, updated_at
maniflex.WithDeletedAt // optional — adds deleted_at (timestamp soft delete)
// maniflex.WithIsDeleted // alternative — adds is_deleted (bool)
Title string `json:"title" mfx:"required,filterable,sortable"`
Body string `json:"body" mfx:"required"`
Status string `json:"status" mfx:"required,enum:draft|published"`
UserID string `json:"user_id" mfx:"required,filterable"` // BelongsTo User
}
```
Table name = pluralised snake-case of struct name (`BlogPost` → `blog_posts`).
Override with `ModelConfig.TableName`.
Validation:
- Struct must be a struct type and embed `BaseModel`, else `MustRegister` panics.
- Field types: scalars, `*time.Time`, slices for relations, `map[string]any`,
structs for companions.
## The `mfx:` tag — complete list
Comma-separated. Whitespace trimmed. Unknown directives ignored.
**Validation:**
- `required` — must be present on create
- `enum:a|b|c` — pipe-separated allowed values
- `min:N`, `max:N` — numeric bounds
- `default:V` — used when field absent on create (cast to type)
**Write access:**
- `readonly` — stripped from all writes
- `immutable` — accepted on create, rejected on update
**Response visibility:**
- `writeonly` — accepted on write, hidden in responses (e.g. password)
- `hidden` — hidden in responses **and** stripped from create/update schemas
**Query:**
- `filterable` — usable in `?filter=`
- `sortable` — usable in `?sort=`
- `searchable` — full-text search hint
**Schema:**
- `unique` — `UNIQUE` constraint at the column
**Relations** (semicolon-separated sub-options):
- `relation:Name` — explicit FK; requires companion field `Name` of target type
- `relation:Name;onDelete:cascade|setNull|restrict` — referential action
- `through:JunctionModel` — declares M2M via junction (on slice fields)
**Files:**
- `file` — multipart file field; column stores storage key
- `max_size:N` — `KB`/`MB`/`GB` suffix or bytes
- `accept:pattern1|pattern2` — MIME-type patterns
- `auto_delete:false` — keep stored file when row deleted or field replaced
**Encryption:**
- `encrypted` — AES-256-GCM at rest; not `filterable`/`sortable`
- `key:NAME` — keyID for KeyProvider (default `"default"`)
- `encrypted,unique` — adds `{field}_hmac` companion column for uniqueness
**Versioning** (on embedded `BaseModel` field):
- `mfx:"versioned"` — adds `{model}_history` sibling table
- `mfx:"versioned:diff_only"` — store diffs only, skip snapshots
**Scheduled** (on `*time.Time` fields):
- `scheduled;soft-delete` — needs `WithDeletedAt`/`WithIsDeleted`
- `scheduled;hard-delete`
- `scheduled;field=NAME;to=VALUE` — required pair
- `scheduled;field=NAME;from=OLD;to=NEW` — guarded transition
**Exclusion:** `json:"-"`, `db:"-"`, or `mfx:"-"` removes the field entirely.
## Relations
| Kind | Declared by | Relation key |
|---|---|---|
| BelongsTo (convention) | `UserID string` field → `User` | `user` |
| BelongsTo (explicit) | `ManagerID string \`mfx:"relation:Manager"\`` + `Manager User` companion | `manager` |
| HasMany | `Posts []Post` slice field | `posts` |
| ManyToMany | `Tags []Tag \`mfx:"through:ProductTag"\`` on both sides + junction model registered | `tags` |
Include in queries: `?include=user,posts,tags`. Nested filter: `?filter=user.role:eq:admin`.
Junction models for M2M are registered like any model.
## The 6-step pipeline
`Auth → Deserialize → Validate → Service → DB → Response`. Each step has a
default + `*StepRegistry` on `server.Pipeline`.
Middleware signature:
```go
type MiddlewareFunc func(ctx *maniflex.ServerContext, next func() error) error
```
Register:
```go
server.Pipeline.Service.Register(myFn,
maniflex.ForModel("User", "Order"), // optional, by struct name
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate), // optional
maniflex.AtPosition(maniflex.Before), // Before (default) | After | Replace
maniflex.WithName("my-fn"), // optional, for traces
)
```
Operations: `OpList`, `OpRead`, `OpCreate`, `OpUpdate`, `OpDelete`, `OpOptions`, `OpAction`. A `HEAD` request runs as the `GET` it mirrors (`OpRead` / `OpList`); `OpHead` is never set.
`OpAction` uses a trimmed pipeline: `Auth → action middleware → handler → Response`.
Validate/Service/DB middleware never fires for actions.
**Short-circuit pattern (always pair these):**
```go
ctx.Abort(http.StatusUnauthorized, "UNAUTHORIZED", "missing token")
return nil // do NOT call next()
```
Calling `next()` after `Abort` lets downstream steps run and possibly overwrite `ctx.Response`. Always return without `next()`.
**Default step behaviours:**
- Auth: passthrough. Populate `ctx.Auth` here.
- Deserialize: parses query params → `ctx.Query`; body → `ctx.ParsedBody`; multipart → `ctx.Files`. 4 MB JSON body limit (multipart is `FilesConfig.MaxUploadBytes`, default 32 MB).
- Validate: enforces `mfx:` tag rules on create/update. Strips `readonly`/`id`; strips `immutable` on update.
- Service: passthrough. Business logic goes here.
- DB: dispatches to adapter. Routes through `ctx.Tx` when set. Maps `ErrNotFound`→404, `*ErrConstraint`→409, `context.DeadlineExceeded`→504, client disconnect→499 (`maniflex.StatusClientClosedRequest`, no body).
- Response: builds `APIResponse` from `ctx.DBResult`. List adds `meta`. Delete returns 204.
## ServerContext (per-request, not goroutine-safe)
Common fields:
- `Request *http.Request`, `Writer http.ResponseWriter`, `Ctx context.Context`
- `Model *ModelMeta`, `Operation Operation`, `ResourceID string`
- `RequestID string`, `TraceID string`
- `RawBody []byte`, `ParsedBody *RequestBody` (read-only — `ctx.Field` to read, `ctx.SetField`/`DeleteField` to mutate), `Query *QueryParams`, `Files map[string]*UploadedFile`
- `DBResult any` (`*ListResult` for list; the record otherwise — a typed `*T` on reads)
- `Response *APIResponse`
- `Auth *AuthInfo`, `Tx Tx`
Methods:
- `Abort(status int, code, message string)` — sets `ctx.Response`; caller returns nil without `next()`.
- `BindJSON(v any) error` — decode body into `v`; calls Abort on error.
- `URLParam(name) string`, `QueryParam(name) string`
- `Set(k, v)` / `Get(k) (any, bool)` — cross-step storage
- `Logger() *slog.Logger` — pre-seeded with request_id, trace_id, service
- `HasRole(role string) bool`
- `BeginTx(ctx, opts *TxOptions) (Tx, error)` — start a transaction
- `LockForUpdate(modelName, id) (map[string]any, error)` — requires `ctx.Tx`
- `RawQuery(sql, args...) ([]map[string]any, error)` — routes through ctx.Tx
- `RawExec(sql, args...) (int64, error)`
- `GetModel(name) *ModelAccessor` — CRUD on any registered model; routes through ctx.Tx
ModelAccessor methods: `List(q)`, `Read(id)`, `Create(data)`, `Update(id, data)`, `Delete(id)`.
`AuthInfo`:
```go
type AuthInfo struct {
UserID string
Roles []string
Claims map[string]any
TenantID string
IdentityType AuthIdentityType // IdentityHuman | IdentityServiceAccount | IdentityAnonymous
Scopes []string
SessionID string
AuthMethod string // "jwt" | "api_key" | "session" | ...
}
```
## Transactions
```go
// Option A — middleware-wrapped, automatic commit/rollback
server.Pipeline.Service.Register(
maniflex.WithTransaction(nil), // nil opts = default isolation
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete),
)
// Option B — manual
tx, err := ctx.BeginTx(ctx.Ctx, nil)
if err != nil { return err }
ctx.Tx = tx
defer tx.Rollback() // no-op after Commit
// ... ctx.GetModel(...).Create/Update/Delete all routed through tx ...
return tx.Commit()
```
`WithTransaction` commits if `next()` returned nil AND `ctx.Response` is nil or 2xx. Otherwise rolls back. Idempotent — registering twice is fine, second call sees existing `ctx.Tx`.
SQLite does not support nested transactions. Its write connections open with `BEGIN IMMEDIATE` (`_txlock=immediate`, set by `sqlite.Open`), so read-then-write transactions serialise instead of racing.
## Errors
Sentinels (use `errors.Is` / `errors.As`):
```go
maniflex.ErrNotFound // 404 NOT_FOUND
*maniflex.ErrConstraint // 409 CONFLICT (Table, Column, Detail)
```
Built-in error codes the framework emits:
`INVALID_JSON`, `EMPTY_BODY`, `BODY_READ_ERROR`, `INVALID_QUERY`,
`MULTIPART_ERROR`, `NOT_FOUND`, `CONFLICT`, `VALIDATION_ERROR`,
`DB_ERROR`, `TX_BEGIN_ERROR`, `TX_COMMIT_ERROR`, `NO_STORAGE`,
`TIMEOUT`, `PANIC`, `ENCRYPTION_NOT_CONFIGURED`.
Envelope: `{"error": {"code": "...", "message": "...", "details": ...}}`
Success envelope: `{"data": ...}`; list adds `"meta": {total, page, limit, pages}`.
## Querying (only on opt-in fields)
Operators: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `like`, `ilike`, `contains`, `starts_with`, `ends_with`, `in`, `not_in`, `between`, `is_null`, `not_null`.
`like`/`ilike` take a raw SQL pattern (`%`, `_` are wildcards). `contains`/`starts_with`/`ends_with` take a literal value — `%` and `_` are escaped and match themselves — and are case-insensitive. Use the latter for user-typed text.
```
?filter=status:eq:published
?filter=views:gte:100&filter=status:eq:published # ANDed
?filter=tag:in:go,rust,zig
?filter=title:contains:intro # literal, case-insensitive
?filter=author.name:ilike:%ursula% # relation dot notation
?sort=created_at:desc&sort=title:asc
?include=user,comments,tags
?page=2&limit=20 # default 20, max 200
```
Soft-deleted rows are filtered out of list/read/include automatically. To see them: `?filter=deleted_at:not_null`.
## Custom Actions
```go
server.Action(maniflex.ActionConfig{
Method: "POST",
Path: "/orders/{id}/cancel",
Handler: cancelOrder,
Middleware: []maniflex.MiddlewareFunc{
auth.JWTAuth(secret),
// any maniflex.MiddlewareFunc — runs between Auth and the handler
},
})
func cancelOrder(ctx *maniflex.ServerContext) error {
id := ctx.URLParam("id")
var req MyReq
if err := ctx.BindJSON(&req); err != nil { return nil }
// ... work via ctx.GetModel / ctx.RawExec / ctx.BeginTx ...
ctx.Response = &maniflex.APIResponse{StatusCode: http.StatusOK, Data: result}
return nil
}
```
Action handler owns body parsing, validation, transactions. Validate/Service/DB pipeline steps do NOT run for actions.
## File uploads
Tag a string field `mfx:"file,max_size:2MB,accept:image/*"`. Configure `maniflex.Config.FilesConfig.Storage`.
Two upload styles:
1. Multipart POST/PATCH to the model endpoint — fields become `ctx.ParsedBody`, file parts become `ctx.Files`, storage key is written into the column.
2. Two-step: `POST /files` (multipart, field name `file`) returns `{"data":{"key":...}}`. Pass the key as a JSON string on the file field.
Standalone routes (when `FileStorage` set):
- `POST /files` — upload
- `GET /files/{key...}` — download (sets Content-Type, Content-Disposition, Content-Length)
- `DELETE /files/{key...}`
Without `FileStorage`, multipart and `/files/*` return 501 NO_STORAGE.
Auto-cleanup: stored file deleted on hard-delete or field overwrite. `auto_delete:false` opts out. Soft-delete does not trigger cleanup.
## Catalogue middleware
```go
import (
"github.com/xaleel/maniflex/middleware/auth"
"github.com/xaleel/maniflex/middleware/body"
"github.com/xaleel/maniflex/middleware/validate"
"github.com/xaleel/maniflex/middleware/service"
"github.com/xaleel/maniflex/middleware/db"
"github.com/xaleel/maniflex/middleware/response"
"github.com/xaleel/maniflex/middleware/openapi"
"github.com/xaleel/maniflex/middleware/idempotency"
)
// AUTH (Auth step)
auth.JWTAuth(secret, auth.JWTOptions{Issuer, Audience, TenantClaim, ScopesClaim, PublicKey})
auth.APIKeyAuth("X-API-Key", auth.APIKeyEntry{Key, Auth: maniflex.AuthInfo{...}}, ...)
auth.RequireRole("admin")
auth.AllowPublicRead() // passthrough on read/list; must run BEFORE an aborting authenticator
auth.BlockOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete)
// (no AllowPublicWrite — keep an op public by NOT scoping JWTAuth onto it; scoping is inclusion-only)
// BODY (Deserialize / Validate steps)
body.MaxBodySize(16 << 20) // override 4MB JSON default (also caps multipart, default 32MB — this LOWERS it)
body.StripUnknownFields()
body.CoerceTypes()
// VALIDATE (Validate step)
validate.UniqueField(sqlDB, maniflex.SQLite, "email") // driver: maniflex.Postgres | maniflex.SQLite
validate.RegexField("phone", `^\+?[0-9]{7,15}$`)
validate.ForbiddenValues("role", "superadmin")
validate.RequireAtLeastOne("name", "email")
validate.CrossFieldValidate(func(body map[string]any) error { return ... })
validate.DateRange("start_date", "end_date") // end must not be before start
validate.RequireWhen("reason", "status:eq:rejected") // conditional required
// SERVICE (Service step — usually Before)
service.HashField("password", svcbcrypt.Hasher()) // import middleware/service/bcrypt
service.SlugifyField("title", "slug")
service.SetField("user_id", func(ctx) any { return ctx.Auth.UserID })
service.StripField("password_confirm")
service.CopyField("email", "billing_email")
service.Timestamp("last_seen_at")
service.TimestampWhen("published_at", "status", "published")
service.OwnerScope("user_id") // unconditional SetField on create
// Side effects are in package events (NOT service):
events.Emit(bus) // DB-step middleware, AtPosition(After)
// Webhook/SendEmail are event-bus SUBSCRIBERS, not middleware — wire with bus.Subscribe:
bus.Subscribe(ctx, events.Subscription{Patterns: []string{"order.*"},
Handler: events.Webhook(events.WebhookConfig{URL: u, Secret: s})})
// DB (DB step)
db.ForceFilter("org_id", func(ctx) any { return ctx.Auth.Claims["org_id"] })
db.Tenancy("org_id", func(ctx) string { return ctx.Auth.TenantID })
db.Paginate(50)
db.RateLimit(db.RateLimitConfig{RequestsPerMinute: 10, KeyFunc: func(ctx) string {...}}) // Backend: for a cross-replica counter
db.AuditLog(sink) // AtPosition(After), or default Before with db.WithChanges()
db.Invalidate(cache, func(ctx) []string { return ["keys", ...] }) // AtPosition(After)
// RESPONSE (Response step)
response.CORSHeaders("https://app.example.com") // origins required; "*" allowed but panics with credentials
response.Cache(300) // AtPosition(After)
response.TransformField("avatar_url", func(v any) any { return cdn+v.(string) })
response.RedactField("phone", func(ctx) bool { return !ctx.HasRole("support") })
response.Envelope(func(ctx, data, meta) any { return ... })
response.AddHeader("Strict-Transport-Security", "max-age=63072000")
response.Logging(slog.Default()) // AtPosition(After)
response.Metrics(collector) // AtPosition(After)
// OPENAPI (OpenAPI.Generate step, maniflex.After position)
openapi.SetTitle("My API")
openapi.SetDescription("...")
openapi.AddServer("https://api.example.com", "Production")
openapi.AddSecurityScheme("bearerAuth", maniflex.OASSecurityScheme{Type: "http", Scheme: "bearer"})
openapi.AddExtension(func(spec *maniflex.OpenAPISpec) { /* mutate freely */ })
// IDEMPOTENCY (Deserialize step, AtPosition(After), scoped to OpCreate)
idempotency.Middleware(idempotency.Config{
Store: maniflex.NewMemoryCache(), // or any maniflex.CacheStore (e.g. Redis)
TTL: 24 * time.Hour,
KeyFunc: func(ctx) string { return ctx.Auth.UserID },
HeaderRequired: false,
})
// Reads Idempotency-Key header. Replays cached 2xx for same key+body.
// Same key + different body → 422 IDEMPOTENCY_KEY_REUSED.
// Adds "Idempotent-Replayed: true" on replay.
```
## Encryption at rest
```go
import "github.com/xaleel/maniflex/pkg/encryption"
server := maniflex.New(maniflex.Config{
KeyProvider: &encryption.EnvKeyProvider{Prefix: "MYAPP_KEY"},
// or: &encryption.VaultKeyProvider{Address, Token, Mount: "transit"}
})
type Patient struct {
maniflex.BaseModel
SSN string `json:"ssn" mfx:"encrypted,key:patient-pii"` // → MYAPP_KEY_PATIENT_PII env var
}
```
- Storage: `enc:<base64(envelope)>`.
- `EnvKeyProvider` needs base64-encoded 32-byte keys in env vars.
- Encrypted fields cannot be `filterable`/`sortable`.
- `encrypted,unique` adds `{field}_hmac TEXT UNIQUE` companion column.
- `maniflex.RotateEncryptionKey(ctx, server, "Model", oldKeyID, newKeyID)` re-encrypts in pages of 100. Keep both keys active.
## Versioning
```go
server.MustRegister(Invoice{}, maniflex.ModelConfig{Versioned: true})
// or VersionedDiffOnly: true to skip snapshots
```
Creates `invoice_history` table (the source name + `_history`, not pluralised) with columns: `id, record_id, version, operation, actor_id, timestamp, request_id, diff, [snapshot]`.
Both the diff **and** the snapshot exclude hidden/writeonly/encrypted fields and HMAC companions — a history row is built from the decrypted record, so anything left in would be stored as plaintext (audit MS-3).
Read history at `GET /invoices/{id}/history` — newest first, `?page=`/`?limit=` (default 20). The history model is `Headless`: there is **no** `/invoice_history` endpoint. It runs the parent's read pipeline, so a caller who cannot read the record gets the same `404` for its history; scope with `ForOperation(maniflex.OpRead, maniflex.OpReadHistory)`. A soft-deleted record keeps readable history; a hard-deleted one does not (nothing left to authorise against).
## Scheduled runner
```go
import "github.com/xaleel/maniflex/scheduled"
runner, _ := scheduled.New(server, scheduled.Config{
Interval: time.Minute,
BatchSize: 500,
OnDelete: func(model, id string) { ... },
OnSetField: func(model, id, field, to string) { ... },
})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
runner.Start(ctx)
defer runner.Stop()
```
Scans `*time.Time` columns with `mfx:"scheduled"` tag. Per-model transactional batches. Hooks fire after commit. Use `runner.Sweep(ctx)` for one-shot ticks.
For multi-replica deployments: run runner in one replica, or use `maniflex/scheduled/jobsx` to dispatch sweeps through a job queue.
## Configuration
```go
type Config struct {
Port int // default 8080
PathPrefix string // default "/api"
ServiceName string // adds "service" attr to logs, X-Service-Name header
DB DBAdapter // required before Start
DisableAutoMigrate bool // migration runs by default; set to skip it
QueryTimeout time.Duration // per-request DB deadline; 0 = unlimited
ShutdownTimeout time.Duration // default 30s
Logger *slog.Logger
PanicLogger *slog.Logger
Trace PipelineTrace // {Enabled, Steps, Timings, Aborts, Bodies, Skips}
FileStorage FileStorage
KeyProvider KeyProvider
HealthCheckDB bool // GET /health pings DB
HealthTimeout time.Duration // default 3s
}
```
`maniflex.ConfigFromEnv(prefix) (Config, error)` reads PORT, DB_WRITE_URL, DB_READ_URL, QUERY_TIMEOUT_MS, SHUTDOWN_TIMEOUT_S, SERVICE_NAME, HEALTH_CHECK_DB — those and no others. A non-empty prefix is applied with an underscore (`ORDERS_PORT`). Unset variables are left zero for `ApplyDefaults`; a variable that is set but unreadable (`PORT=808O`) returns an error naming it. Do not discard the error.
## Database adapters
```go
// SQLite (dev) — write connections get _txlock=immediate automatically.
db, err := sqlite.Open("./app.db", server.Registry())
db, err := sqlite.Open(":memory:", server.Registry())
// PostgreSQL (prod) — Open(writeDSN, readDSN, registry) is positional.
db, err := postgres.Open(os.Getenv("DB_WRITE_URL"), os.Getenv("DB_READ_URL"), server.Registry())
// Pool/session tuning → OpenWithConfig(writeDSN, readDSN, registry, writePool, readPool, session):
db, err := postgres.OpenWithConfig(
os.Getenv("DB_WRITE_URL"), os.Getenv("DB_READ_URL"), server.Registry(),
postgres.PoolConfig{MaxOpenConns: 25, MaxIdleConns: 5, ConnMaxLifetime: 30 * time.Minute}, // write
postgres.PoolConfig{MaxOpenConns: 25, MaxIdleConns: 5, ConnMaxLifetime: 30 * time.Minute}, // read
postgres.SessionConfig{ApplicationName: "myapp"},
)
```
Reads route to ReadURL outside an active transaction; reads inside a tx go to write primary. AutoMigrate adds missing columns; never drops. Logs drift warnings.
## ModelConfig (per-model options)
```go
server.MustRegister(MyModel{}, maniflex.ModelConfig{
TableName: "custom_table",
SoftDelete: maniflex.SoftDeleteConfig{Enabled: true, Field: "deleted_at", FieldType: maniflex.SoftDeleteTimestamp},
Middleware: &maniflex.ModelMiddleware{
Auth: []maniflex.MiddlewareFunc{...},
Deserialize: []maniflex.MiddlewareFunc{...},
Validate: []maniflex.MiddlewareFunc{...},
Service: []maniflex.MiddlewareFunc{...},
DB: []maniflex.MiddlewareFunc{...},
Response: []maniflex.MiddlewareFunc{...},
},
Versioned: true,
VersionedDiffOnly: false,
Indices: []maniflex.IndexSpec{{Name, Columns, Unique}},
})
```
`Register` accepts `...any`. Slice arguments are flattened one level — so you can do:
```go
var AuthModels = []any{User{}, Role{}}
var OrderModels = []any{Order{}, OrderLine{}}
server.MustRegister(AuthModels, OrderModels) // both flattened
```
A `ModelConfig` value applies to the model immediately preceding it.
## Common patterns
### Public sign-up (scope auth away from it; inclusion-only)
```go
// Protect updates/deletes everywhere; protect creates only where needed.
// "User" is not scoped, so POST /users (sign-up) stays public.
server.Pipeline.Auth.Register(auth.JWTAuth(secret),
maniflex.ForOperation(maniflex.OpUpdate, maniflex.OpDelete))
server.Pipeline.Auth.Register(auth.JWTAuth(secret),
maniflex.ForModel("Post", "Comment"), maniflex.ForOperation(maniflex.OpCreate))
```
### Hash password on User create/update
```go
server.Pipeline.Service.Register(service.HashField("password", svcbcrypt.Hasher()), // import middleware/service/bcrypt
maniflex.ForModel("User"), maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate))
```
### Multi-tenancy
```go
server.Pipeline.DB.Register(db.Tenancy("org_id",
func(ctx *maniflex.ServerContext) string { return ctx.Auth.TenantID }))
```
### Audit every write
```go
server.Pipeline.DB.Register(db.AuditLog(sink, db.WithChanges()),
maniflex.ForOperation(maniflex.OpCreate, maniflex.OpUpdate, maniflex.OpDelete))
// Note: WithChanges() requires default Before position, NOT After.
```
### Transactional create with stock lock
```go
server.Pipeline.Service.Register(maniflex.WithTransaction(nil),
maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
server.Pipeline.Service.Register(func(ctx *maniflex.ServerContext, next func() error) error {
bookID, _ := ctx.Field("book_id")
book, err := ctx.LockForUpdate("Book", bookID.(string))
if err != nil { return err }
if book["stock"].(int64) < 1 {
ctx.Abort(http.StatusConflict, "OUT_OF_STOCK", "")
return nil
}
return next()
}, maniflex.ForModel("Order"), maniflex.ForOperation(maniflex.OpCreate))
```
### Custom default sort
```go
server.Pipeline.Deserialize.Register(func(ctx *maniflex.ServerContext, next func() error) error {
if err := next(); err != nil { return err }
if len(ctx.Query.Sorts) == 0 {
ctx.Query.Sorts = append(ctx.Query.Sorts, maniflex.SortExpr{
Field: "created_at", Direction: maniflex.SortDesc,
})
}
return nil
}, maniflex.ForModel("Article"), maniflex.ForOperation(maniflex.OpList), maniflex.AtPosition(maniflex.After))
```
### Background worker reading registered models
```go
// In a goroutine launched alongside server.Start(). Background code has no
// ServerContext — build one with NewBackground, then use ctx.GetModel.
bg := maniflex.NewBackground(context.Background(), server.DB(), server.Registry())
events := bg.GetModel("OutboxEvent")
rows, _ := events.List(&maniflex.QueryParams{
Filters: []*maniflex.FilterExpr{{Field: "status", Operator: maniflex.OpEq, Value: "pending"}},
Limit: 20,
})
for _, ev := range rows {
// ... process ...
events.Update(ev["id"].(string), map[string]any{"status": "done"})
}
```
## Project layout (recommended)
Small app — single `main.go`. Past ~5 models, split by responsibility:
```
main.go wiring only (4 steps)
config.go maniflex.Config assembly
models/ one file per model
middleware/ custom middleware + register.go
actions/ custom action handlers
internal/ framework-free business logic
```
Large monolith — split by domain:
```
domains/auth/ models.go + middleware.go + register.go (exports Models []any and Register(s))
domains/orders/ ditto
domains/catalog/ ditto
main.go server.MustRegister(auth.Models, orders.Models, catalog.Models)
auth.Register(server); orders.Register(server); catalog.Register(server)
```
## Hard rules (ordered by frequency of being violated)
1. **Register models before `sqlite.Open` / `postgres.Open`.** Adapter reads
registry at open time.
2. **After `ctx.Abort`, return without `next()`.** Abort only sets `ctx.Response`;
it does not stop the chain.
3. **Soft-deleted rows are auto-filtered from list/read/include.** To see them,
filter on the marker (`?filter=deleted_at:not_null`).
4. **`OpAction` skips Validate/Service/DB.** Per-action middleware list runs
between Auth and the handler. The handler owns body parsing.
5. **`WithTransaction` on Service step requires `maniflex.Before` position** (default).
Or use `maniflex.Replace` on the DB step.
6. **`db.AuditLog(sink, db.WithChanges())` must be registered at `maniflex.Before`
position**, not After — it needs the pre-image.
7. **Encrypted fields can't be `filterable` or `sortable`.** For uniqueness use
`encrypted,unique` (adds HMAC companion).
8. **`AutoMigrate` never drops columns.** Removes need explicit `ALTER TABLE`.
9. **Configure `FileStorage` before any model with `mfx:"file"` is exercised.**
Without it, multipart uploads return 501.
10. **SQLite has no nested transactions.** Calling `BeginTx` inside an active
SQLite tx fails. Reuse `ctx.Tx`.
11. **`Handler()` does not migrate.** Only `Start()` runs auto-migration. When
mounting `Handler()` yourself, call `MigrateOnly`/`AutoMigrate` first.
## Testing pattern
```go
func newTestServer(t *testing.T) (*httptest.Server, *maniflex.Server) {
t.Helper()
server := maniflex.New(maniflex.Config{Port: 0, PathPrefix: "/api"})
server.MustRegister(User{}, Post{})
db, err := sqlite.Open(":memory:", server.Registry())
if err != nil { t.Fatal(err) }
t.Cleanup(func() { db.Close() })
server.SetDB(db)
// Handler() does not migrate — only Start() does. Migrate explicitly.
if err := server.MigrateOnly(context.Background()); err != nil { t.Fatal(err) }
middleware.Register(server)
ts := httptest.NewServer(server.Handler())
t.Cleanup(ts.Close)
return ts, server
}
```
In-memory SQLite per test. `server.Handler()` returns the chi router but does
**not** migrate — `MigrateOnly` creates the tables up front.
## Sentinels & constants
```go
maniflex.OpCreate, maniflex.OpRead, maniflex.OpUpdate, maniflex.OpDelete, maniflex.OpList, maniflex.OpOptions, maniflex.OpAction
maniflex.Before, maniflex.After, maniflex.Replace
maniflex.OpEq, maniflex.OpNeq, maniflex.OpGt, maniflex.OpGte, maniflex.OpLt, maniflex.OpLte, maniflex.OpLike, maniflex.OpILike, maniflex.OpIn, maniflex.OpNotIn, maniflex.OpIsNull, maniflex.OpNotNull, maniflex.OpBetween, maniflex.OpContains, maniflex.OpStartsWith, maniflex.OpEndsWith
maniflex.SortAsc, maniflex.SortDesc
maniflex.OnDeleteCascade, maniflex.OnDeleteSetNull, maniflex.OnDeleteRestrict, maniflex.OnDeleteNoAction
maniflex.IdentityHuman, maniflex.IdentityServiceAccount, maniflex.IdentityAnonymous
maniflex.SoftDeleteTimestamp, maniflex.SoftDeleteBool
maniflex.SchedSoftDelete, maniflex.SchedHardDelete, maniflex.SchedSetField
maniflex.ErrNotFound // sentinel error
*maniflex.ErrConstraint // typed error with Table, Column, Detail
maniflex.ErrNoAdapter
maniflex.ErrFileNotFound
```
## What maniflex does NOT do
- No codegen. Reflection at registration only.
- No GraphQL.
- No built-in CLI. Use standard `go` tooling.
- No multi-database per server instance. One adapter per `*maniflex.Server`.
- No automatic column drops in AutoMigrate. Manual `ALTER TABLE` required.
- No SQLite nested transactions.
- No magic — every behaviour is a function in the `maniflex` package or a
catalogue middleware. Read the source when in doubt.