Story 026: a UAT archive instance (#139). Purpose uat at slot 1, bulk writers allowed; the Taskfile works on UAT, production only by promote
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
parent
6786eabeba
commit
4dceb5fda9
17 changed files with 156 additions and 70 deletions
|
|
@ -1,5 +1,5 @@
|
|||
# Copy to .env (gitignored). Both are optional.
|
||||
# The archive (or mirror) task build and task serve read. Default: ~/.local/share/bytestone/archive
|
||||
#BYTESTONE_ARCHIVE=/home/me/.local/share/bytestone/archive
|
||||
# The archive (or mirror) task build and task serve read. Default: ~/.local/share/bytestone/uat, the mirror of UAT (story 026)
|
||||
#BYTESTONE_ARCHIVE=/home/me/.local/share/bytestone/uat
|
||||
# Where task serve listens. Default: localhost:8321+slot, the instance's slot address
|
||||
#BYTESTONE_ADDR=localhost:8321
|
||||
|
|
|
|||
46
README.md
46
README.md
|
|
@ -125,14 +125,16 @@ are to items the same person holds; another's is shown as a bare id.
|
|||
| Brand page | `docs/brand.html`: the palette and ramp, the type, the stone, tiles and the day's margin as the build draws them (`life-weaver brand DIR`) |
|
||||
| Archive | The primary store: a directory per person ([docs/archive.md](docs/archive.md)) |
|
||||
| Archive instance | One archive tree, named by its `archive.yaml` (`name`, `purpose`, `slot`); `build` and `serve` report which they are on |
|
||||
| Slot | An instance's place among the ten that may serve at once, 0 to 9, and so its address: `serve` listens on `localhost:8321+slot`. Production is slot 0, so its clients default to `localhost:8321`; a scratch or scale instance names its own |
|
||||
| Production | The one instance on the NAS, `purpose: production`; bulk writers refuse it |
|
||||
| Slot | An instance's place among the ten that may serve at once, 0 to 9, and so its address: `serve` listens on `localhost:8321+slot`. Production is slot 0, so its clients default to `localhost:8321`, and UAT is slot 1, said or not; a scratch or scale instance names its own, 2 to 9 |
|
||||
| Production | The archive of record, on the NAS, `purpose: production`; bulk writers refuse it, and nothing writes it but a promotion |
|
||||
| UAT | The staging instance beside production on the NAS, `purpose: uat`, slot 1: a full copy, people and items, that every depositor writes and i.bytestone.uk is built from; bulk writers may write it (story 026) |
|
||||
| Promotion | `task promote`: production becomes exactly what UAT is now, tree and site |
|
||||
| Item | One archived thing with its own id, kept as a bag in its home person's directory |
|
||||
| Kind | What an item is: a *photo*, a print scanned here with `photo.yaml` beside it, or a *page*, with `page.yaml`. Read off the bag; the receipt names it |
|
||||
| Page | A file received from elsewhere (a copy emailed, a phone photograph of prints), kept unaltered as an item of its own; the prints on it are photos cut from it, as from a bed scan |
|
||||
| Cut from | A photo's tie to the page it was cut from and where on it the print lay; the page lists its photos |
|
||||
| Supersedes | A later scan's tie to the copy of the same print it replaces; the site shows the scan in the copy's place and the copy stays |
|
||||
| Mirror | The laptop's copy of the archive's people tree without items and renditions, which the build reads |
|
||||
| Mirror | The laptop's copy of UAT's people tree without items and renditions, which the build reads |
|
||||
| Site | The static output: `/` the front page, `/<id>/` page, `/<id>/<file>` downloads, `/<id>/<name>/` document pages |
|
||||
| Front page | The featured life with their family as the hourglass chart, the count of lives, the slogan and the search (stories 006, 020) |
|
||||
| Featured life | The person the front page opens with: the best-recorded (portrait and bio) with the most kin, else the best-connected |
|
||||
|
|
@ -167,25 +169,29 @@ page there (story 025).
|
|||
## Usage
|
||||
|
||||
```sh
|
||||
task archive:pull # NAS → mirror (~/.local/share/bytestone/archive): archive.yaml and the people tree, less items and renditions
|
||||
task uat:seed # once: UAT on the NAS copied from production entire, named uat
|
||||
task archive:pull # UAT on the NAS → mirror (~/.local/share/bytestone/uat): archive.yaml and the people tree, less items and renditions
|
||||
task person:pull ID=0 # one person entire, items included
|
||||
task build # mirror → site/, then the front page's wasm (task wasm)
|
||||
task archive:push # mirror → NAS (adds and updates; never deletes on the NAS)
|
||||
task archive:push # mirror → UAT on the NAS (adds and updates; never deletes on the NAS)
|
||||
task person:push ID=0 # one person entire
|
||||
task serve # the archive service's API over the mirror (below)
|
||||
task preview # preview site/ locally
|
||||
task publish # mirror → NAS, NAS → mirror, then docs and site to their hosts (tp pages deploy)
|
||||
task promote # i.bytestone.org becomes what i.bytestone.uk serves now
|
||||
task publish # mirror → UAT, UAT → mirror, then docs and site to their hosts (tp pages deploy)
|
||||
task promote:diff # what promote would change in production, deletions included
|
||||
task promote # production becomes what UAT is now: the tree on the NAS, and i.bytestone.org what i.bytestone.uk serves
|
||||
task check # fmt, vet, lint, test
|
||||
task perf # the search ladder: GET /people?q= at each scale of register
|
||||
```
|
||||
|
||||
Edit a record in the mirror, then `task archive:push` or `person:push`.
|
||||
`task publish` brings the mirror and the NAS level, builds and sends
|
||||
`site/` and `docs/` to their hosts. The site it sends is i.bytestone.uk,
|
||||
where a publish is looked at; `task promote` then copies it, as served,
|
||||
over i.bytestone.org, the production site of lives, on the host itself.
|
||||
Production changes only by a promotion.
|
||||
The mirror is UAT's. Edit a record in it, then `task archive:push` or
|
||||
`person:push`. `task publish` brings the mirror and UAT on the NAS level,
|
||||
builds and sends `site/` and `docs/` to their hosts. The site it sends is
|
||||
i.bytestone.uk, where a publish is looked at; `task promote` then makes
|
||||
production what UAT is, the tree on the NAS, deletions included, and the
|
||||
site as served copied over i.bytestone.org, the production site of
|
||||
lives, on the host itself. Production changes only by a promotion, and
|
||||
`task promote:diff` shows what one would do.
|
||||
|
||||
`.env` (gitignored; `.env.example`) may set `BYTESTONE_ARCHIVE`, the
|
||||
archive the build and the service read, and `BYTESTONE_ADDR`, where the
|
||||
|
|
@ -196,8 +202,8 @@ Every archive instance names itself in `archive.yaml` at its root, and
|
|||
|
||||
```yaml
|
||||
name: bytestone
|
||||
purpose: production # or scratch, or scale
|
||||
slot: 0 # production's; a scratch or scale instance takes 1 to 9
|
||||
purpose: production # or uat, scratch or scale
|
||||
slot: 0 # production's; uat is 1; a scratch or scale instance takes 2 to 9
|
||||
```
|
||||
|
||||
`serve` listens on `localhost:8321+slot` unless `-addr` says otherwise,
|
||||
|
|
@ -207,11 +213,11 @@ refuses, naming what has the address. So several instances serve at once,
|
|||
each where its slot puts it, and a scratch run cannot take production's
|
||||
address by accident.
|
||||
|
||||
The mirror is production's copy and carries production's file, so it is
|
||||
production for this purpose. Anything that writes in bulk (a synthetic
|
||||
generator, a mass import) calls `archive.Load(root)` and then
|
||||
`Instance.Bulk()`, which refuses `production`; a scratch or scale
|
||||
instance is any other tree with its own file.
|
||||
Anything that writes in bulk (a synthetic generator, a mass import)
|
||||
calls `archive.Load(root)` and then `Instance.Bulk()`, which refuses
|
||||
`production`. UAT takes bulk writes: that is what it is for, and the
|
||||
mirror is UAT's, carrying UAT's file. A scratch or scale instance is any
|
||||
other tree with its own file.
|
||||
|
||||
## The archive service
|
||||
|
||||
|
|
|
|||
38
Taskfile.yml
38
Taskfile.yml
|
|
@ -4,17 +4,23 @@ version: '3'
|
|||
dotenv: ['.env']
|
||||
|
||||
vars:
|
||||
# The bytestone archive: the primary store, on the NAS. See docs/archive.md.
|
||||
# The bytestone archive, on the NAS (docs/archive.md). PRODUCTION is the
|
||||
# archive of record, written only by task promote; ARCHIVE is UAT, the
|
||||
# staging copy every depositor writes and the site is built from (story
|
||||
# 026), which the archive tasks work on.
|
||||
NAS: humphrey@10.0.0.1
|
||||
NAS_PORT: 5522
|
||||
ARCHIVE: /volume1/H3Archive/bytestone
|
||||
# The archive the build and the service read: the laptop's mirror unless .env says otherwise.
|
||||
MIRROR: '{{.BYTESTONE_ARCHIVE | default (printf "%s/.local/share/bytestone/archive" .HOME)}}'
|
||||
PRODUCTION: /volume1/H3Archive/bytestone
|
||||
ARCHIVE: /volume1/H3Archive/bytestone-uat
|
||||
# The archive the build and the service read: the laptop's mirror of UAT unless .env says otherwise.
|
||||
MIRROR: '{{.BYTESTONE_ARCHIVE | default (printf "%s/.local/share/bytestone/uat" .HOME)}}'
|
||||
# Where task serve listens.
|
||||
ADDR: '{{if .BYTESTONE_ADDR}}-addr {{.BYTESTONE_ADDR}}{{end}}'
|
||||
RSYNC: rsync -az --chmod=D755,F644 -e "ssh -p {{.NAS_PORT}} -o BatchMode=yes"
|
||||
# What the mirror leaves on the NAS.
|
||||
BULK: --exclude items/ --exclude renditions/
|
||||
# What promotion leaves alone: each instance keeps its own archive.yaml, and Synology's index directories stay where they are.
|
||||
PROMOTE: --exclude archive.yaml --exclude @eaDir
|
||||
|
||||
tasks:
|
||||
fmt:
|
||||
|
|
@ -46,14 +52,23 @@ tasks:
|
|||
cmds:
|
||||
- go test -run '^$' -bench Search -benchmem ./internal/archive/
|
||||
|
||||
uat:seed:
|
||||
desc: 'Seed UAT on the NAS from production, once: the whole tree copied, people and items, and its archive.yaml made UAT''s; refused when the UAT tree exists (story 026)'
|
||||
cmds:
|
||||
- |
|
||||
ssh -p {{.NAS_PORT}} -o BatchMode=yes {{.NAS}} 'set -e
|
||||
if [ -e {{.ARCHIVE}} ]; then echo "{{.ARCHIVE}} exists: UAT is seeded once, from production" >&2; exit 1; fi
|
||||
cp -a {{.PRODUCTION}} {{.ARCHIVE}}
|
||||
printf "name: bytestone-uat\npurpose: uat\nslot: 1\n" > {{.ARCHIVE}}/archive.yaml
|
||||
cat {{.ARCHIVE}}/archive.yaml'
|
||||
archive:pull:
|
||||
desc: Mirror the archive from the NAS (archive.yaml, records and files; not items or renditions), the NAS winning
|
||||
desc: Mirror UAT from the NAS (archive.yaml, records and files; not items or renditions), the NAS winning
|
||||
cmds:
|
||||
- mkdir -p {{.MIRROR}}/people
|
||||
- '{{.RSYNC}} {{.NAS}}:{{.ARCHIVE}}/archive.yaml {{.MIRROR}}/archive.yaml'
|
||||
- '{{.RSYNC}} --delete {{.BULK}} {{.NAS}}:{{.ARCHIVE}}/people/ {{.MIRROR}}/people/'
|
||||
archive:push:
|
||||
desc: Send the mirror's archive.yaml and people tree to the NAS (adds and updates; never deletes there)
|
||||
desc: Send the mirror's archive.yaml and people tree to UAT on the NAS (adds and updates; never deletes there)
|
||||
cmds:
|
||||
- 'ssh -p {{.NAS_PORT}} {{.NAS}} mkdir -p {{.ARCHIVE}}/people'
|
||||
- '{{.RSYNC}} {{.MIRROR}}/archive.yaml {{.NAS}}:{{.ARCHIVE}}/archive.yaml'
|
||||
|
|
@ -93,14 +108,19 @@ tasks:
|
|||
- GOOS=js GOARCH=wasm go build -ldflags=-s -trimpath -o site/kinship.wasm ./cmd/kinship
|
||||
- cp "$(go env GOROOT)/lib/wasm/wasm_exec.js" site/
|
||||
publish:
|
||||
desc: 'Publish: mirror → NAS and NAS → mirror (archive:push, archive:pull), then build and send docs/ and site/ to their hosts (tp pages deploy); the stage-1 form of #41'
|
||||
desc: 'Publish: the mirror → UAT on the NAS and back (archive:push, archive:pull), then build and send docs/ and site/ to their hosts (tp pages deploy): i.bytestone.uk is UAT; the stage-1 form of #41'
|
||||
cmds:
|
||||
- task: archive:push
|
||||
- task: archive:pull
|
||||
- task-plus pages deploy
|
||||
promote:
|
||||
desc: 'Promote: i.bytestone.org, the production site of lives, becomes what i.bytestone.uk serves now; a copy made on the host, no build (#103)'
|
||||
promote:diff:
|
||||
desc: What promote would change in production on the NAS (deletions included); nothing is written
|
||||
cmds:
|
||||
- ssh -p {{.NAS_PORT}} -o BatchMode=yes {{.NAS}} 'rsync -n -i -a --delete {{.PROMOTE}} {{.ARCHIVE}}/ {{.PRODUCTION}}/'
|
||||
promote:
|
||||
desc: 'Promote: production becomes what UAT is now. The UAT tree is copied over production''s on the NAS, people and items, deletions included (run promote:diff first), and i.bytestone.org becomes what i.bytestone.uk serves, a copy made on the host, no build (#103, story 026). The only task that writes production'
|
||||
cmds:
|
||||
- ssh -p {{.NAS_PORT}} -o BatchMode=yes {{.NAS}} 'rsync -a --delete {{.PROMOTE}} {{.ARCHIVE}}/ {{.PRODUCTION}}/'
|
||||
- ssh -o BatchMode=yes deploy@docs.bytestone.uk 'rsync -a --delete /srv/sites/i/ /srv/sites/i-org/'
|
||||
serve:
|
||||
desc: Run the archive service's API over the archive (BYTESTONE_ARCHIVE, BYTESTONE_ADDR from .env)
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
// Package archive describes an archive instance: one tree laid out as
|
||||
// docs/archive.md says, which names itself in archive.yaml at its root.
|
||||
// Several instances exist, production on the NAS, scratch copies and a
|
||||
// scale instance for large imports, and the file is what tells them
|
||||
// apart, so a bulk writer refuses production by asking the tree rather
|
||||
// than trusting the operator's flag.
|
||||
// Several instances exist, production on the NAS, UAT beside it (story
|
||||
// 026), scratch copies and a scale instance for large imports, and the
|
||||
// file is what tells them apart, so a bulk writer refuses production by
|
||||
// asking the tree rather than trusting the operator's flag.
|
||||
package archive
|
||||
|
||||
import (
|
||||
|
|
@ -22,7 +22,8 @@ const File = "archive.yaml"
|
|||
type Purpose string
|
||||
|
||||
const (
|
||||
Production Purpose = "production" // the one instance on the NAS
|
||||
Production Purpose = "production" // the archive of record, on the NAS
|
||||
UAT Purpose = "uat" // the staging copy every depositor writes; production is promoted from it
|
||||
Scratch Purpose = "scratch" // a copy to try things on
|
||||
Scale Purpose = "scale" // large imports and performance tests
|
||||
)
|
||||
|
|
@ -32,15 +33,18 @@ type Instance struct {
|
|||
Name string `yaml:"name"`
|
||||
Purpose Purpose `yaml:"purpose"`
|
||||
// Slot is the instance's place among the Slots that may serve at
|
||||
// once, and so its address: production is slot 0, said or not; a
|
||||
// scratch or scale instance says its own, 1 to Slots-1.
|
||||
// once, and so its address: production is slot 0 and UAT is UATSlot,
|
||||
// said or not; a scratch or scale instance says its own, UATSlot+1 to
|
||||
// Slots-1.
|
||||
Slot int `yaml:"slot"`
|
||||
}
|
||||
|
||||
// BasePort is slot 0's port, production's, which the archive's clients
|
||||
// default to; slot n serves on BasePort+n. Slots is how many there are.
|
||||
// default to; slot n serves on BasePort+n. UATSlot is UAT's, so its
|
||||
// clients find it without reading a tree. Slots is how many there are.
|
||||
const (
|
||||
BasePort = 8321
|
||||
UATSlot = 1
|
||||
Slots = 10
|
||||
)
|
||||
|
||||
|
|
@ -62,27 +66,47 @@ func Load(root string) (Instance, error) {
|
|||
}
|
||||
return Instance{}, err
|
||||
}
|
||||
var in Instance
|
||||
if err := yaml.Unmarshal(data, &in); err != nil {
|
||||
var file struct {
|
||||
Name string `yaml:"name"`
|
||||
Purpose Purpose `yaml:"purpose"`
|
||||
Slot *int `yaml:"slot"` // nil when the file does not say
|
||||
}
|
||||
if err := yaml.Unmarshal(data, &file); err != nil {
|
||||
return Instance{}, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
in := Instance{Name: file.Name, Purpose: file.Purpose}
|
||||
if in.Name == "" {
|
||||
return Instance{}, fmt.Errorf("%s: no name", path)
|
||||
}
|
||||
named := fmt.Sprintf("%s, %s, %s or %s", Production, UAT, Scratch, Scale)
|
||||
switch in.Purpose {
|
||||
case Production, Scratch, Scale:
|
||||
case Production, UAT, Scratch, Scale:
|
||||
case "":
|
||||
return Instance{}, fmt.Errorf("%s: no purpose (%s, %s or %s)", path, Production, Scratch, Scale)
|
||||
return Instance{}, fmt.Errorf("%s: no purpose (%s)", path, named)
|
||||
default:
|
||||
return Instance{}, fmt.Errorf("%s: purpose %q is not %s, %s or %s", path, in.Purpose, Production, Scratch, Scale)
|
||||
return Instance{}, fmt.Errorf("%s: purpose %q is not %s", path, in.Purpose, named)
|
||||
}
|
||||
// Production and UAT have fixed slots, said or not; the others name
|
||||
// their own among the rest.
|
||||
fixed := map[Purpose]int{Production: 0, UAT: UATSlot}
|
||||
if slot, ok := fixed[in.Purpose]; ok {
|
||||
if file.Slot != nil && *file.Slot != slot {
|
||||
return Instance{}, fmt.Errorf("%s: %s is slot %d, not %d", path, in.Purpose, slot, *file.Slot)
|
||||
}
|
||||
in.Slot = slot
|
||||
return in, nil
|
||||
}
|
||||
if file.Slot == nil {
|
||||
return Instance{}, fmt.Errorf("%s: a %s instance needs a slot of its own, %d to %d (slot 0 is production's, %d is UAT's)", path, in.Purpose, UATSlot+1, Slots-1, UATSlot)
|
||||
}
|
||||
in.Slot = *file.Slot
|
||||
switch {
|
||||
case in.Purpose == Production && in.Slot != 0:
|
||||
return Instance{}, fmt.Errorf("%s: production is slot 0, not %d", path, in.Slot)
|
||||
case in.Purpose != Production && in.Slot == 0:
|
||||
return Instance{}, fmt.Errorf("%s: a %s instance needs a slot of its own, 1 to %d (slot 0 is production's)", path, in.Purpose, Slots-1)
|
||||
case in.Slot == 0:
|
||||
return Instance{}, fmt.Errorf("%s: slot 0 is production's; a %s instance takes %d to %d", path, in.Purpose, UATSlot+1, Slots-1)
|
||||
case in.Slot == UATSlot:
|
||||
return Instance{}, fmt.Errorf("%s: slot %d is UAT's; a %s instance takes %d to %d", path, UATSlot, in.Purpose, UATSlot+1, Slots-1)
|
||||
case in.Slot < 0 || in.Slot >= Slots:
|
||||
return Instance{}, fmt.Errorf("%s: slot %d is not 0 to %d", path, in.Slot, Slots-1)
|
||||
return Instance{}, fmt.Errorf("%s: slot %d is not %d to %d", path, in.Slot, UATSlot+1, Slots-1)
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
|
@ -92,6 +116,7 @@ func (in Instance) String() string { return fmt.Sprintf("%s (%s)", in.Name, in.P
|
|||
|
||||
// Bulk is what a bulk writer (a synthetic generator, a mass import) asks
|
||||
// before it starts: ErrProduction, naming the instance, for production.
|
||||
// UAT takes bulk writes: that is what it is for (story 026).
|
||||
func (in Instance) Bulk() error {
|
||||
if in.Purpose == Production {
|
||||
return fmt.Errorf("%s: %w", in, ErrProduction)
|
||||
|
|
|
|||
|
|
@ -13,8 +13,10 @@ func TestLoad(t *testing.T) {
|
|||
want Instance
|
||||
}{
|
||||
{"production", Instance{Name: "bytestone", Purpose: Production}}, // slot 0, said or not
|
||||
{"scratch", Instance{Name: "nas-copy", Purpose: Scratch, Slot: 1}},
|
||||
{"scratch", Instance{Name: "nas-copy", Purpose: Scratch, Slot: 2}},
|
||||
{"scale", Instance{Name: "ladder", Purpose: Scale, Slot: 2}},
|
||||
{"uat", Instance{Name: "bytestone-uat", Purpose: UAT, Slot: 1}}, // story 026: UAT is slot 1
|
||||
{"uatnoslot", Instance{Name: "bytestone-uat", Purpose: UAT, Slot: 1}}, // said or not
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, err := Load(filepath.Join("testdata", c.dir))
|
||||
|
|
@ -39,6 +41,9 @@ func TestLoadRefuses(t *testing.T) {
|
|||
{"noslot", "slot"}, // #71: a scratch or scale instance needs a slot of its own
|
||||
{"badslot", "slot"}, // 12 is past the ten
|
||||
{"prodslot", "slot 0"}, // production is slot 0
|
||||
{"uatslot0", "slot 1"}, // story 026: UAT is slot 1
|
||||
{"uatslot2", "slot 1"},
|
||||
{"scaleslot1", "UAT"}, // slot 1 is UAT's; scratch and scale take 2 to 9
|
||||
}
|
||||
for _, c := range cases {
|
||||
_, err := Load(filepath.Join("testdata", c.dir))
|
||||
|
|
@ -61,7 +66,7 @@ func TestBulk(t *testing.T) {
|
|||
if err == nil || !strings.Contains(err.Error(), "bytestone") {
|
||||
t.Errorf("production: %v does not name the instance", err)
|
||||
}
|
||||
for _, p := range []Purpose{Scratch, Scale} {
|
||||
for _, p := range []Purpose{Scratch, Scale, UAT} {
|
||||
if err := (Instance{Name: "x", Purpose: p}).Bulk(); err != nil {
|
||||
t.Errorf("%s: %v", p, err)
|
||||
}
|
||||
|
|
@ -77,4 +82,7 @@ func TestAddrFollowsTheSlot(t *testing.T) {
|
|||
if got := (Instance{Name: "h3", Purpose: Scale, Slot: 2}).Addr(); got != "localhost:8323" {
|
||||
t.Errorf("slot 2: %s", got)
|
||||
}
|
||||
if got := (Instance{Name: "bytestone-uat", Purpose: UAT, Slot: 1}).Addr(); got != "localhost:8322" {
|
||||
t.Errorf("UAT: %s", got)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
3
archive/testdata/scaleslot1/archive.yaml
vendored
Normal file
3
archive/testdata/scaleslot1/archive.yaml
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
name: x
|
||||
purpose: scale
|
||||
slot: 1
|
||||
2
archive/testdata/scratch/archive.yaml
vendored
2
archive/testdata/scratch/archive.yaml
vendored
|
|
@ -1,3 +1,3 @@
|
|||
name: nas-copy
|
||||
purpose: scratch
|
||||
slot: 1
|
||||
slot: 2
|
||||
|
|
|
|||
3
archive/testdata/uat/archive.yaml
vendored
Normal file
3
archive/testdata/uat/archive.yaml
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
name: bytestone-uat
|
||||
purpose: uat
|
||||
slot: 1
|
||||
2
archive/testdata/uatnoslot/archive.yaml
vendored
Normal file
2
archive/testdata/uatnoslot/archive.yaml
vendored
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
name: bytestone-uat
|
||||
purpose: uat
|
||||
3
archive/testdata/uatslot0/archive.yaml
vendored
Normal file
3
archive/testdata/uatslot0/archive.yaml
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
name: x
|
||||
purpose: uat
|
||||
slot: 0
|
||||
3
archive/testdata/uatslot2/archive.yaml
vendored
Normal file
3
archive/testdata/uatslot2/archive.yaml
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
name: x
|
||||
purpose: uat
|
||||
slot: 2
|
||||
1
changes/139.added.md
Normal file
1
changes/139.added.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
Story 026, a UAT archive instance (#139): a fourth purpose, `uat`, at slot 1 as production is slot 0, said or not, that bulk writers may write; scratch and scale instances take 2 to 9. UAT is a full copy of production on the NAS beside it, people and items, seeded once by `task uat:seed`, mirrored to the laptop at `~/.local/share/bytestone/uat`, and what every depositor writes and i.bytestone.uk is built from: `archive:pull`, `archive:push`, `build`, `serve` and `publish` work on it. Production becomes what `task promote` makes it, exactly UAT, the tree with deletions after `task promote:diff` and the site as before; the only task that writes production
|
||||
|
|
@ -46,7 +46,7 @@ func writeBag(t *testing.T, root, id string, payload map[string]string) string {
|
|||
// not sound; it is the monthly fixity run.
|
||||
func TestFixityAuditsEveryHeldItem(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
write(t, filepath.Join(root, "archive.yaml"), []byte("name: scratch\npurpose: scratch\nslot: 1\n"))
|
||||
write(t, filepath.Join(root, "archive.yaml"), []byte("name: scratch\npurpose: scratch\nslot: 2\n"))
|
||||
writeBag(t, root, "0", map[string]string{"front.tif": "II*\x00one", "photo.yaml": "batch: 1\n"})
|
||||
rotten := writeBag(t, root, "1", map[string]string{"front.tif": "II*\x00two"})
|
||||
if err := os.MkdirAll(filepath.Join(root, "people/0/00/items/2.partial"), 0o755); err != nil {
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ import (
|
|||
// naming the squatter; something that is not life-weaver, refuse as
|
||||
// taken.
|
||||
func TestServeRefusesAnAddressAlreadyServing(t *testing.T) {
|
||||
h3 := archive.Instance{Name: "h3", Purpose: archive.Scale, Slot: 1}
|
||||
h3 := archive.Instance{Name: "h3", Purpose: archive.Scale, Slot: 2}
|
||||
geni := archive.Instance{Name: "geni", Purpose: archive.Scale, Slot: 2}
|
||||
for _, c := range []struct {
|
||||
there archive.Instance
|
||||
|
|
|
|||
|
|
@ -10,15 +10,17 @@ builder. Proposed 2026-10-24; stage 1 is life-weaver#22.
|
|||
| Term | Meaning |
|
||||
|---|---|
|
||||
| Archive | The primary store: one directory tree laid out as below |
|
||||
| Archive instance | One such tree, named by `archive.yaml` at its root (`name`, `purpose`: `production`, `scratch` or `scale`, `slot`: 0 for production, 1 to 9 for the others, which puts the service on `localhost:8321+slot`); the `archive` package reads it, `build` and `serve` report it |
|
||||
| Production | The one instance on the NAS. A bulk writer (a synthetic generator, a mass import) asks the instance first and refuses production |
|
||||
| Archive instance | One such tree, named by `archive.yaml` at its root (`name`, `purpose`: `production`, `uat`, `scratch` or `scale`, `slot`: 0 for production, 1 for UAT, 2 to 9 for the others, which puts the service on `localhost:8321+slot`); the `archive` package reads it, `build` and `serve` report it |
|
||||
| Production | The archive of record, on the NAS. A bulk writer (a synthetic generator, a mass import) asks the instance first and refuses production; nothing writes production but a promotion |
|
||||
| UAT | The staging instance beside production on the NAS (story 026): a full copy, people and items, seeded once from production, that every depositor writes and i.bytestone.uk is built from. Bulk writers may write it |
|
||||
| Promotion | `task promote`: production becomes exactly what UAT is now, the tree copied over with deletions and the site copied over i.bytestone.org; `task promote:diff` lists the changes first |
|
||||
| Register | The person records, `people/<dir>/person.yaml` (README) |
|
||||
| Person directory | Everything of one person under `people/<dir>/`, found from the id; the unit of work, sync, publish and audit |
|
||||
| Files | What a record names, under the person's `files/`; the lecture's images and outputs are Victoria Drummond's |
|
||||
| Item | One archived thing with its own id, a BagIt bag in its home person's `items/<item-id>/` (life-weaver#19; the format statement). Its kind is read off the bag: a *photo* (`photo.yaml`) or a *page* (`page.yaml`: a file received from elsewhere, kept as sent; the prints on it are photos cut from it, #72) |
|
||||
| Rendition | A derived access copy of an item, under the person's `renditions/<item>/<recipe>/`: `display.jpg`, `thumb.jpg`, `tiny.jpg`, `recipe.yaml`, the back's under `back/`. Made by the depositor at a recipe version and carried with the bag (#39); rebuildable, not preserved |
|
||||
| Derived | Outputs of a per-person activity (maps, thumbnails) under `derived/`, each with its recipe version; never regenerated by a code release |
|
||||
| Mirror | The laptop's copy of production's `archive.yaml` and people tree without `items/` and `renditions/`, the build's input until the build runs on the NAS; it carries production's file, so it is production to a bulk writer |
|
||||
| Mirror | The laptop's copy of UAT's `archive.yaml` and people tree without `items/` and `renditions/`, the build's input until the build runs on the NAS; it carries UAT's file, so it takes bulk writes as UAT does |
|
||||
| Site | The read copy the public sees, built from the archive and rsynced to its host |
|
||||
|
||||
## Endpoint and layout
|
||||
|
|
@ -26,13 +28,13 @@ builder. Proposed 2026-10-24; stage 1 is life-weaver#22.
|
|||
| | |
|
||||
|---|---|
|
||||
| Host | `humphrey@10.0.0.1`, SSH port 5522 (the Synology NAS, HDTLServer) |
|
||||
| Root | `/volume1/H3Archive/bytestone` |
|
||||
| Root | `/volume1/H3Archive/bytestone` (production); `/volume1/H3Archive/bytestone-uat` (UAT) |
|
||||
| Transport | rsync over SSH. DSM must have the rsync service enabled for the account, or its rsync answers *Permission denied* after a successful login |
|
||||
| Capacity | volume1 has 5.5 TB free; the layout is meant to hold 1–2 TB before it is restructured |
|
||||
| Capacity | volume1 has 5.5 TB free; the layout is meant to hold 1–2 TB before it is restructured, twice over now that UAT is a full copy |
|
||||
|
||||
```
|
||||
bytestone/
|
||||
archive.yaml name: bytestone, purpose: production (#36)
|
||||
bytestone/ production; bytestone-uat/ is laid out the same
|
||||
archive.yaml name: bytestone, purpose: production (#36); name: bytestone-uat, purpose: uat, slot: 1 in UAT (story 026)
|
||||
people/<dir>/
|
||||
person.yaml the record; its file paths are relative to this directory
|
||||
files/ what the record names
|
||||
|
|
@ -49,9 +51,9 @@ holds more than 1024 entries, a directory is either a shard or a person,
|
|||
and nothing caps the id's length. `pid.Dir` and `life-weaver dir` compute
|
||||
it.
|
||||
|
||||
The Taskfile's `NAS`, `ARCHIVE` and `MIRROR` vars are the one place the
|
||||
endpoint is written. Moving the archive to a dedicated share later is a
|
||||
`mv` on the NAS and a change to `ARCHIVE`.
|
||||
The Taskfile's `NAS`, `PRODUCTION`, `ARCHIVE` (UAT) and `MIRROR` vars are
|
||||
the one place the endpoint is written. Moving the archive to a dedicated
|
||||
share later is a `mv` on the NAS and a change to those.
|
||||
|
||||
## Flow
|
||||
|
||||
|
|
@ -62,7 +64,7 @@ depositors: Depositors {
|
|||
lecture: victoria-drummond-lecture
|
||||
laptop: records edited on the laptop
|
||||
}
|
||||
archive: Archive, on the NAS {
|
||||
uat: UAT, on the NAS {
|
||||
person: "people/<dir>/" {
|
||||
record: person.yaml
|
||||
files
|
||||
|
|
@ -71,13 +73,19 @@ archive: Archive, on the NAS {
|
|||
derived
|
||||
}
|
||||
}
|
||||
production: Production, on the NAS
|
||||
site: i.bytestone.uk
|
||||
depositors -> archive: "stage 1: rsync over SSH\nstage 2: the archive service's API"
|
||||
archive -> site: "stage 1: pull to the mirror, build, tp pages deploy\nstage 3: build on the NAS, rsync out"
|
||||
prod_site: i.bytestone.org
|
||||
depositors -> uat: "stage 1: rsync over SSH\nstage 2: the archive service's API"
|
||||
uat -> site: "stage 1: pull to the mirror, build, tp pages deploy\nstage 3: build on the NAS, rsync out"
|
||||
uat -> production: "task promote: the tree, with deletions"
|
||||
site -> prod_site: "task promote: the site as served"
|
||||
```
|
||||
|
||||
Writes go from depositors to the archive; publishing goes from the archive
|
||||
to the site. Nothing goes from the site back.
|
||||
Writes go from depositors to UAT; publishing goes from UAT to
|
||||
i.bytestone.uk; a promotion makes production and i.bytestone.org what
|
||||
UAT and its site are. Nothing goes from a site back, and nothing writes
|
||||
production but the promotion.
|
||||
|
||||
## Stages
|
||||
|
||||
|
|
@ -227,4 +235,8 @@ not to look up; the receipt stays the cheap call. bytestone-scan's
|
|||
| Where the build runs | Stage 1: the laptop, from the mirror. Stage 3: the NAS | Always the laptop | At 1–2 TB the published files cannot pass through a laptop. The site build copies every published file into `site/`; on the NAS (Btrfs) that copy can be a reflink |
|
||||
| Who makes renditions | The depositor, at its recipe version; the service refuses a deposit without them and places them beside the bag | The archive develops them from the master and `photo.yaml`, with the recipe in a package both import | The scanner has the recipe and the operator's eye on the result already; the archive stays an image-agnostic store that holds, checks and serves. Cost: non-scanner photographs (`files/`) need their own thumbnail route (#3), and re-development at a new recipe (bytestone-scan#18) is a depositor's job over the archive |
|
||||
| A tree with no `archive.yaml` | Refused by `build`, `serve` and the loader | Assumed `scratch` | The guard must not be defeatable by forgetting the file; the mirror gets its copy from `archive:pull`, and a scratch or scale tree is made by writing its file first |
|
||||
| What UAT is (story 026) | A full archive instance of its own on the NAS, items included, mirrored to the laptop as production was | A site-only stage built from production's archive, as before | A bulk load has to be an archive before it is a site, and the loader must never write production; the scanner's items will go to UAT too. Cost: a second copy of the data on volume1 |
|
||||
| What promotion does | Production becomes exactly UAT: copied with deletion, each instance keeping its own `archive.yaml`, after `promote:diff` | Adds and updates only, as `archive:push` | A record withdrawn from UAT (a living person, a suspect tie) must not outlive it in production; exactness is what makes "reviewed in UAT" mean anything. The dry run and a human running it are the guard |
|
||||
| Slots after UAT | UAT is slot 1, fixed, as production is 0 | Any of 1 to 9 | Clients find UAT without reading a tree; scratch and scale instances take 2 to 9 |
|
||||
| Seeding UAT | Once, from production, refused thereafter | Copying production to UAT on demand | After the seed nothing reaches production that did not come through UAT; a reverse copy would hide that |
|
||||
| Records' history | Not kept yet | A git repo in `people/`, as the format statement keeps bag sidecars | Needed, but the service of stage 2 decides who commits; not before |
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# 026 — A UAT archive instance (#139)
|
||||
|
||||
**Status:** proposed
|
||||
**Status:** done locally
|
||||
**Roadmap:** none — the archive's stage 2 (docs/archive.md) gains a staging instance; the bytestone-loader loads and bytestone-loader #2 depend on it
|
||||
|
||||
## Why
|
||||
|
|
|
|||
2
internal/site/testdata/site/archive.yaml
vendored
2
internal/site/testdata/site/archive.yaml
vendored
|
|
@ -1,3 +1,3 @@
|
|||
name: fixture
|
||||
purpose: scratch
|
||||
slot: 1
|
||||
slot: 2
|
||||
|
|
|
|||
Loading…
Reference in a new issue