feat: add deployment foundation and cross-device handoff
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# Operation State Implementation Plan
|
||||
|
||||
> **For agentic workers:** Use superpowers:executing-plans with test-driven-development. No production access or automatic commits.
|
||||
|
||||
**Goal:** Persist operation identity and status under an exclusive OS lock, refusing duplicate execution and ambiguous state.
|
||||
|
||||
**Architecture:** An internal state package acquires a nonblocking per-directory host lock and returns a session. Every mutation saves a complete bounded snapshot using write/sync/rename; the session becomes unusable on persistence errors. The directory must be a pre-existing trusted local directory, not a network share.
|
||||
|
||||
**Tech Stack:** Go standard library, Linux flock / Windows LockFileEx, os.Root.
|
||||
|
||||
**Spec:** docs/2026-09-25-complete-optimization-proposal.md sections 7, 8, 12, 14.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Current directory and codex/new-deployment-architecture branch; preserve prior edits.
|
||||
- No CLI write endpoint or Docker/application operation is enabled by this package.
|
||||
- No automatic deletion of lock files, stale-task replay or corrupted-state reset.
|
||||
- Lock-file inode must remain stable; close releases the OS lock.
|
||||
- Metadata integrity is not authorization, and terminal success must be verified by the future executor.
|
||||
- Linux is the production target; Windows provides development tests. Power-loss durability on Windows is not claimed.
|
||||
|
||||
## Task 1: Exclusive host sessions
|
||||
|
||||
Files: internal/state/store.go, lock_linux.go, lock_windows.go, lock_unsupported.go, store_test.go.
|
||||
|
||||
Interface: `Acquire(directory, hostID string) (*Session, error)`; `Session.Close() error`.
|
||||
|
||||
- [x] Write tests: second session returns ErrBusy, closed session rejects use, independent directories do not block each other, invalid IDs fail, killed child releases OS lock.
|
||||
- [x] Run `go test ./internal/state`; observe missing implementation failure.
|
||||
- [x] Implement nonblocking OS lock with private regular file inside os.Root; refuse unsupported systems and symlink state files.
|
||||
- [x] Run tests using real temp directories and subprocesses, not mocked locks.
|
||||
|
||||
```go
|
||||
second, err := Acquire(dir, "host-one")
|
||||
if second != nil || !errors.Is(err, ErrBusy) { t.Fatal("host lock bypassed") }
|
||||
```
|
||||
|
||||
## Task 2: Persistent idempotency and state transitions
|
||||
|
||||
Files: internal/state/operation.go, snapshot.go, operation_test.go.
|
||||
|
||||
Interfaces: `Begin(id, planHash string) (Operation, bool, error)`; `Get(id string) (Operation, error)`; `Advance(id string, revision uint64, next Status) (Operation, error)`.
|
||||
|
||||
- [x] Tests: duplicate ID/hash returns original with created=false; same ID/different hash conflicts; reopened state persists; unrelated new operation blocked by unresolved prior operation; stale revision and running-to-running rejected; terminal results immutable.
|
||||
- [x] Tests: corrupt/truncated/oversized/host-mismatched snapshots rejected; write failure poisons session and never returns success; surviving running state is not silently reset.
|
||||
- [x] Implement canonical checksummed snapshot, bounded read, unique temporary file, file sync, atomic rename and Linux directory sync. Any persistence failure requires closing/reopening and checking the actual state.
|
||||
- [x] Run full tests and real killed-child recovery test. Abrupt process death is not a power-loss durability test.
|
||||
|
||||
```go
|
||||
again, created, err := session.Begin("op-one", hash)
|
||||
if err != nil || created || again.Revision != 1 { t.Fatal("duplicate submission changed state") }
|
||||
```
|
||||
|
||||
## Task 3: Verification and documentation
|
||||
|
||||
- [x] Run gofmt, `go test ./... -count=1`, `go vet ./...`, native CLI smoke tests and Linux cross-build including state tests.
|
||||
- [x] Independent read-only code review; repair actionable findings with regression tests.
|
||||
- [x] Update docs/implementation-status.md and internal/state/README.md with lifecycle, trusted-root requirements, residual risks, and remaining systemd/executor work.
|
||||
|
||||
Additional evidence: Ubuntu WSL full suite and `go test -race ./internal/state` passed.
|
||||
Windows symlink tests skipped for unavailable privilege; both passed on Linux.
|
||||
Regression tests cover missing initialized snapshots and 113,357-record capacity boundary.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Package Verification and Inspection Implementation Plan
|
||||
|
||||
> **For agentic workers:** Execute with test-driven-development; package worker and local inspection work have disjoint write scopes. No commits, production connections, or deployment writes.
|
||||
|
||||
**Goal:** Add read-only package verification and a factual local prerequisite report to deployctl.
|
||||
|
||||
**Architecture:** A shared bounded strict JSON decoder supports nested typed arrays. An immutable package verifier returns verified manifest and file bytes, never executes templates. A local probe reports OS/architecture and tool/runtime presence, with unperformed checks explicitly marked unknown.
|
||||
|
||||
**Tech Stack:** Go standard library. Existing local Windows and Ubuntu WSL test runtimes.
|
||||
|
||||
**Spec:** docs/2026-09-25-complete-optimization-proposal.md sections 5, 7, 12, 14.
|
||||
|
||||
## Boundaries and integration
|
||||
|
||||
- Parent owns internal/wire, internal/inspect, internal/cli, docs and CLI tests.
|
||||
- Package worker owns only internal/appbundle; requirements are in package-verifier-brief.md beside this plan.
|
||||
- Package code consumes `wire.Decode(reader io.Reader, target any, limit int64) error`.
|
||||
- CLI consumes `appbundle.Verify(directory, expectedDigest string) (Verified, error)`; Verified has Manifest, Files map[string][]byte, Digest string. CLI never prints file bodies.
|
||||
- Local inspect is not full host acceptance; Docker daemon, Compose, firewall, DNS, permissions and port availability remain explicitly unverified.
|
||||
- Expected digest must come from a trusted channel. Matching it does not establish publisher identity on its own.
|
||||
|
||||
## Task 1: Shared strict wire decoder
|
||||
|
||||
- [x] Move strict JSON decoder from internal/cli to internal/wire, keeping all CLI tests passing.
|
||||
- [x] Add failing tests for nested array unknown/missing/case-alias fields and null elements, duplicate keys and input limit.
|
||||
- [x] Extend recursive validation through typed slices and reject invalid UTF-8. Preserve canonical UTC timestamp policy.
|
||||
- [x] Test the API `wire.Decode(strings.NewReader(input), &value, 1024)` using real JSON.
|
||||
|
||||
## Task 2: Application package verifier
|
||||
|
||||
- [x] Worker implements bounded manifest validation, pinned multi-component images, exact file inventory, SHA-256 content checks and path/symlink rejection using tests first.
|
||||
- [x] Parent reviews written code and runs all tests, including Linux symlink cases.
|
||||
- [x] Independent review checks all file access and integrity boundaries.
|
||||
|
||||
## Task 3: Read-only inspection and CLI
|
||||
|
||||
- [x] internal/inspect provides `Collect() Report` and a testable filesystem probe. Report includes OS, architecture, Linux support, systemd runtime presence and Docker client presence, with explicit unchecked prerequisites.
|
||||
- [x] Missing or inaccessible files produce missing/unknown observations, never false success. No shell commands, network, credentials or environment changes.
|
||||
- [x] CLI `inspect` emits observations with deploymentReady=false.
|
||||
- [x] CLI `verify-package` accepts `{directory, expectedDigest}`, emits manifest, digest, file count and executable=false; rejects invalid input with redacted diagnostics.
|
||||
- [x] Add actual CLI tests for new commands and unsupported writes.
|
||||
|
||||
## Verification
|
||||
|
||||
- [x] Windows go test/vet, CLI smoke; Linux full tests, race check and build.
|
||||
- [x] Record separate implemented, unverified and not-yet-supported capabilities.
|
||||
|
||||
## Execution notes
|
||||
|
||||
Task 1 + Task 2 share only the Decode interface; Task 2 must not modify it. Task 3 consumes their exports without exposing package bytes. Directory paths are accepted only for read-only local verification. Files remain subject to trusted staging ownership; future execution must use the verified snapshot rather than reopening mutable package files.
|
||||
|
||||
Independent review found an alternate Docker path omitted when the primary path
|
||||
was invalid or inaccessible. Three failing regression cases reproduced it before
|
||||
the fix. Fallback now prefers presence, then uncertainty, invalidity and absence.
|
||||
Focused re-review found no remaining issues. Final Windows and Linux checks passed;
|
||||
see implementation-status.md for the initial non-reproduced Linux compiler error.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Runner Foundation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox syntax for tracking.
|
||||
|
||||
**Goal:** Deliver a runnable, read-only deployctl foundation that strictly validates deployment requests and produces expiring, state-bound plans without executing legacy scripts.
|
||||
|
||||
**Architecture:** Go standard-library CLI consumes bounded JSON over stdin. Domain validation and planning are separate from CLI transport. No deployment writes are enabled before the executor and application recovery contracts pass their own acceptance tests.
|
||||
|
||||
**Tech Stack:** Go, standard testing package, JSON protocol v1.
|
||||
|
||||
**Spec:** docs/2026-09-25-complete-optimization-proposal.md
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Traefik + local React/Node panel + Go runner are approved choices.
|
||||
- Do not connect to production or modify existing application data.
|
||||
- Preserve current uncommitted documents and claude-dev-stack deletion.
|
||||
- No implicit system upgrade, arbitrary shell field, mutable image tag or fake successful apply.
|
||||
- This plan is the first independently testable work package, not the full product.
|
||||
- No commit or push without user request.
|
||||
|
||||
## Delivery sequence after this package
|
||||
|
||||
Remote state/locking/idempotency → trusted packages and Compose runtime → Traefik environment → Gitea backup/upgrade/restore → Joplin verification → local panel → other applications → isolated Linux acceptance and replacement of legacy source. These require separate implementation plans; they are not represented as completed by this package.
|
||||
|
||||
## Task 1: Validated deployment intent
|
||||
|
||||
Files: go.mod; internal/planner/intent.go; internal/planner/intent_test.go.
|
||||
|
||||
Interface: Intent.Validate() error. Intent includes protocolVersion, hostId, instanceId, appId, packageDigest, imageDigest, domain, observedStateDigest.
|
||||
|
||||
- [x] Write table-driven tests: valid request succeeds; traversal IDs, missing fields, unsupported protocol, floating images, invalid domains and invalid hashes fail.
|
||||
- [x] Run `go test ./internal/planner` and observe the missing implementation failure.
|
||||
- [x] Implement strict lower-case IDs, sha256 digest validation and ASCII DNS names; do not normalize invalid input silently.
|
||||
- [x] Run tests and `go vet ./...`.
|
||||
|
||||
Example boundary assertion:
|
||||
|
||||
```go
|
||||
request.InstanceID = "../gitea"
|
||||
if request.Validate() == nil { t.Fatal("accepted path traversal") }
|
||||
```
|
||||
|
||||
## Task 2: Expiring, state-bound plan
|
||||
|
||||
Files: internal/planner/plan.go; internal/planner/plan_test.go.
|
||||
|
||||
Interfaces: Build(Intent, time.Time) (Plan, error); Plan.Verify(Intent, time.Time) error.
|
||||
|
||||
- [x] Test repeatable hashes at fixed time, separate instance identities, expiry boundary, altered intent/hash and observed-state drift rejection.
|
||||
- [x] Observe failing tests before implementation.
|
||||
- [x] Implement SHA-256 over typed JSON with an empty hash field, 15-minute TTL, deterministic project/data paths and verification of all derived fields.
|
||||
- [x] Run `go test ./internal/planner`.
|
||||
|
||||
```go
|
||||
if err := plan.Verify(intent, plan.ExpiresAt); err == nil { t.Fatal("accepted expired plan") }
|
||||
```
|
||||
|
||||
Plan hash is an integrity binding, NOT authentication or approval. Supplied host/state are unverified offline input; no write operation may trust them without future remote inspection.
|
||||
|
||||
## Task 3: Read-only CLI and contract
|
||||
|
||||
Files: cmd/deployctl/main.go; internal/cli/run.go; internal/cli/run_test.go; protocol/README.md; protocol/examples/plan-request.json.
|
||||
|
||||
Interface: cli.Run(args []string, in io.Reader, out, diagnostics io.Writer, now func() time.Time) int.
|
||||
|
||||
- [x] Tests exercise actual JSON input/output: plan and verify success, unknown fields, duplicate fields, trailing objects, oversized input, unsupported commands including apply, and no input echo on errors.
|
||||
- [x] Observe missing CLI behavior failure.
|
||||
- [x] Implement `version`, `plan`, `verify-plan` only; strict size-bounded decoder; generic diagnostics; nonzero exit on refusal. `apply` stays unavailable.
|
||||
- [x] Build and execute the real CLI with sample stdin; verify output parses and `apply` fails.
|
||||
- [x] Document exact wire format, offline limitation, commands and remaining stages.
|
||||
|
||||
## Verification and handoff
|
||||
|
||||
- [x] gofmt; go test ./... -count=1; go vet ./...; build Windows and Linux binaries.
|
||||
- [x] Confirm no external dependencies and no production connection/process execution in CLI.
|
||||
- [x] Record actual verification results and remaining work in docs/implementation-status.md.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Package verifier worker requirements
|
||||
|
||||
Implement ONLY internal/appbundle/*.go and optional internal/appbundle/README.md in G:/Works/server-deploy. No commits, no other files, no subagents, no production access. Use apply_patch, tests first. Parent handles CLI, shared decoder, environment inspection and integration review.
|
||||
|
||||
API: `Verify(directory, expectedDigest string) (Verified, error)`.
|
||||
Verified: `Manifest Manifest`, `Files map[string][]byte`, `Digest string`.
|
||||
Manifest JSON fields (all required, no extras): protocolVersion (1), appId (lowercase ID), version (numeric x.y.z), runtime (compose), entrypoint (relative path to listed file), platforms ([]string of linux/amd64 or linux/arm64, unique, nonempty), components ([]Component), files ([]File).
|
||||
Component fields: name (lowercase ID), image (lowercase repository@sha256:64hex; no tag, port, whitespace or shell syntax in initial subset). Components 1..32, names unique; repository components use lowercase ASCII alnum with separators dot/underscore/hyphen, slash between components, no empty segments.
|
||||
File fields: path (portable relative slash path), digest (sha256:64 lowercase hex). Files 1..128, unique; manifest.json itself excluded from files. No slash root, drive, backslash, colon, dot/dotdot segments, dotfiles, empty segments, trailing spaces/dots, Windows device-name segments (including extensions), or case-colliding names. Allowed path segment alphabet lowercase ASCII a-z0-9 underscore hyphen dot, first character alphanumeric. Max path 240 bytes, segment 100. ID rules consistent with planner: first a-z, remaining a-z0-9-, max48.
|
||||
|
||||
Manifest file name manifest.json; maximum 1MiB. ExpectedDigest pins SHA-256 of the exact raw manifest bytes (manifest pins every file). Validate digest before parsing. Use shared `server-deploy/internal/wire.Decode(io.Reader, target, int64 limit)` being implemented by parent; it rejects duplicate/unknown/missing/case-alias fields, nulls including slice elements, trailing data and oversized input.
|
||||
|
||||
Read only regular files under os.Root, reject symlinks including intermediate directory links, file size max4MiB, total payload <=16MiB. Exact inventory: reject any unlisted files, secrets, symlinks, special files or unnecessary directories; allow only parent directories of declared files plus manifest.json. Bound enumeration to declared inventory rather than unbounded walk; reject extras immediately. No extraction, writes, processes, network or arbitrary command execution. Require absolute directory; staging root is caller-selected trusted directory (ancestors trusted), don't claim hostile concurrent mutation is fully prevented. Return verified bytes for later consumers; do not reopen files to execute anything.
|
||||
|
||||
Entrypoint must be listed. DO NOT claim Compose semantics safe/validated: contents are authenticated as bytes only. Signature trust store and executable policy come later; expected hash is a caller trust anchor, not publisher authentication. Do not hardcode live app image hashes or fetch versions.
|
||||
|
||||
Tests with real temp directories: valid two-component package, wrong manifest digest, changed/missing/extra file, nested valid file, symlink/intermediate symlink (skip only missing Windows privilege, Linux tested by parent), traversal, duplicate components/file paths, unpinned image, unsupported protocol/runtime/platform, missing/unknown/null component fields, oversized payload, entrypoint missing. Fixture hashes may be calculated from known test bytes, but test expected acceptance/rejection independently.
|
||||
|
||||
Toolchain: C:/Users/Joywayer/AppData/Local/Temp/server-deploy-go-d67bef26e1ee49718e2b62311429f729/go/bin/go.exe. Set GOCACHE to a dedicated temp path if needed. Parent creates wire shortly; don't change it to unblock. Report test results and changed paths, plus any blockers/concerns, without declaring full deployment readiness.
|
||||
Reference in New Issue
Block a user