mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-10 23:15:40 +00:00
feat(attachments): import workspace bundle with rehydrate + UUID remap (TASK-885) (#306)
* feat(attachments): import workspace bundle with attachment rehydrate + UUID remap (TASK-885) POST /workspaces/import now accepts a tar.gz bundle (Content-Type: application/gzip) and rebuilds the workspace + attachments + items in one round trip. JSON imports still work — content-type dispatch in handleImportWorkspace routes the request. Three-phase flow: 1. Walk the tar, capture pad-export.json + manifest.json + every attachment blob into memory. 2. Run the existing ImportWorkspace path to create the workspace + collections + items + comments + links + versions. New IDs are generated; item.slug is preserved (the existing remap path doesn't re-slugify). 3. For each manifest entry, rehydrate the blob through the storage backend (re-validate MIME + re-hash defensively, don't trust the manifest), insert a fresh attachments row. Build an oldID→newID map keyed on attachment uuid. 4. Walk every imported item's content + fields, replace "pad-attachment:OLD" with "pad-attachment:NEW" in one transactional pass. Refresh FTS afterward (direct UPDATE bypasses triggers). Phase 2 errors per-attachment are logged and skipped — the workspace keeps importing rather than rolling back. The import handler returns the new workspace and the operator can inspect logs for any attachment that didn't make it. CLI: - pad workspace export now defaults to --bundle (.tar.gz) since pad import handles bundles. --json reverts to legacy items-only. - pad import auto-detects format by file extension (.tar.gz / .tgz → application/gzip). Other extensions go through the legacy JSON path. - New Client.PostRawWithContentType for explicit-content-type POSTs. Tests: - TestImportBundle_RoundTrip: upload → embed in markdown → export source → import to FRESH server → verify attachment list has 1 row with new UUID → item content rewritten to new UUID and old UUID is gone → download new blob matches original bytes. - TestImportBundle_LegacyJSONStillWorks: JSON content-type still hits the legacy path. - TestImportBundle_RejectsBadGzip: garbage gzip body returns 400. Parent: PLAN-866. With TASK-884 + TASK-885 merged, the round-trip acceptance criterion (export → import → images intact) is met. * fix(attachments): stream import end-to-end per Codex (round 1) Two memory regressions Codex caught on PR #306: P1 (server). importBundle was buffering every blob into a map[string][]byte during a first pass, then iterating the manifest on a second pass. A 2 GiB bundle full of 25 MiB attachments would pin ~2 GiB of heap. Reworked to single-pass streaming: pad-export.json → import workspace + build slug→id map attachments/manifest.json → index entries by tar path attachments/<uuid>.<ext> → look up entry, rehydrate now The export bundler always writes pad-export.json + manifest.json BEFORE any blob (deterministic order from handlers_export_bundle.go), so this works without buffering. Bundles that violate the ordering — a third-party tool that writes blobs first — return 400 with a clear error. Memory footprint now bounded by the largest single blob (≤25 MiB) regardless of bundle size. Stale blobs without a manifest entry are skipped (their bytes io.Copy'd to io.Discard so the tar reader stays in sync). Unknown top-level entries (forward-compat for future bundle additions) are also consumed and ignored rather than left dangling. P2 (CLI). pad import used os.ReadFile, buffering the entire bundle client-side before posting. Switched to os.Open + a new Client.PostStreamWithContentType helper that streams the body directly into the request — together with the server-side fix, import is end-to-end streaming. Tests: - TestImportBundle_RejectsOutOfOrderTar: hand-crafted bundle with a blob before pad-export.json returns 400 with "ordering" in the message. - existing TestImportBundle_RoundTrip / LegacyJSONStillWorks / RejectsBadGzip continue to pass under the new streaming flow. * fix(cli): give streaming endpoints a 1h timeout per Codex (round 2) Codex P1 round 2: PostStreamWithContentType + RawStream were both using the shared 10s-timeout httpClient. The default works fine for normal API calls but kills a multi-GiB bundle import or export over anything slower than a local network — Client.Timeout fires mid-stream with "Client.Timeout exceeded". Added a dedicated streamClient on Client with a 1h timeout, used by both RawStream (export bundle download) and PostStreamWithContentType (import bundle upload). 1h is generous enough for ~100 MB/s uplinks shipping a 350 GiB bundle and still caps a hung connection eventually. The 10s default stays in place for every other call — short timeouts are the right SLA for normal API requests and protect the CLI from hanging on a wedged server. * fix(attachments): make import bundle cap configurable per Codex (round 3) Codex P1: the 2 GiB import cap was hard-coded with a comment promising operator override "later" — but no setter existed, so workspaces over 2 GiB stream out fine on export and fail on re-import. Added Server.SetImportBundleMaxBytes wired from the PAD_IMPORT_BUNDLE_MAX_BYTES env var in cmd/pad/main.go. Mirrors the existing PAD_ATTACHMENT_MAX_BYTES pattern. Default stays at 2 GiB so the typical workspace works without configuration; operators with larger exports can raise it without recompiling. The per-blob cap (importBlobMaxBytes = 25 MiB) is intentionally kept constant — it bounds in-flight memory regardless of total bundle size, and a 25 MiB-per-blob ceiling matches the upload handler's default, so a bundle can never smuggle larger blobs than the upload endpoint accepts. * fix(attachments): scale per-blob import cap with PAD_ATTACHMENT_MAX_BYTES per Codex (round 4) Codex P1 round 4: importBlobMaxBytes was hard-coded at 25 MiB but the upload handler's per-file cap is configurable via PAD_ATTACHMENT_MAX_BYTES. An operator who raised the upload cap to allow 50 MiB attachments could export a workspace successfully (WorkspaceAttachmentsForExport doesn't gate on size) but the re-import would reject every blob over 25 MiB. Replaced the const with effectiveBlobMaxBytes() which reads s.attachmentMaxBytes (or falls back to defaultAttachmentMaxBytes). The pad-export.json cap also scales with this value (4×) so a content-heavy workspace doesn't trip its own JSON ceiling on a server with raised attachment limits. Error message on a too-large blob now points the operator at PAD_ATTACHMENT_MAX_BYTES so they know which knob to turn rather than digging through code to find the cap. * fix(attachments): independent metadata cap for bundle import per Codex (round 5) Codex P2 round 5: tying pad-export.json + manifest.json caps to PAD_ATTACHMENT_MAX_BYTES regressed deployments that LOWER the attachment cap. A 1 MiB attachment cap would force metadata to fit in 4 MiB / 1 MiB respectively — but metadata size scales with workspace item count, not attachment blob sizes, so a tight upload limit shouldn't gate it. Added importMetadataMaxBytes = 100 MiB constant for both metadata files. effectiveBlobMaxBytes() still drives the per-blob cap which genuinely tracks attachment-upload policy.
This commit is contained in:
+55
-18
@@ -377,6 +377,19 @@ func serveCmd() *cobra.Command {
|
||||
srv.SetAttachments(attachReg, attachMax)
|
||||
slog.Info("Attachment storage wired", "backend", "fs", "dir", attachDir)
|
||||
|
||||
// Workspace bundle import cap. Default is 2 GiB inside
|
||||
// internal/server; PAD_IMPORT_BUNDLE_MAX_BYTES lets
|
||||
// operators with larger exports raise the ceiling without
|
||||
// recompiling (Codex review on PR #306 round 3).
|
||||
if v := os.Getenv("PAD_IMPORT_BUNDLE_MAX_BYTES"); v != "" {
|
||||
if n, perr := strconv.ParseInt(v, 10, 64); perr == nil && n > 0 {
|
||||
srv.SetImportBundleMaxBytes(n)
|
||||
slog.Info("Import bundle cap overridden", "max_bytes", n)
|
||||
} else {
|
||||
slog.Warn("PAD_IMPORT_BUNDLE_MAX_BYTES ignored — not a positive integer", "value", v)
|
||||
}
|
||||
}
|
||||
|
||||
// Wire the image processor used for thumbnail derivation
|
||||
// (TASK-878) and the editor's rotate/crop tools (TASK-879/880).
|
||||
// The default build picks the pure-Go backend (no cgo);
|
||||
@@ -4526,24 +4539,23 @@ Examples:
|
||||
|
||||
func exportCmd() *cobra.Command {
|
||||
var outputFile string
|
||||
var bundle bool
|
||||
var jsonOnly bool
|
||||
cmd := &cobra.Command{
|
||||
Use: "export",
|
||||
Short: "Export workspace to JSON or a self-contained tar.gz bundle",
|
||||
Long: `Export the current workspace (collections, items, comments, versions)
|
||||
to a portable JSON file. Pass --bundle to include attachment blobs in
|
||||
a tar.gz bundle:
|
||||
Short: "Export workspace as a self-contained tar.gz bundle",
|
||||
Long: `Export the current workspace (collections, items, comments, versions,
|
||||
and attachments) to a portable tar.gz bundle:
|
||||
|
||||
pad-export.json — workspace metadata + items + collections + ...
|
||||
attachments/manifest.json — uuid → {filename, mime, size, content_hash}
|
||||
attachments/<uuid>.<ext> — original attachment blobs
|
||||
|
||||
The default stays JSON for now so existing pad import flows keep
|
||||
working. The bundle becomes the default once pad import learns to
|
||||
unpack tar.gz (planned for TASK-885).`,
|
||||
Pass --json to emit the legacy items-only JSON file (no attachments).
|
||||
Both formats can be re-imported via 'pad workspace import'.`,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
client, _ := getClient()
|
||||
ws := getWorkspace()
|
||||
bundle := !jsonOnly
|
||||
|
||||
path := "/workspaces/" + ws + "/export"
|
||||
defaultExt := ".json"
|
||||
@@ -4614,7 +4626,7 @@ unpack tar.gz (planned for TASK-885).`,
|
||||
},
|
||||
}
|
||||
cmd.Flags().StringVarP(&outputFile, "output", "o", "", "output file path (default: stdout)")
|
||||
cmd.Flags().BoolVar(&bundle, "bundle", false, "emit a tar.gz bundle including attachment blobs (TASK-885 will make this the default once pad import handles bundles)")
|
||||
cmd.Flags().BoolVar(&jsonOnly, "json", false, "emit legacy items-only JSON (no attachments)")
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -4624,31 +4636,56 @@ func importCmd() *cobra.Command {
|
||||
var nameFlag string
|
||||
cmd := &cobra.Command{
|
||||
Use: "import <file>",
|
||||
Short: "Import workspace from JSON export",
|
||||
Long: `Import a workspace from a previously exported JSON file. Creates a new workspace with regenerated IDs.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
Short: "Import workspace from JSON export or tar.gz bundle",
|
||||
Long: `Import a workspace from a previously exported file. Creates a new
|
||||
workspace with regenerated IDs.
|
||||
|
||||
Accepts both formats produced by 'pad workspace export':
|
||||
- .json (legacy, items only)
|
||||
- .tar.gz (new bundle, includes attachment blobs)
|
||||
|
||||
Format is detected by file extension. Override workspace name with --name.`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
client, _ := getClient()
|
||||
filePath := args[0]
|
||||
|
||||
data, err := os.ReadFile(filePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
|
||||
path := "/workspaces/import"
|
||||
if nameFlag != "" {
|
||||
path += "?name=" + nameFlag
|
||||
}
|
||||
|
||||
// Detect bundle by extension. .tar.gz / .tgz route through
|
||||
// the gzip-stream import path; everything else goes the
|
||||
// legacy JSON route. We don't sniff magic bytes — extension
|
||||
// is the explicit signal the user gave us.
|
||||
contentType := "application/json"
|
||||
low := strings.ToLower(filePath)
|
||||
if strings.HasSuffix(low, ".tar.gz") || strings.HasSuffix(low, ".tgz") {
|
||||
contentType = "application/gzip"
|
||||
}
|
||||
|
||||
// Stream the file rather than reading it all in. A multi-
|
||||
// GiB bundle would otherwise pin that much memory client-
|
||||
// side before the first byte hits the wire — defeats the
|
||||
// server's streaming import (Codex P2 on PR #306 round 1).
|
||||
f, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open file: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var ws models.Workspace
|
||||
if err := client.PostRaw(path, data, &ws); err != nil {
|
||||
if err := client.PostStreamWithContentType(path, f, contentType, &ws); err != nil {
|
||||
return fmt.Errorf("import: %w", err)
|
||||
}
|
||||
|
||||
fmt.Printf("Imported workspace %q (slug: %s)\n", ws.Name, ws.Slug)
|
||||
fmt.Printf(" Collections: imported\n")
|
||||
fmt.Printf(" Items, comments, links, versions: imported\n")
|
||||
if contentType == "application/gzip" {
|
||||
fmt.Printf(" Attachments: rehydrated from bundle\n")
|
||||
}
|
||||
fmt.Printf(" All IDs regenerated\n")
|
||||
return nil
|
||||
},
|
||||
|
||||
+48
-4
@@ -18,8 +18,14 @@ import (
|
||||
type Client struct {
|
||||
baseURL string
|
||||
httpClient *http.Client
|
||||
authToken string // session or API token, sent as Authorization: Bearer
|
||||
agentName string // optional agent name, sent as X-Pad-Agent header
|
||||
// streamClient has a much longer timeout than httpClient and is
|
||||
// used by RawStream / PostStreamWithContentType for endpoints
|
||||
// that can transfer multi-GiB payloads (workspace export
|
||||
// bundles). Sharing the default 10s timeout would kill those
|
||||
// transfers mid-flight on anything but a local network.
|
||||
streamClient *http.Client
|
||||
authToken string // session or API token, sent as Authorization: Bearer
|
||||
agentName string // optional agent name, sent as X-Pad-Agent header
|
||||
}
|
||||
|
||||
func NewClient(host string, port int) *Client {
|
||||
@@ -34,6 +40,17 @@ func NewClientFromURL(baseURL string) *Client {
|
||||
httpClient: &http.Client{
|
||||
Timeout: 10 * time.Second,
|
||||
},
|
||||
// Long-running transfer client for streaming endpoints
|
||||
// (workspace export bundles in/out, future S3 downloads).
|
||||
// 10s on the default client is the right SLA for normal API
|
||||
// calls but kills a multi-GiB bundle upload mid-stream over
|
||||
// anything but a fast local link. 1 hour is generous enough
|
||||
// for ~100 MB/s uplinks shipping a 350 GiB bundle and still
|
||||
// caps a hung connection eventually. (Codex review on PR
|
||||
// #306 round 2.)
|
||||
streamClient: &http.Client{
|
||||
Timeout: 1 * time.Hour,
|
||||
},
|
||||
}
|
||||
|
||||
// Auto-load credentials if available
|
||||
@@ -417,7 +434,7 @@ func (c *Client) RawStream(path string, w io.Writer) (int64, *http.Response, err
|
||||
if err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
resp, err := c.httpClient.Do(req)
|
||||
resp, err := c.streamClient.Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, fmt.Errorf("request failed: %w", err)
|
||||
}
|
||||
@@ -431,11 +448,19 @@ func (c *Client) RawStream(path string, w io.Writer) (int64, *http.Response, err
|
||||
|
||||
// PostRaw sends raw bytes to the API and decodes the JSON response.
|
||||
func (c *Client) PostRaw(path string, data []byte, result interface{}) error {
|
||||
return c.PostRawWithContentType(path, data, "application/json", result)
|
||||
}
|
||||
|
||||
// PostRawWithContentType is the explicit-content-type variant of
|
||||
// PostRaw. Used by the bundle import path to send a tar.gz as
|
||||
// application/gzip so the server's content-type dispatch routes the
|
||||
// request to the bundle handler instead of the JSON decoder.
|
||||
func (c *Client) PostRawWithContentType(path string, data []byte, contentType string, result interface{}) error {
|
||||
req, err := c.newRequest("POST", path, bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Content-Type", contentType)
|
||||
resp, err := c.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("request failed: %w", err)
|
||||
@@ -444,6 +469,25 @@ func (c *Client) PostRaw(path string, data []byte, result interface{}) error {
|
||||
return c.handleResponse(resp, result)
|
||||
}
|
||||
|
||||
// PostStreamWithContentType POSTs a streaming body (typically an
|
||||
// *os.File for a multi-GiB bundle import) without buffering the full
|
||||
// payload in memory client-side. Mirrors the server's streaming
|
||||
// import path — together they keep import memory bounded by the
|
||||
// largest single blob (~25 MiB) rather than the full bundle size.
|
||||
func (c *Client) PostStreamWithContentType(path string, body io.Reader, contentType string, result interface{}) error {
|
||||
req, err := c.newRequest("POST", path, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", contentType)
|
||||
resp, err := c.streamClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("request failed: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
return c.handleResponse(resp, result)
|
||||
}
|
||||
|
||||
// --- Auth API ---
|
||||
|
||||
// LoginResponse is the response from POST /auth/login.
|
||||
|
||||
@@ -0,0 +1,437 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/PerpetualSoftware/pad/internal/attachments"
|
||||
"github.com/PerpetualSoftware/pad/internal/models"
|
||||
)
|
||||
|
||||
// defaultImportBundleMaxBytes caps an uploaded bundle. Mirrors the
|
||||
// upload handler's defaultAttachmentMaxBytes scaling: a workspace
|
||||
// export can contain thousands of attachments, so this is much
|
||||
// higher than any single-file upload limit. The cap exists primarily
|
||||
// to bound the temp-file footprint on the import host. Operators
|
||||
// running larger workspaces should override via
|
||||
// Server.SetImportBundleMaxBytes (wired from PAD_IMPORT_BUNDLE_MAX_BYTES
|
||||
// in cmd/pad/main.go).
|
||||
const defaultImportBundleMaxBytes int64 = 2 << 30 // 2 GiB
|
||||
|
||||
// importMetadataMaxBytes is the size ceiling for the small JSON
|
||||
// payloads inside a bundle (pad-export.json + attachments/manifest.json).
|
||||
// Independent of the per-blob cap so a deployment that LOWERS
|
||||
// PAD_ATTACHMENT_MAX_BYTES (e.g. to 1 MiB for a tightly-controlled
|
||||
// host) doesn't inadvertently reject metadata for a workspace
|
||||
// nobody intended to gate on attachment-blob limits. (Codex P2 on
|
||||
// PR #306 round 5.)
|
||||
//
|
||||
// 100 MiB comfortably holds a workspace with many thousands of
|
||||
// items + version history. Bumped via the import bundle cap, not
|
||||
// per-attachment cap, since metadata size scales with the export's
|
||||
// item count, not attachment sizes.
|
||||
const importMetadataMaxBytes int64 = 100 << 20 // 100 MiB
|
||||
|
||||
// effectiveBlobMaxBytes returns the per-blob ceiling for bundle
|
||||
// import — matches whatever the upload handler accepts so an
|
||||
// operator who raised PAD_ATTACHMENT_MAX_BYTES on the source can
|
||||
// re-import the resulting export on a destination configured the
|
||||
// same way. Codex flagged the hard-coded 25 MiB cap on PR #306
|
||||
// round 4: a workspace with attachments uploaded under a larger
|
||||
// cap would round-trip the export but fail the re-import.
|
||||
func (s *Server) effectiveBlobMaxBytes() int64 {
|
||||
if s.attachmentMaxBytes > 0 {
|
||||
return s.attachmentMaxBytes
|
||||
}
|
||||
return defaultAttachmentMaxBytes
|
||||
}
|
||||
|
||||
// handleImportWorkspaceBundle accepts a tar.gz bundle produced by
|
||||
// the export endpoint and rebuilds the workspace including all
|
||||
// attachment blobs.
|
||||
//
|
||||
// The handler does the JSON / bundle dispatch; the actual work is
|
||||
// done by importBundle which is unit-testable without the http stack.
|
||||
//
|
||||
// Bundle layout (matches handlers_export_bundle.go):
|
||||
//
|
||||
// pad-export.json
|
||||
// attachments/manifest.json
|
||||
// attachments/<uuid>.<ext>
|
||||
//
|
||||
// Two-phase flow:
|
||||
// 1. Parse pad-export.json, run the existing ImportWorkspace path to
|
||||
// create the workspace + items. Returns an item-ID map (old → new).
|
||||
// 2. For each manifest entry, find the matching tar entry, rehydrate
|
||||
// the blob through the storage backend (re-validate MIME + hash),
|
||||
// and insert an attachment row pointed at the remapped item.
|
||||
// Build an attachment-ID map (old → new) as we go.
|
||||
// 3. Scan all imported items' content + fields for pad-attachment:OLD
|
||||
// references and rewrite to pad-attachment:NEW.
|
||||
//
|
||||
// Errors before phase 2 begins return a clean HTTP error. Errors mid-
|
||||
// rehydrate are logged with attachment_id context; the workspace is
|
||||
// kept (it has live items) and the partial attachment state is left
|
||||
// for the operator to inspect. Orphan GC will eventually reclaim any
|
||||
// blob whose row insertion failed — the upload-handler's "blob may be
|
||||
// orphan on disk" comment applies here too.
|
||||
func (s *Server) handleImportWorkspaceBundle(w http.ResponseWriter, r *http.Request) {
|
||||
if s.attachments == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "attachments_disabled",
|
||||
"Attachment storage is not configured on this server")
|
||||
return
|
||||
}
|
||||
// Bound the request body BEFORE the gzip reader spools any of it.
|
||||
maxBytes := s.importBundleMaxBytes
|
||||
if maxBytes <= 0 {
|
||||
maxBytes = defaultImportBundleMaxBytes
|
||||
}
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
|
||||
|
||||
gz, err := gzip.NewReader(r.Body)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_bundle",
|
||||
"Could not read gzip stream: "+err.Error())
|
||||
return
|
||||
}
|
||||
defer gz.Close()
|
||||
|
||||
newName := r.URL.Query().Get("name")
|
||||
userID := currentUserID(r)
|
||||
|
||||
ws, err := s.importBundle(r.Context(), gz, newName, userID)
|
||||
if err != nil {
|
||||
// Errors from importBundle are already shaped with status hints —
|
||||
// surface as 400 unless the underlying error wraps an http hint.
|
||||
var statusErr *importStatusError
|
||||
if errors.As(err, &statusErr) {
|
||||
writeError(w, statusErr.status, statusErr.code, statusErr.message)
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadRequest, "import_failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Mirror the JSON-import path's owner-attachment so the workspace
|
||||
// shows up under the importer's account.
|
||||
if userID != "" {
|
||||
_ = s.store.AddWorkspaceMember(ws.ID, userID, "owner")
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, ws)
|
||||
}
|
||||
|
||||
// importBundle reads a tar (already gzip-decompressed) from r and
|
||||
// orchestrates the two-phase import. Returns the new workspace.
|
||||
//
|
||||
// Single-pass streaming: the export bundler always writes
|
||||
// pad-export.json + attachments/manifest.json BEFORE any blob, so
|
||||
// we can run ImportWorkspace + parse the manifest as soon as those
|
||||
// two entries land, then stream-rehydrate each subsequent blob
|
||||
// without ever holding the full bundle in memory. Bundles that
|
||||
// violate the ordering — e.g. a third-party tool that put blobs
|
||||
// first — are rejected with a clear error.
|
||||
//
|
||||
// Memory footprint: at most one blob (≤ effectiveBlobMaxBytes) held
|
||||
// at a time during rehydration, plus the small JSON payloads at the
|
||||
// front. A 2 GiB bundle with thousands of 25 MiB images now needs
|
||||
// ~25 MiB peak rather than ~2 GiB. (Codex P1 on PR #306 round 1.)
|
||||
//
|
||||
// Split out from the handler so tests can drive it with a tar.Reader
|
||||
// over an in-memory bundle and assert on the resulting state without
|
||||
// a live HTTP server.
|
||||
func (s *Server) importBundle(ctx context.Context, r io.Reader, newName, ownerID string) (*models.Workspace, error) {
|
||||
tr := tar.NewReader(r)
|
||||
blobCap := s.effectiveBlobMaxBytes()
|
||||
|
||||
var ws *models.Workspace
|
||||
var manifestByPath map[string]*models.AttachmentManifestEntry
|
||||
var oldItemIDToSlug, slugToNewID map[string]string
|
||||
oldAttachToNew := map[string]string{}
|
||||
exportSeen := false
|
||||
manifestSeen := false
|
||||
|
||||
for {
|
||||
hdr, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read tar entry: %w", err)
|
||||
}
|
||||
if hdr.Typeflag != tar.TypeReg && hdr.Typeflag != tar.TypeRegA { //nolint:staticcheck // TypeRegA accepted for older bundles
|
||||
continue
|
||||
}
|
||||
|
||||
switch {
|
||||
case hdr.Name == "pad-export.json":
|
||||
// pad-export.json can grow large for content-heavy
|
||||
// workspaces (items + version history). Use the
|
||||
// metadata-specific cap so deployments that lower
|
||||
// PAD_ATTACHMENT_MAX_BYTES (e.g. to 1 MiB) don't
|
||||
// inadvertently make metadata fail.
|
||||
if hdr.Size > importMetadataMaxBytes {
|
||||
return nil, fmt.Errorf("pad-export.json exceeds %d-byte cap (declared %d)", importMetadataMaxBytes, hdr.Size)
|
||||
}
|
||||
buf, err := readEntry(tr, hdr.Size)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read pad-export.json: %w", err)
|
||||
}
|
||||
var export models.WorkspaceExport
|
||||
if err := json.Unmarshal(buf, &export); err != nil {
|
||||
return nil, &importStatusError{
|
||||
status: http.StatusBadRequest, code: "bad_bundle",
|
||||
message: "Bundle pad-export.json could not be decoded: " + err.Error(),
|
||||
}
|
||||
}
|
||||
ws, err = s.store.ImportWorkspace(&export, newName, ownerID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("import workspace: %w", err)
|
||||
}
|
||||
oldItemIDToSlug = make(map[string]string, len(export.Items))
|
||||
for _, it := range export.Items {
|
||||
oldItemIDToSlug[it.ID] = it.Slug
|
||||
}
|
||||
slugToNewID, err = s.store.WorkspaceItemSlugMap(ws.ID)
|
||||
if err != nil {
|
||||
return ws, fmt.Errorf("build slug→id map: %w", err)
|
||||
}
|
||||
exportSeen = true
|
||||
|
||||
case hdr.Name == "attachments/manifest.json":
|
||||
if !exportSeen {
|
||||
return nil, &importStatusError{
|
||||
status: http.StatusBadRequest, code: "bad_bundle",
|
||||
message: "Bundle ordering violation: manifest.json before pad-export.json",
|
||||
}
|
||||
}
|
||||
// Manifest size scales with attachment count, not blob
|
||||
// content, so use the metadata cap rather than the
|
||||
// per-blob one (same rationale as pad-export.json above).
|
||||
if hdr.Size > importMetadataMaxBytes {
|
||||
return ws, fmt.Errorf("manifest.json exceeds %d-byte cap (declared %d)",
|
||||
importMetadataMaxBytes, hdr.Size)
|
||||
}
|
||||
buf, err := readEntry(tr, hdr.Size)
|
||||
if err != nil {
|
||||
return ws, fmt.Errorf("read manifest.json: %w", err)
|
||||
}
|
||||
var manifest models.AttachmentManifest
|
||||
if err := json.Unmarshal(buf, &manifest); err != nil {
|
||||
return ws, fmt.Errorf("manifest decode: %w (workspace created but attachments not restored)", err)
|
||||
}
|
||||
if manifest.Version > exportBundleVersion {
|
||||
return ws, fmt.Errorf("manifest version %d not supported by this server (max %d)",
|
||||
manifest.Version, exportBundleVersion)
|
||||
}
|
||||
manifestByPath = make(map[string]*models.AttachmentManifestEntry, len(manifest.Entries))
|
||||
for i := range manifest.Entries {
|
||||
e := &manifest.Entries[i]
|
||||
manifestByPath[bundleAttachmentPath(e.ID, e.Filename)] = e
|
||||
}
|
||||
manifestSeen = true
|
||||
|
||||
case strings.HasPrefix(hdr.Name, "attachments/"):
|
||||
if !exportSeen {
|
||||
return nil, &importStatusError{
|
||||
status: http.StatusBadRequest, code: "bad_bundle",
|
||||
message: "Bundle ordering violation: attachment blob before pad-export.json",
|
||||
}
|
||||
}
|
||||
if !manifestSeen {
|
||||
return ws, &importStatusError{
|
||||
status: http.StatusBadRequest, code: "bad_bundle",
|
||||
message: "Bundle ordering violation: attachment blob before manifest.json",
|
||||
}
|
||||
}
|
||||
entry, ok := manifestByPath[hdr.Name]
|
||||
if !ok {
|
||||
// Blob has no manifest entry — could be a stale entry
|
||||
// from a bundle the operator hand-edited. Skip the
|
||||
// bytes (consume the tar slot) and move on.
|
||||
if _, err := io.Copy(io.Discard, io.LimitReader(tr, hdr.Size)); err != nil {
|
||||
return ws, fmt.Errorf("skip unmanifested blob %s: %w", hdr.Name, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if hdr.Size > blobCap {
|
||||
return ws, fmt.Errorf("blob %s exceeds %d-byte cap (declared %d) — raise PAD_ATTACHMENT_MAX_BYTES on this server to allow",
|
||||
hdr.Name, blobCap, hdr.Size)
|
||||
}
|
||||
blob, err := readEntry(tr, hdr.Size)
|
||||
if err != nil {
|
||||
return ws, fmt.Errorf("read blob %s: %w", hdr.Name, err)
|
||||
}
|
||||
newAttID, err := s.rehydrateAttachment(ctx, ws.ID, entry, blob,
|
||||
oldItemIDToSlug, slugToNewID, ownerID)
|
||||
if err != nil {
|
||||
slog.Warn("import: rehydrate failed",
|
||||
"attachment_id", entry.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
oldAttachToNew[entry.ID] = newAttID
|
||||
|
||||
default:
|
||||
// Unknown top-level entry — consume it so the tar reader
|
||||
// stays in sync, then forward-compat ignore. Future
|
||||
// bundle versions might add a CHANGELOG.md or schema
|
||||
// migration script we don't recognize yet.
|
||||
if _, err := io.Copy(io.Discard, io.LimitReader(tr, hdr.Size)); err != nil {
|
||||
return ws, fmt.Errorf("skip unknown entry %s: %w", hdr.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !exportSeen {
|
||||
return nil, &importStatusError{
|
||||
status: http.StatusBadRequest, code: "bad_bundle",
|
||||
message: "Bundle is missing pad-export.json",
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 3: rewrite pad-attachment:OLD references in every imported
|
||||
// item's content + fields to pad-attachment:NEW. Done via store
|
||||
// helper so we get a single transactional pass and the FTS index
|
||||
// is updated correctly.
|
||||
if len(oldAttachToNew) > 0 {
|
||||
if err := s.store.RemapAttachmentReferencesInWorkspace(ws.ID, oldAttachToNew); err != nil {
|
||||
slog.Warn("import: attachment reference remap failed",
|
||||
"workspace_id", ws.ID, "error", err)
|
||||
// Non-fatal — items still exist with stale references.
|
||||
// Operator can re-run a remap manually if needed.
|
||||
}
|
||||
}
|
||||
|
||||
// Drop the storage-usage cache — the imported attachments
|
||||
// just bumped the workspace total.
|
||||
s.storageInfoCache.invalidate(ws.ID)
|
||||
|
||||
return ws, nil
|
||||
}
|
||||
|
||||
// readEntry reads exactly size bytes from a tar reader (the rest of
|
||||
// the current entry) into a buffer, validating that the read length
|
||||
// matches the header's declared Size. Tar entries are bounded by the
|
||||
// caller; this helper just makes the read+verify pattern uniform.
|
||||
func readEntry(tr *tar.Reader, size int64) ([]byte, error) {
|
||||
buf, err := io.ReadAll(io.LimitReader(tr, size+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(buf)) != size {
|
||||
return nil, fmt.Errorf("read %d bytes, header says %d", len(buf), size)
|
||||
}
|
||||
return buf, nil
|
||||
}
|
||||
|
||||
// rehydrateAttachment runs the upload-handler's MIME validation,
|
||||
// hash, and store.Put for one manifest entry, then inserts a fresh
|
||||
// attachments row in the new workspace. Returns the new UUID. The
|
||||
// new row points at the remapped item id when the original was
|
||||
// attached to one; orphan attachments stay orphaned.
|
||||
func (s *Server) rehydrateAttachment(
|
||||
ctx context.Context,
|
||||
workspaceID string,
|
||||
entry *models.AttachmentManifestEntry,
|
||||
blob []byte,
|
||||
oldItemIDToSlug, slugToNewID map[string]string,
|
||||
ownerID string,
|
||||
) (string, error) {
|
||||
// Defense in depth: re-validate the MIME against the allowlist on
|
||||
// the first 512 bytes. Trusting the manifest's mime field would
|
||||
// let a malicious bundle smuggle a blocked type past the upload
|
||||
// gate. If the actual bytes don't sniff to the manifest's mime,
|
||||
// trust the sniffed value (matches the upload handler's policy).
|
||||
head := blob
|
||||
if len(head) > 512 {
|
||||
head = head[:512]
|
||||
}
|
||||
allowed, code, vErr := attachments.ValidateUpload(head, entry.Filename)
|
||||
if vErr != nil {
|
||||
return "", fmt.Errorf("mime validation (%s): %w", code, vErr)
|
||||
}
|
||||
|
||||
// Hash the blob ourselves rather than trusting the manifest. A
|
||||
// bundle could lie about content_hash; the storage layer's
|
||||
// hash-verify guards us at write time, but hashing locally lets
|
||||
// the dedupe path work even when the supplied hash is wrong.
|
||||
sum := sha256.Sum256(blob)
|
||||
hash := hex.EncodeToString(sum[:])
|
||||
|
||||
// Hand the bytes to the configured backend. Same path the upload
|
||||
// handler uses; FSStore re-hashes defensively.
|
||||
store, err := s.attachments.Resolve(attachments.FSPrefix + ":" + hash)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("resolve attachment store: %w", err)
|
||||
}
|
||||
storageKey, err := store.Put(ctx, hash, allowed.MIME, strings.NewReader(string(blob)))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("store.Put: %w", err)
|
||||
}
|
||||
|
||||
// Translate the old item id (from the manifest) into the new id
|
||||
// via item.slug, which ImportWorkspace preserves.
|
||||
var newItemIDPtr *string
|
||||
if entry.ItemID != "" {
|
||||
if slug, ok := oldItemIDToSlug[entry.ItemID]; ok {
|
||||
if newID, ok := slugToNewID[slug]; ok && newID != "" {
|
||||
newItemIDPtr = &newID
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
uploadedBy := entry.UploadedBy
|
||||
if uploadedBy == "" {
|
||||
uploadedBy = ownerID
|
||||
}
|
||||
if uploadedBy == "" {
|
||||
uploadedBy = "system"
|
||||
}
|
||||
|
||||
att := &models.Attachment{
|
||||
WorkspaceID: workspaceID,
|
||||
ItemID: newItemIDPtr,
|
||||
UploadedBy: uploadedBy,
|
||||
StorageKey: storageKey,
|
||||
ContentHash: hash,
|
||||
MimeType: allowed.MIME,
|
||||
SizeBytes: int64(len(blob)),
|
||||
Filename: entry.Filename,
|
||||
Width: entry.Width,
|
||||
Height: entry.Height,
|
||||
}
|
||||
if err := s.store.CreateAttachment(att); err != nil {
|
||||
return "", fmt.Errorf("create attachment row: %w", err)
|
||||
}
|
||||
|
||||
// Re-derive thumbnails for image originals. Mirrors the upload
|
||||
// handler — runs async via goAsync so the import handler doesn't
|
||||
// stall on imaging work, and Server.Stop() waits for in-flight
|
||||
// derivation before close.
|
||||
if allowed.Category == attachments.CategoryImage && s.imageProcessor != nil {
|
||||
original := att.ID
|
||||
s.goAsync(func() { s.deriveThumbnails(original) })
|
||||
}
|
||||
|
||||
return att.ID, nil
|
||||
}
|
||||
|
||||
// importStatusError lets importBundle return errors with HTTP-status
|
||||
// hints attached, so the handler doesn't have to repeat the
|
||||
// classification. Keeps importBundle pure-Go-testable.
|
||||
type importStatusError struct {
|
||||
status int
|
||||
code string
|
||||
message string
|
||||
}
|
||||
|
||||
func (e *importStatusError) Error() string { return e.message }
|
||||
@@ -0,0 +1,261 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/PerpetualSoftware/pad/internal/attachments"
|
||||
"github.com/PerpetualSoftware/pad/internal/models"
|
||||
)
|
||||
|
||||
// TestImportBundle_RoundTrip pins TASK-885's acceptance criterion:
|
||||
// export from one workspace → import into a fresh server → items
|
||||
// keep their pad-attachment:UUID references intact (rewritten to
|
||||
// the new attachment ids), the blobs are reachable through the
|
||||
// download endpoint, and the byte content matches the original
|
||||
// upload.
|
||||
//
|
||||
// The most realistic test of a feature that touches three layers
|
||||
// (export tar, import dispatch, attachment-reference remap). If any
|
||||
// stage drops a UUID or fails to rewrite content, the final item
|
||||
// won't render the image and this test catches it.
|
||||
func TestImportBundle_RoundTrip(t *testing.T) {
|
||||
// 1. Source workspace: upload an attachment, attach it to an
|
||||
// item, embed the pad-attachment: reference in the item's
|
||||
// markdown content.
|
||||
src, srcSlug := testServerWithAttachments(t)
|
||||
|
||||
body := realPNG()
|
||||
rr := doMultipartUpload(src, srcSlug, "logo.png", body)
|
||||
if rr.Code != http.StatusCreated {
|
||||
t.Fatalf("upload: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var upload struct {
|
||||
ID string `json:"id"`
|
||||
URL string `json:"url"`
|
||||
}
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &upload); err != nil {
|
||||
t.Fatalf("decode upload: %v", err)
|
||||
}
|
||||
|
||||
// Create an item that references the attachment in markdown.
|
||||
itemContent := fmt.Sprintf("Hello world\n\n\n", upload.ID)
|
||||
rr = doRequest(src, "POST", "/api/v1/workspaces/"+srcSlug+"/collections/docs/items",
|
||||
map[string]any{"title": "With Image", "content": itemContent})
|
||||
if rr.Code != http.StatusCreated {
|
||||
t.Fatalf("create item: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
// 2. Export the source workspace as a bundle.
|
||||
rr = doRequest(src, "GET", "/api/v1/workspaces/"+srcSlug+"/export?format=tar", nil)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("export: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
bundle := rr.Body.Bytes()
|
||||
|
||||
// 3. Spin up a fresh server (independent storage) and import the
|
||||
// bundle into it. Using a separate server is what proves the
|
||||
// UUID remap actually works — same-server import would
|
||||
// accidentally pass even if the remap were broken.
|
||||
dest, _ := testServerWithAttachments(t)
|
||||
|
||||
req := httptest.NewRequest("POST", "/api/v1/workspaces/import?name=Imported",
|
||||
bytes.NewReader(bundle))
|
||||
req.Header.Set("Content-Type", "application/gzip")
|
||||
req.RemoteAddr = "127.0.0.1:1234"
|
||||
rr = httptest.NewRecorder()
|
||||
dest.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusCreated {
|
||||
t.Fatalf("import: status=%d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var newWS models.Workspace
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &newWS); err != nil {
|
||||
t.Fatalf("decode new ws: %v", err)
|
||||
}
|
||||
|
||||
// 4. The destination workspace must have a new attachments table
|
||||
// with the rehydrated row, and the imported item's content
|
||||
// must reference the NEW attachment id (not the old one).
|
||||
rr = doRequest(dest, "GET", "/api/v1/workspaces/"+newWS.Slug+"/attachments", nil)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("list attachments: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var attResp struct {
|
||||
Attachments []struct {
|
||||
ID string `json:"id"`
|
||||
Filename string `json:"filename"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
} `json:"attachments"`
|
||||
}
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &attResp); err != nil {
|
||||
t.Fatalf("decode att list: %v", err)
|
||||
}
|
||||
if len(attResp.Attachments) != 1 {
|
||||
t.Fatalf("imported attachments: got %d, want 1", len(attResp.Attachments))
|
||||
}
|
||||
newAttID := attResp.Attachments[0].ID
|
||||
if newAttID == upload.ID {
|
||||
t.Fatalf("attachment id was NOT remapped (got %s, original %s)", newAttID, upload.ID)
|
||||
}
|
||||
if attResp.Attachments[0].SizeBytes != int64(len(body)) {
|
||||
t.Errorf("imported attachment size=%d, want %d",
|
||||
attResp.Attachments[0].SizeBytes, len(body))
|
||||
}
|
||||
if attResp.Attachments[0].Filename != "logo.png" {
|
||||
t.Errorf("imported attachment filename=%q, want logo.png",
|
||||
attResp.Attachments[0].Filename)
|
||||
}
|
||||
|
||||
// 5. Item content must reference the NEW attachment id (rewrite
|
||||
// pass worked) and NOT the old one. Read the imported item
|
||||
// via the docs collection's items endpoint.
|
||||
rr = doRequest(dest, "GET", "/api/v1/workspaces/"+newWS.Slug+"/collections/docs/items", nil)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("list items: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var items []models.Item
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &items); err != nil {
|
||||
t.Fatalf("decode items: %v body=%s", err, rr.Body.String())
|
||||
}
|
||||
var imported *models.Item
|
||||
for i := range items {
|
||||
if items[i].Title == "With Image" {
|
||||
imported = &items[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if imported == nil {
|
||||
t.Fatalf("imported item not found; got %d items", len(items))
|
||||
}
|
||||
if !strings.Contains(imported.Content, "pad-attachment:"+newAttID) {
|
||||
t.Errorf("imported content missing new attachment ref %s; content=%q",
|
||||
newAttID, imported.Content)
|
||||
}
|
||||
if strings.Contains(imported.Content, "pad-attachment:"+upload.ID) {
|
||||
t.Errorf("imported content still has stale old attachment ref %s; content=%q",
|
||||
upload.ID, imported.Content)
|
||||
}
|
||||
|
||||
// 6. Download the rehydrated blob and confirm bytes match the
|
||||
// original upload. This is the strongest guarantee that the
|
||||
// storage backend correctly received the bytes from the bundle.
|
||||
rr = doRequest(dest, "GET",
|
||||
"/api/v1/workspaces/"+newWS.Slug+"/attachments/"+newAttID, nil)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("download imported blob: %d", rr.Code)
|
||||
}
|
||||
if !bytes.Equal(rr.Body.Bytes(), body) {
|
||||
t.Errorf("imported blob differs from original upload (got %d bytes, want %d)",
|
||||
rr.Body.Len(), len(body))
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportBundle_LegacyJSONStillWorks confirms the JSON dispatch
|
||||
// still works alongside the new bundle path. Hits the same endpoint
|
||||
// with JSON content-type — must route to the existing handler.
|
||||
func TestImportBundle_LegacyJSONStillWorks(t *testing.T) {
|
||||
srv, slug := testServerWithAttachments(t)
|
||||
|
||||
if rr := doRequest(srv, "POST", "/api/v1/workspaces/"+slug+"/collections/docs/items",
|
||||
map[string]any{"title": "Plain", "content": "no attachments"}); rr.Code != http.StatusCreated {
|
||||
t.Fatalf("create item: %d", rr.Code)
|
||||
}
|
||||
rr := doRequest(srv, "GET", "/api/v1/workspaces/"+slug+"/export", nil)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("export json: %d", rr.Code)
|
||||
}
|
||||
jsonExport := rr.Body.Bytes()
|
||||
|
||||
// Import into a fresh server using JSON content-type.
|
||||
dest, _ := testServerWithAttachments(t)
|
||||
req := httptest.NewRequest("POST", "/api/v1/workspaces/import?name=JsonImport",
|
||||
bytes.NewReader(jsonExport))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.RemoteAddr = "127.0.0.1:1234"
|
||||
rr = httptest.NewRecorder()
|
||||
dest.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusCreated {
|
||||
t.Fatalf("legacy JSON import: status=%d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportBundle_RejectsOutOfOrderTar pins the streaming-import
|
||||
// invariant added in PR #306 round 1 (Codex P1): the bundle MUST
|
||||
// place pad-export.json + manifest.json BEFORE any blob so the
|
||||
// server can stream-rehydrate without buffering. Bundles that put
|
||||
// blobs first would still work in the previous implementation but
|
||||
// would force buffering of every blob — we now reject them up front
|
||||
// with a clear error rather than silently buffering.
|
||||
func TestImportBundle_RejectsOutOfOrderTar(t *testing.T) {
|
||||
srv, _ := testServerWithAttachments(t)
|
||||
|
||||
// Hand-craft a tar.gz where a blob entry comes BEFORE
|
||||
// pad-export.json. Strict-format violation.
|
||||
var buf bytes.Buffer
|
||||
gzw := gzip.NewWriter(&buf)
|
||||
tw := tar.NewWriter(gzw)
|
||||
if err := tw.WriteHeader(&tar.Header{Name: "attachments/abcd.png", Mode: 0o644, Size: 4}); err != nil {
|
||||
t.Fatalf("write header: %v", err)
|
||||
}
|
||||
if _, err := tw.Write([]byte{0xde, 0xad, 0xbe, 0xef}); err != nil {
|
||||
t.Fatalf("write blob: %v", err)
|
||||
}
|
||||
// Even if we'd write a manifest after, the blob coming first
|
||||
// already violates the contract. Stop here.
|
||||
tw.Close()
|
||||
gzw.Close()
|
||||
|
||||
req := httptest.NewRequest("POST", "/api/v1/workspaces/import", bytes.NewReader(buf.Bytes()))
|
||||
req.Header.Set("Content-Type", "application/gzip")
|
||||
req.RemoteAddr = "127.0.0.1:1234"
|
||||
rr := httptest.NewRecorder()
|
||||
srv.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusBadRequest {
|
||||
t.Errorf("out-of-order bundle: status=%d, want 400; body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if !strings.Contains(rr.Body.String(), "ordering") {
|
||||
t.Errorf("expected ordering-violation error, got body=%s", rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportBundle_RejectsBadGzip pins the early-error path: a
|
||||
// truncated/invalid gzip body must return 400, not 500. Catches
|
||||
// regressions where the gzip-reader error gets swallowed and the
|
||||
// handler proceeds with a half-decompressed stream.
|
||||
func TestImportBundle_RejectsBadGzip(t *testing.T) {
|
||||
srv := testServer(t)
|
||||
// Wire the attachment registry so the dispatcher doesn't 503.
|
||||
srv.SetAttachments(attachments.NewRegistry(), 0)
|
||||
|
||||
garbage := []byte("not a gzip stream")
|
||||
req := httptest.NewRequest("POST", "/api/v1/workspaces/import",
|
||||
bytes.NewReader(garbage))
|
||||
req.Header.Set("Content-Type", "application/gzip")
|
||||
req.RemoteAddr = "127.0.0.1:1234"
|
||||
rr := httptest.NewRecorder()
|
||||
srv.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusBadRequest {
|
||||
t.Errorf("bad gzip: status=%d, want 400; body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// readBundleAsBytes is a tiny helper for callers that want the raw
|
||||
// bundle to feed straight into another POST. Kept separate from
|
||||
// readBundle (which extracts a map) so import-side tests don't have
|
||||
// to re-encode the bundle.
|
||||
func readBundleAsBytes(t *testing.T, body io.Reader) []byte {
|
||||
t.Helper()
|
||||
out, err := io.ReadAll(body)
|
||||
if err != nil {
|
||||
t.Fatalf("read bundle: %v", err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -343,6 +343,26 @@ func (s *Server) handleExportWorkspace(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
func (s *Server) handleImportWorkspace(w http.ResponseWriter, r *http.Request) {
|
||||
// Content-Type dispatch:
|
||||
// application/gzip / application/x-gzip / application/x-tar
|
||||
// → tar.gz bundle path (TASK-885) — handles attachments.
|
||||
// anything else → JSON path (legacy items-only).
|
||||
//
|
||||
// We prefer Content-Type over file-magic sniffing so a misnamed
|
||||
// upload fails fast with a clear error rather than silently going
|
||||
// through the wrong code path. The CLI's pad import command sets
|
||||
// the right header based on the file extension; web UI does the
|
||||
// same when uploading a .tar.gz.
|
||||
ct := strings.TrimSpace(r.Header.Get("Content-Type"))
|
||||
if i := strings.IndexByte(ct, ';'); i >= 0 {
|
||||
ct = ct[:i]
|
||||
}
|
||||
ct = strings.ToLower(strings.TrimSpace(ct))
|
||||
if ct == "application/gzip" || ct == "application/x-gzip" || ct == "application/x-tar" {
|
||||
s.handleImportWorkspaceBundle(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
var data models.WorkspaceExport
|
||||
// WorkspaceExport contains all collections, items, comments, and item
|
||||
// versions for the workspace — even a modest project export blows past
|
||||
|
||||
@@ -79,6 +79,13 @@ type Server struct {
|
||||
// guarding.
|
||||
storageInfoCache *storageInfoCache
|
||||
|
||||
// importBundleMaxBytes caps a single workspace import bundle.
|
||||
// 0 → defaultImportBundleMaxBytes (2 GiB). Set via
|
||||
// SetImportBundleMaxBytes from cmd/pad/main.go using the
|
||||
// PAD_IMPORT_BUNDLE_MAX_BYTES env var so operators with larger
|
||||
// exports can opt in without recompiling.
|
||||
importBundleMaxBytes int64
|
||||
|
||||
// bg tracks fire-and-forget goroutines spawned by request handlers
|
||||
// (TouchUserActivity in middleware_auth, async email sends, etc.) so
|
||||
// the server can drain them before shutdown / test cleanup. Without
|
||||
@@ -279,6 +286,16 @@ func (s *Server) SetImageProcessor(p attachments.Processor) {
|
||||
s.imageProcessor = p
|
||||
}
|
||||
|
||||
// SetImportBundleMaxBytes overrides the default 2 GiB cap on a
|
||||
// single workspace import bundle. Set to 0 to fall back to the
|
||||
// default. Wired from PAD_IMPORT_BUNDLE_MAX_BYTES in cmd/pad/main.go
|
||||
// so operators with workspaces over 2 GiB can opt in without
|
||||
// recompiling. Larger caps trade memory headroom (one blob in
|
||||
// flight at a time, ≤25 MiB) for a longer import wall-clock.
|
||||
func (s *Server) SetImportBundleMaxBytes(n int64) {
|
||||
s.importBundleMaxBytes = n
|
||||
}
|
||||
|
||||
// SetSecureCookies enables the Secure flag on all cookies.
|
||||
func (s *Server) SetSecureCookies(secure bool) {
|
||||
s.secureCookies = secure
|
||||
|
||||
@@ -554,6 +554,126 @@ func (s *Store) SoftDeleteAttachment(id string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// WorkspaceItemSlugMap returns slug → id for every live item in a
|
||||
// workspace. Used by the bundle import path (TASK-885) to remap an
|
||||
// old attachment's item_id (from the manifest) to the freshly-
|
||||
// generated id, via item.slug which ImportWorkspace preserves.
|
||||
//
|
||||
// Soft-deleted items are excluded; the import path can't realistically
|
||||
// recreate an attachment under a deleted parent without also
|
||||
// resurrecting the parent, and the manifest's item_id only has
|
||||
// meaning for live items at export time.
|
||||
func (s *Store) WorkspaceItemSlugMap(workspaceID string) (map[string]string, error) {
|
||||
rows, err := s.db.Query(s.q(`
|
||||
SELECT id, slug FROM items WHERE workspace_id = ? AND deleted_at IS NULL
|
||||
`), workspaceID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("workspace item slug map: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]string{}
|
||||
for rows.Next() {
|
||||
var id, slug string
|
||||
if err := rows.Scan(&id, &slug); err != nil {
|
||||
return nil, fmt.Errorf("scan slug map: %w", err)
|
||||
}
|
||||
out[slug] = id
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("iterate slug map: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// RemapAttachmentReferencesInWorkspace rewrites every
|
||||
// "pad-attachment:OLD" reference in items.content + items.fields
|
||||
// to "pad-attachment:NEW" for every (old, new) pair in the map.
|
||||
// Run after a bundle import has rehydrated attachments so item
|
||||
// content points at the new attachment ids instead of the source
|
||||
// workspace's ids.
|
||||
//
|
||||
// Implementation: a single transaction that loads every item's
|
||||
// content+fields, runs strings.ReplaceAll for each pair, and
|
||||
// writes back only when something changed. ReplaceAll is safe
|
||||
// because attachment UUIDs don't appear as substrings of other
|
||||
// UUIDs (RFC4122 hex, length 36, all unique by construction).
|
||||
//
|
||||
// FTS reindex via the existing rebuild helper happens AFTER the
|
||||
// transaction commits — direct UPDATE bypasses the SQLite FTS
|
||||
// triggers the same way ImportWorkspace's INSERTs do.
|
||||
func (s *Store) RemapAttachmentReferencesInWorkspace(workspaceID string, oldToNew map[string]string) error {
|
||||
if len(oldToNew) == 0 {
|
||||
return nil
|
||||
}
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin remap tx: %w", err)
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
rows, err := tx.Query(s.q(`SELECT id, content, fields FROM items WHERE workspace_id = ? AND deleted_at IS NULL`), workspaceID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("scan items for remap: %w", err)
|
||||
}
|
||||
type rowUpdate struct {
|
||||
id string
|
||||
content string
|
||||
fields string
|
||||
}
|
||||
var updates []rowUpdate
|
||||
for rows.Next() {
|
||||
var id, content, fields string
|
||||
if err := rows.Scan(&id, &content, &fields); err != nil {
|
||||
rows.Close()
|
||||
return fmt.Errorf("scan item: %w", err)
|
||||
}
|
||||
newContent := remapAttachmentRefs(content, oldToNew)
|
||||
newFields := remapAttachmentRefs(fields, oldToNew)
|
||||
if newContent != content || newFields != fields {
|
||||
updates = append(updates, rowUpdate{id: id, content: newContent, fields: newFields})
|
||||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return fmt.Errorf("iterate items for remap: %w", err)
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
for _, u := range updates {
|
||||
if _, err := tx.Exec(s.q(`UPDATE items SET content = ?, fields = ? WHERE id = ?`),
|
||||
u.content, u.fields, u.id); err != nil {
|
||||
return fmt.Errorf("update item %s: %w", u.id, err)
|
||||
}
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("commit remap: %w", err)
|
||||
}
|
||||
|
||||
// Refresh FTS so search queries see the rewritten content.
|
||||
s.rebuildFTSForWorkspace(workspaceID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// remapAttachmentRefs replaces "pad-attachment:OLD" with
|
||||
// "pad-attachment:NEW" for every (old, new) pair in the map. Pure
|
||||
// string operation — kept private so callers go through
|
||||
// RemapAttachmentReferencesInWorkspace which also handles the FTS
|
||||
// reindex.
|
||||
//
|
||||
// The "pad-attachment:" prefix is part of the search key so we don't
|
||||
// accidentally rewrite a UUID that happens to appear in unrelated
|
||||
// content (e.g. an item title that mentions an attachment id by
|
||||
// accident — unlikely but free to guard against).
|
||||
func remapAttachmentRefs(s string, oldToNew map[string]string) string {
|
||||
for old, fresh := range oldToNew {
|
||||
if old == "" || fresh == "" || old == fresh {
|
||||
continue
|
||||
}
|
||||
s = strings.ReplaceAll(s, "pad-attachment:"+old, "pad-attachment:"+fresh)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// WorkspaceAttachmentsForExport returns every original (non-derived,
|
||||
// non-deleted) attachment in the workspace so the export bundler
|
||||
// can stream them into the tar. Derived rows (thumbnails) are
|
||||
|
||||
Reference in New Issue
Block a user