mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-20 17:43:26 +00:00
0fd5d0cdfb
* fix(store): swap plainto_tsquery → websearch_to_tsquery for PG FTS (BUG-842)
`TestListItems_FTS_HyphenatedSearchTerm/task-five` has been failing on
every Go (PostgreSQL) CI run because `plainto_tsquery('english',
'task-five')` doesn't match the asciihword lexeme(s) the english parser
produces for an indexed `task-five-distinctive`. The result is that
every PG full-text search for hyphenated terms returns zero rows.
`websearch_to_tsquery` (Postgres 11+) is purpose-built for arbitrary
user input and tokenizes hyphenated terms the same way `to_tsvector`
does for the indexed document, so the query intersects the index
correctly. Swapped in three spots in the postgres dialect — FTSMatch,
FTSSnippet, FTSRank — and updated the caller-side comments that
referenced plainto_tsquery. SQLite path is unchanged: it goes through
items_fts MATCH with sanitizeFTSQuery, never through these methods.
* fix(server): drain background goroutines on Stop() (BUG-842)
`TestAdminBillingStats_SidecarSidecarError_DegradesToLocalOnly` (and
other server tests) have been flaking on the Go (PostgreSQL) CI runner
with `TempDir RemoveAll cleanup: directory not empty`. Root cause:
several request handlers spawned bare `go func() { ... }()` goroutines
that touched the SQLite WAL DB after the test function returned.
testServer's t.Cleanup closed the store but had no way to drain those
goroutines first, so a fire-and-forget WAL write could re-create the
`-wal`/`-shm` files between Close() and t.TempDir's RemoveAll.
Add a Server.bg sync.WaitGroup, a Server.goAsync helper that wraps a
WaitGroup-tracked goroutine, and a Server.Stop() that blocks until
every goAsync closure has finished. Convert the four known
fire-and-forget sites to goAsync:
- middleware_auth.go (TouchUserActivity)
- handlers_auth.go (password reset email)
- handlers_cloud.go (stripe_processed_events pruning)
- handlers_members.go (workspace invitation email)
Wire `srv.Stop()` into both testServer (server_test.go) and
newMetricsTestServer (metrics_auth_test.go) so cleanup order is
Stop → Close → TempDir RemoveAll. Add
TestServer_Stop_DrainsBackgroundGoroutines to pin the contract: a
goAsync goroutine must block Stop until it returns.
* fix(store): correct PG FTS hyphenation via OR-combined plainto_tsquery (BUG-842)
The previous attempt swapped plainto_tsquery → websearch_to_tsquery,
which was wrong: websearch_to_tsquery treats `-` as a NEGATION operator
(Google-style), so `task-five` becomes `task & !five` and the search
returns 0 rows for the same reason as before. This commit reverts the
swap and applies the actual fix.
PG's english parser indexes `task-five-distinctive` as
`{task-five-distinct, task, five, distinct}` — the asciihword AND its
parts. plainto_tsquery applied to the partial query `task-five`
produces `task-fiv & task & five`: the stemmed asciihword for the
PARTIAL query (`task-fiv`) is NOT in the vector, so the AND fails.
Replacing the hyphen with a space makes plainto emit `task & five`,
which DOES match — but doing that unconditionally breaks `BUG-842`-
style queries: PG indexes the `-842` suffix as a negative-number
lexeme, so `plainto_tsquery('BUG-842')` matches via `-842`, while
`plainto_tsquery('BUG 842')` searches for `842` and misses.
The fix ORs the two query variants together so the search vector is
matched against either the raw user query OR its hyphen-as-space form.
Both `task-five` (against `task-five-distinctive`) and `BUG-842`
(against `BUG-842 fix the cleanup race`) hit. Verified locally against
postgres:17-alpine via PAD_TEST_POSTGRES_URL — both 10x stress and
race-detector runs are green.
Surfaces:
- dialect.go: FTSMatch / FTSSnippet / FTSRank now consume TWO
placeholders each in the PG dialect.
- items.go: listItemsFTS PG branch + SearchItems PG branch update
args to pass (raw, sanitized) for every PG `?` placeholder.
- search.go: SearchItems main / count / facets PG branches updated
likewise. New sanitizePGFTSQuery helper alongside sanitizeFTSQuery.
- documents.go: ListDocuments PG branch updated.
Tests:
- TestListItems_FTS_HyphenatedSearchTerm extended with a `BUG-842`
case to pin the OR-combined logic — naive hyphen-stripping would
silently regress this.
- New TestSanitizePGFTSQuery unit test.
* chore: gofmt 11 files with import-order issues (BUG-842 PR cleanup)
The Go (SQLite) CI job has been failing on `main` (and every PR built
against it) because golangci-lint flags 11 files whose third-party
imports are intermixed with internal imports — the import-grouping
rule that gofmt enforces. None of these were introduced by the
BUG-842 PR; they're pre-existing on main. The PR can't go green
without this cleanup, though, so it's bundled here.
Pure mechanical change — `gofmt -w <files>` only re-orders import
groups; no logic changes. Files touched:
cmd/pad/configure.go
cmd/pad/main.go
internal/cli/format.go
internal/server/handlers_admin_invitations.go
internal/server/handlers_admin_users.go
internal/server/handlers_grants.go
internal/server/handlers_share_links.go
internal/server/handlers_stars.go
internal/server/middleware_auth.go
internal/store/store.go
internal/store/store_test.go
After this commit `gofmt -l ./cmd ./internal` returns clean.
298 lines
9.6 KiB
Go
298 lines
9.6 KiB
Go
package store
|
|
|
|
import (
|
|
"fmt"
|
|
"strings"
|
|
)
|
|
|
|
// DriverType identifies the database backend.
|
|
type DriverType string
|
|
|
|
const (
|
|
DriverSQLite DriverType = "sqlite"
|
|
DriverPostgres DriverType = "postgres"
|
|
)
|
|
|
|
// Dialect encapsulates SQL syntax differences between database backends.
|
|
// The Store calls dialect methods to generate backend-specific SQL fragments.
|
|
type Dialect interface {
|
|
// Driver returns the driver type.
|
|
Driver() DriverType
|
|
|
|
// Placeholder returns the nth parameter placeholder (1-indexed).
|
|
// SQLite: "?", PostgreSQL: "$1", "$2", etc.
|
|
Placeholder(n int) string
|
|
|
|
// Rebind converts a query with "?" placeholders to the dialect's format.
|
|
// For SQLite this is a no-op. For PostgreSQL, "?" becomes "$1", "$2", etc.
|
|
Rebind(query string) string
|
|
|
|
// JSONExtractText returns SQL to extract a text value from a JSON column.
|
|
// SQLite: json_extract(col, '$.key')
|
|
// PostgreSQL: col->>'key'
|
|
JSONExtractText(column, key string) string
|
|
|
|
// JSONExtractPath returns SQL to extract a value at a dotted path from a JSON column.
|
|
// SQLite: json_extract(col, '$.path.to.key')
|
|
// PostgreSQL: col #>> '{path,to,key}'
|
|
JSONExtractPath(column, path string) string
|
|
|
|
// JSONSet returns SQL to set a value at a path in a JSON column.
|
|
// SQLite: json_set(col, '$.key', ?)
|
|
// PostgreSQL: jsonb_set(col::jsonb, '{key}', ?::jsonb)
|
|
// Returns the SQL fragment and any extra placeholders used.
|
|
JSONSet(column, key string) string
|
|
|
|
// JSONRemove returns SQL to remove a key from a JSON column.
|
|
// SQLite: json_remove(col, '$.key')
|
|
// PostgreSQL: col::jsonb - 'key'
|
|
JSONRemove(column, key string) string
|
|
|
|
// Now returns the SQL expression for the current UTC timestamp.
|
|
// SQLite: datetime('now')
|
|
// PostgreSQL: NOW() AT TIME ZONE 'UTC'
|
|
Now() string
|
|
|
|
// NowRFC3339 returns the SQL expression for current UTC time in RFC3339 format.
|
|
// SQLite: strftime('%Y-%m-%dT%H:%M:%SZ', 'now')
|
|
// PostgreSQL: TO_CHAR(NOW() AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')
|
|
NowRFC3339() string
|
|
|
|
// GroupConcat returns SQL for string aggregation with a separator.
|
|
// SQLite: GROUP_CONCAT(DISTINCT expr)
|
|
// PostgreSQL: STRING_AGG(DISTINCT expr, ',')
|
|
GroupConcat(expr string, distinct bool) string
|
|
|
|
// BoolToInt converts a Go bool to a query parameter value.
|
|
// SQLite: 0/1 (integers)
|
|
// PostgreSQL: true/false (native booleans)
|
|
BoolToInt(b bool) interface{}
|
|
|
|
// ILike returns the case-insensitive LIKE operator.
|
|
// SQLite: LIKE (case-insensitive by default)
|
|
// PostgreSQL: ILIKE
|
|
ILike() string
|
|
|
|
// Concat returns SQL to concatenate string expressions.
|
|
// SQLite: expr1 || expr2
|
|
// PostgreSQL: expr1 || expr2 (same, but useful as abstraction point)
|
|
Concat(exprs ...string) string
|
|
|
|
// FTSMatch returns the full-text search WHERE clause fragment.
|
|
// SQLite: "table MATCH ?" — consumes ONE arg.
|
|
// PostgreSQL: an OR-combined plainto_tsquery match that consumes TWO
|
|
// args — the raw user query AND its hyphen-sanitized form. See
|
|
// sanitizePGFTSQuery for the rationale (BUG-842).
|
|
FTSMatch(table, column string) string
|
|
|
|
// FTSSnippet returns SQL for highlighted search result snippets.
|
|
// SQLite: snippet(fts_table, col_idx, '<mark>', '</mark>', '...', 32)
|
|
// — consumes ONE arg.
|
|
// PostgreSQL: ts_headline backed by the same OR-combined tsquery as
|
|
// FTSMatch — consumes TWO args (raw, sanitized).
|
|
FTSSnippet(ftsTable string, colIndex int, sourceColumn string) string
|
|
|
|
// FTSRank returns the column/expression for full-text relevance ranking.
|
|
// SQLite: rank (built-in FTS5 column) — consumes ZERO args.
|
|
// PostgreSQL: ts_rank backed by the same OR-combined tsquery as
|
|
// FTSMatch — consumes TWO args (raw, sanitized).
|
|
FTSRank(table, column string) string
|
|
|
|
// JSONArrayContains returns SQL + the arg to check if a JSON array column
|
|
// contains a given text value.
|
|
// SQLite: "column LIKE ?" with arg `%"value"%`
|
|
// PostgreSQL: "column::jsonb @> ?::jsonb" with arg `["value"]`
|
|
JSONArrayContains(column, value string) (string, interface{})
|
|
}
|
|
|
|
// ---------- SQLite dialect ----------
|
|
|
|
type sqliteDialect struct{}
|
|
|
|
func (d *sqliteDialect) Driver() DriverType { return DriverSQLite }
|
|
|
|
func (d *sqliteDialect) Placeholder(_ int) string { return "?" }
|
|
|
|
func (d *sqliteDialect) Rebind(query string) string { return query }
|
|
|
|
func (d *sqliteDialect) JSONExtractText(column, key string) string {
|
|
return fmt.Sprintf("json_extract(%s, '$.%s')", column, key)
|
|
}
|
|
|
|
func (d *sqliteDialect) JSONExtractPath(column, path string) string {
|
|
return fmt.Sprintf("json_extract(%s, '$.%s')", column, path)
|
|
}
|
|
|
|
func (d *sqliteDialect) JSONSet(column, key string) string {
|
|
return fmt.Sprintf("json_set(%s, '$.%s', ?)", column, key)
|
|
}
|
|
|
|
func (d *sqliteDialect) JSONRemove(column, key string) string {
|
|
return fmt.Sprintf("json_remove(%s, '$.%s')", column, key)
|
|
}
|
|
|
|
func (d *sqliteDialect) Now() string {
|
|
return "datetime('now')"
|
|
}
|
|
|
|
func (d *sqliteDialect) NowRFC3339() string {
|
|
return "strftime('%Y-%m-%dT%H:%M:%SZ', 'now')"
|
|
}
|
|
|
|
func (d *sqliteDialect) GroupConcat(expr string, distinct bool) string {
|
|
if distinct {
|
|
return fmt.Sprintf("GROUP_CONCAT(DISTINCT %s)", expr)
|
|
}
|
|
return fmt.Sprintf("GROUP_CONCAT(%s)", expr)
|
|
}
|
|
|
|
func (d *sqliteDialect) BoolToInt(b bool) interface{} {
|
|
if b {
|
|
return 1
|
|
}
|
|
return 0
|
|
}
|
|
|
|
func (d *sqliteDialect) ILike() string { return "LIKE" }
|
|
|
|
func (d *sqliteDialect) Concat(exprs ...string) string {
|
|
return strings.Join(exprs, " || ")
|
|
}
|
|
|
|
func (d *sqliteDialect) FTSMatch(table, _ string) string {
|
|
return fmt.Sprintf("%s MATCH ?", table)
|
|
}
|
|
|
|
func (d *sqliteDialect) FTSSnippet(ftsTable string, colIndex int, _ string) string {
|
|
return fmt.Sprintf("snippet(%s, %d, '<mark>', '</mark>', '...', 32)", ftsTable, colIndex)
|
|
}
|
|
|
|
func (d *sqliteDialect) FTSRank(_, _ string) string {
|
|
return "rank"
|
|
}
|
|
|
|
func (d *sqliteDialect) JSONArrayContains(column, value string) (string, interface{}) {
|
|
return column + " LIKE ?", "%\"" + value + "\"%"
|
|
}
|
|
|
|
// ---------- PostgreSQL dialect ----------
|
|
|
|
type postgresDialect struct{}
|
|
|
|
func (d *postgresDialect) Driver() DriverType { return DriverPostgres }
|
|
|
|
func (d *postgresDialect) Placeholder(n int) string {
|
|
return fmt.Sprintf("$%d", n)
|
|
}
|
|
|
|
func (d *postgresDialect) Rebind(query string) string {
|
|
return rebindQuery(query)
|
|
}
|
|
|
|
func (d *postgresDialect) JSONExtractText(column, key string) string {
|
|
return fmt.Sprintf("%s->>'%s'", column, key)
|
|
}
|
|
|
|
func (d *postgresDialect) JSONExtractPath(column, path string) string {
|
|
parts := strings.Split(path, ".")
|
|
return fmt.Sprintf("%s #>> '{%s}'", column, strings.Join(parts, ","))
|
|
}
|
|
|
|
func (d *postgresDialect) JSONSet(column, key string) string {
|
|
return fmt.Sprintf("jsonb_set(COALESCE(%s, '{}')::jsonb, '{%s}', to_jsonb(?::text))", column, key)
|
|
}
|
|
|
|
func (d *postgresDialect) JSONRemove(column, key string) string {
|
|
return fmt.Sprintf("(%s::jsonb - '%s')", column, key)
|
|
}
|
|
|
|
func (d *postgresDialect) Now() string {
|
|
return "(NOW() AT TIME ZONE 'UTC')"
|
|
}
|
|
|
|
func (d *postgresDialect) NowRFC3339() string {
|
|
return "TO_CHAR(NOW() AT TIME ZONE 'UTC', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"')"
|
|
}
|
|
|
|
func (d *postgresDialect) GroupConcat(expr string, distinct bool) string {
|
|
if distinct {
|
|
return fmt.Sprintf("STRING_AGG(DISTINCT %s, ',')", expr)
|
|
}
|
|
return fmt.Sprintf("STRING_AGG(%s, ',')", expr)
|
|
}
|
|
|
|
func (d *postgresDialect) BoolToInt(b bool) interface{} {
|
|
return b
|
|
}
|
|
|
|
func (d *postgresDialect) ILike() string { return "ILIKE" }
|
|
|
|
func (d *postgresDialect) Concat(exprs ...string) string {
|
|
return strings.Join(exprs, " || ")
|
|
}
|
|
|
|
// PG FTS uses an OR-combined plainto_tsquery to handle hyphenated user
|
|
// queries correctly. For an indexed title `task-five-distinctive`, the
|
|
// english parser writes `task-five-distinct, task, five, distinct`. A
|
|
// raw `plainto_tsquery('english', 'task-five')` produces
|
|
// `task-fiv & task & five` — the stemmed asciihword `task-fiv` is NOT in
|
|
// the vector, so the AND fails and the search returns 0 rows. Replacing
|
|
// the hyphen with a space (`'task five'`) makes plainto emit
|
|
// `task & five`, which DOES match. We can't unconditionally do that,
|
|
// though: titles like `BUG-842` are indexed as `bug, -842` (negative
|
|
// number lexeme) — `plainto_tsquery('BUG-842')` matches them via `-842`,
|
|
// but `plainto_tsquery('BUG 842')` would search for the lexeme `842` and
|
|
// miss. ORing the two query variants together covers both cases. See
|
|
// BUG-842 and sanitizePGFTSQuery.
|
|
//
|
|
// Each method below consumes TWO `?` placeholders (raw query, sanitized
|
|
// query). Callers pass them in args via sanitizePGFTSQueryArgs.
|
|
func (d *postgresDialect) FTSMatch(table, column string) string {
|
|
return fmt.Sprintf(
|
|
"%s.%s @@ (plainto_tsquery('english', ?) || plainto_tsquery('english', ?))",
|
|
table, column,
|
|
)
|
|
}
|
|
|
|
func (d *postgresDialect) FTSSnippet(_ string, _ int, sourceColumn string) string {
|
|
return fmt.Sprintf(
|
|
"ts_headline('english', %s, plainto_tsquery('english', ?) || plainto_tsquery('english', ?), 'StartSel=<mark>,StopSel=</mark>,MaxFragments=1,MaxWords=32')",
|
|
sourceColumn,
|
|
)
|
|
}
|
|
|
|
func (d *postgresDialect) FTSRank(table, column string) string {
|
|
return fmt.Sprintf(
|
|
"ts_rank(%s.%s, plainto_tsquery('english', ?) || plainto_tsquery('english', ?))",
|
|
table, column,
|
|
)
|
|
}
|
|
|
|
func (d *postgresDialect) JSONArrayContains(column, value string) (string, interface{}) {
|
|
return column + "::jsonb @> ?::jsonb", `["` + value + `"]`
|
|
}
|
|
|
|
// ---------- Helper ----------
|
|
|
|
// rebindQuery converts "?" placeholders to PostgreSQL's "$1", "$2", etc.
|
|
// Respects string literals (single quotes) and does not modify "?" inside them.
|
|
func rebindQuery(query string) string {
|
|
var buf strings.Builder
|
|
buf.Grow(len(query) + 16)
|
|
n := 0
|
|
inString := false
|
|
for i := 0; i < len(query); i++ {
|
|
ch := query[i]
|
|
if ch == '\'' {
|
|
inString = !inString
|
|
buf.WriteByte(ch)
|
|
} else if ch == '?' && !inString {
|
|
n++
|
|
fmt.Fprintf(&buf, "$%d", n)
|
|
} else {
|
|
buf.WriteByte(ch)
|
|
}
|
|
}
|
|
return buf.String()
|
|
}
|