mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Josh Siviour
86014ac08a
CreateTags is a read-modify-write of one resource's whole tag map. It was serialised by a sync.Mutex on the service struct, which exists once per process, while the handler answers on the spinifex-workers queue group and so runs on whichever node NATS picks. Two nodes read the same map, each added its own key, and the later write won. Every call returned success. Six parallel writers against one security group on a three-node cluster left four tags. On a single node the mutex hides it entirely, which is why no test caught it. The tag index moves from one S3 object per resource to a JetStream KV bucket, where the merge runs inside kvutil.Update and commits at the revision it read. A writer that loses the race re-reads the winner's keys and adds its own on top. S3 could not host the fix: predastore's PutObject has no If-Match, so there is nothing to compare and swap against. Tags written before the move are not migrated in a batch. A batch that half-failed would leave a resource looking untagged, and the next write would commit that as the truth. Instead the CAS seeds a missing entry from the old object inside the attempt, so a resource migrates on its first write and a failure to read the old object fails that one call. DescribeTags reads the union of both key sets, preferring the bucket, and DeleteAllTags clears both so old tags cannot reappear through the fallback. The process-local mutex is removed rather than left beside the CAS. The regression test drives six separate service values over one bucket, because that is what makes it able to fail: with the old code every caller inside one process was already serialised, and only a second node could lose an update.