gsadmin/garage
1
0
Fork 0
mirror of https://github.com/deuxfleurs-org/garage.git synced 2026-07-27 16:28:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
726 Commits 31 Branches 89 Tags
c692f55d5ce2c3ed08db7fbc4844debcc0aeb134
Commit Graph

495 Commits

Author SHA1 Message Date
Alex Auvolat c692f55d5c K2V: Fix end parameter and add tests (fix #305) 2022-05-17 11:50:23 +02:00
Alex Auvolat 7b474855e3 Make background runner terminate correctly 2022-05-17 11:38:31 +02:00
Alex 5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat def78c5e6f Update netapp to 0.4.4, fix #300 2022-05-09 12:08:47 +02:00
Alex 277a20ec44 Fix layout show to not show changes when there are no changes (#297) 
fixes #295, partially

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/297
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-09 11:14:55 +02:00
Alex Auvolat c9ef3e461b fix clippy 2022-04-19 12:50:40 +02:00
Alex Auvolat c93008d333 Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex Auvolat e5341ca47b Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
Alex Auvolat 5d404dcd54 Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
Alex Auvolat 94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat cb5836d53c Bring maximum exponential backoff time down from 16h to 1h 2022-04-07 11:49:29 +02:00
Alex Auvolat 0091002ef2 New replication modes and their documentation 2022-03-28 16:26:04 +02:00
Alex Auvolat 8f9cf3a5d1 fix a clippy lint 2022-03-28 15:48:55 +02:00
Alex Auvolat 913f7754bb Add blocks in errored state to garage stats 2022-03-28 15:47:23 +02:00
Alex Auvolat 42dde54126 Log admin GET requests at debug level instead of info 
to reduce noise in logs
 2022-03-28 15:46:52 +02:00
Alex Auvolat 9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat 509d256c58 Make layout optimization work in relative terms 2022-03-24 15:27:14 +01:00
Alex Auvolat 2814d41842 Allow garage layout assign to assign to several nodes at once 2022-03-24 15:27:13 +01:00
Alex Auvolat 7e0e2ffda2 Slight change and add comment to layout assignation algo 2022-03-24 15:27:13 +01:00
Alex Auvolat 413ab0eaed Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex Auvolat 3dc9214172 Add lots of comments on how the resync queue works 
(I don't really want to change/refactor that code though)
 2022-03-23 10:25:39 +01:00
Alex Auvolat 077dd1cde9 Clippy 2022-03-23 10:25:39 +01:00
Alex Auvolat e480aaf338 Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex Auvolat 8fd6745745 Move block RC code to separate rc.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat c3982a90b6 Move DataBlock out of manager.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat c1d9854d2c Move block manager to separate module 2022-03-23 10:25:15 +01:00
trinity-1686a 8565f7dc31 cleanup 2022-03-23 10:22:37 +01:00
trinity-1686a 8db6b84559 add test for create bucket and put website with streaming signature 2022-03-23 10:22:37 +01:00
trinity-1686a 1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
KokaKiwi e934934f14 garage_api: Update streaming payload stream unit tests 2022-03-23 10:22:36 +01:00
KokaKiwi 98545a16dd garage_api: Handle streaming payload early in request handling 2022-03-23 10:22:36 +01:00
Alex Auvolat db46cdef79 Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex Auvolat ba6b56ae68 Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex Auvolat 0af314b295 Add comment for fsync 2022-03-14 11:54:00 +01:00
Alex Auvolat d78bf379fb Fix resync queue to not drop items 2022-03-14 11:51:37 +01:00
Alex Auvolat f7e6f4616f Spawn a single resync worker 2022-03-14 11:51:37 +01:00
Alex Auvolat dc5ec4ecf9 Add appropriate fsync() calls in write_block 
to ensure that data is persisted properly
 2022-03-14 11:51:32 +01:00
Alex Auvolat fe62d01b7e Implement exponential backoff for resync retries 2022-03-14 11:41:20 +01:00
Alex Auvolat 9b2b531f4d Make admin server optional 2022-03-14 10:54:25 +01:00
Alex Auvolat 2377a92f6b Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
Alex Auvolat 203e8d2c34 Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex Auvolat f869ca625d Add spans to table calls, change span names in RPC 2022-03-14 10:54:12 +01:00
Alex Auvolat 0cc31ee169 add missing netapp telemetry feature 2022-03-14 10:54:11 +01:00
Alex Auvolat dc8d0496cc Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
Alex Auvolat d9a35359bf Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
Alex Auvolat 2a5609b292 Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex Auvolat 818daa5c78 Refactor how durations are measured 2022-03-14 10:53:35 +01:00
Alex Auvolat f0d0cd9a20 Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
Alex Auvolat 55d4471599 Remove ... at end of hex IDs 2022-03-14 10:52:31 +01:00