FDN-02/FDN-03/FDN-07/FDN-08: Migrations-Rollback, Objekt-Storage-Interface, go.sum-Fix, Observability
- FDN-02: Rollback-fähige Down-Migrationen (024-026), archivdms seed dev CLI - FDN-03: internal/objectstore Interface + lokaler WORM-Treiber, signierte Download-URLs - FDN-07: go.mod/go.sum vervollständigt (fehlender go-ldap/v3-Eintrag), CI-Pipeline (.gitea/workflows/ci.yml, bereits in FDN-01 committet) damit lauffähig - FDN-08: Request-ID-Middleware, /metrics-Endpoint, Panic-Recovery, Login/Logout/Me technisches Logging inkl. Access-Log je Anfrage
This commit is contained in:
@@ -1,5 +1,94 @@
|
|||||||
# archivdms – Dev Log
|
# archivdms – Dev Log
|
||||||
|
|
||||||
|
## 2026-08-11 – FDN-08 Nachbesserung: Login-Request erzeugte keine Logzeile mit request_id
|
||||||
|
|
||||||
|
**Zeit:** ca. 0,5 h (Verifikationsbefund nachvollzogen, Ursache eingegrenzt, Access-Log + Auth-Logging ergänzt, Symbole gegengeprüft)
|
||||||
|
|
||||||
|
**Befund vom Deploy auf 192.168.1.204:** Test-Request gegen `/api/auth/login` erzeugte keine Zeile mit `request_id`. **Es war KEINE Regression der Call-Site-Umstellung** — der Grep über `internal/api/*.go` bestätigt: außer den zwei dokumentierten Aufrufen in `SetStorageConfig` (`server.go:113/117`, kein Request-Kontext) existiert kein `s.logger.` mehr. Zwei Vorlücken waren die Ursache:
|
||||||
|
|
||||||
|
1. `handleLogin`/`handleLogout`/`handleMe` in `internal/api/auth_handlers.go` haben **noch nie** technisch geloggt — nur Audit-Log-Einträge geschrieben. `s.reqLog` konnte dort nichts umstellen, weil es keine Call-Site gab.
|
||||||
|
2. `metricsMiddleware` schrieb nur bei `status >= 500` eine Zeile. Ein erfolgreicher (200) oder abgelehnter (401) Login lief damit komplett lautlos durch — AK1 war faktisch nur für Fehler-Requests belegbar.
|
||||||
|
|
||||||
|
**Geändert:**
|
||||||
|
|
||||||
|
- `internal/api/observability.go`: `metricsMiddleware` schreibt jetzt für **jede** Anfrage genau eine Access-Log-Zeile mit `request_id`, gestaffelt nach Status (5xx=Error „request failed", 4xx=Warn „request rejected", Rest=Debug „request completed") inkl. `bytes`. Debug für den Normalfall, damit das Log bei Level Info nicht zuläuft; unter `LOG_LEVEL=debug` ist jeder Request nachverfolgbar.
|
||||||
|
- `internal/api/auth_handlers.go`: `handleLogin` loggt Body-Parse-Fehler (Warn), Fehlschlag (Warn, `username`+`remote_ip`+`reason`, **kein** Passwort/keine Auth-Interna) und Erfolg (Info, `user_id`/`username`/`tenant_id`/`remote_ip`) über `s.reqLog(ctx)`. `handleLogout` loggt den Logout (Info) und ein fehlgeschlagenes Session-Invalidieren; `handleMe` loggt fehlgeschlagene User-Lookups. Die bisher stillschweigend verworfenen Fehler von `s.users.UpdateLastLogin(...)` und `s.authMgr.Logout(...)` (`_ =`) werden jetzt geprüft und geloggt.
|
||||||
|
|
||||||
|
**Verifikation nach Deploy:** ein fehlgeschlagener Login (falsches Passwort) muss auf Level Info eine Warn-Zeile `login failed` **plus** `request rejected` mit identischer `request_id` erzeugen; ein erfolgreicher Login `login succeeded` mit `request_id`.
|
||||||
|
|
||||||
|
**Ohne Go-Toolchain manuell gegengeprüfte Symbole** (für devops-deploy, falls der Build doch bricht): `auth.Manager.LoginFrom(ctx, id, pw, ip) (string, *userstore.User, error)` und `Logout(token) error` (`internal/auth/auth.go:88/330`), `userstore.Store.GetByID(int64) (*User, error)` und `UpdateLastLogin(int64) error` (`internal/userstore/userstore.go:132/387`), Felder `auth.Session{UserID, Username, TenantID}` und `userstore.User{ID, Username, TenantID}`, `s.reqLog(ctx) *slog.Logger`, `s.remoteIP(r) string`, `slog.Logger.Log(ctx, level, msg, args...)`. Keine neuen Imports nötig (`slog` in `observability.go` bereits vorhanden, `auth_handlers.go` unverändert bei `json`/`net/http`/`audit`).
|
||||||
|
|
||||||
|
## 2026-08-11 – FDN-08: Logging, Metriken & Fehler-Tracking
|
||||||
|
|
||||||
|
**Zeit:** ca. 1,5 h (Ticket + bestehendes slog-Muster sichten, Middleware-Kette, Metrics-Endpunkt, Umstellung der Log-Call-Sites, Tests, Doku)
|
||||||
|
**Ziel:** die drei realen Lücken schließen — Korrelations-ID über alle Schichten, `/metrics`, zentrales Panic-Recovery. Kein Loggerwechsel (`log/slog` bleibt), keine neue Fremdabhängigkeit.
|
||||||
|
|
||||||
|
**Neu:** `internal/api/observability.go` (Request-ID-Middleware inkl. Übernahme/Sanitizing von `X-Request-ID`, `loggerFromCtx`/`s.reqLog(ctx)`, `statusRecorder`, `recoverMiddleware`, `metricsMiddleware`, `normalizeRoute`, prozesslokale `metricsRegistry` mit Latenz-Buckets), `internal/api/metrics_handlers.go` (`GET /metrics` im Prometheus-Textformat via `fmt.Fprintf`, IP-Beschränkung loopback + `api.metrics_allowed_ips`), `internal/api/observability_test.go` (je AK mindestens ein Test + Redaction-Test).
|
||||||
|
|
||||||
|
**Geändert:** `internal/api/server.go` (Felder `metrics`/`baseHandler`, Kette `requestID -> metrics -> recover -> mux` in `New()` gebaut statt Lazy-Init in `ServeHTTP` — sonst Data Race; Route `GET /metrics`), `internal/storage/processing_jobs.go` (`CountProcessingJobsByStatus` für die Queue-Länge, bewusst ohne `tenant_id`-Filter, dokumentiert: einziger Aufrufer ist der aggregierte Metrik-Endpunkt), `config/config.go` + `config/config.yml.example` (`api.metrics_allowed_ips`), `cmd/archivdms/main.go` (Wiring), README-Abschnitt „Betrieb: Logging, Metriken & Fehler-Tracking (FDN-08)".
|
||||||
|
|
||||||
|
**Kleinster Cut bei den Call-Sites:** statt jeden Aufruf umzuschreiben, gibt es `s.reqLog(ctx)`, das den Context-Logger nimmt und sonst auf `s.logger` zurückfällt. Alle 40 bisherigen `s.logger.*`-Aufrufe im Request-/Job-Pfad (`document_handlers.go`, `accounting_`, `public_share_`, `signed_url_`, `dashboard_`, `document_export_`, `document_bulk_export_`, `ocr_word_`) wurden mechanisch darauf umgestellt (`r.Context()` in Handlern, `ctx` in `ProcessDocumentJob`/`ReprocessDocument`/`archiveStagedFile`/`trySplitStagedUpload`/`autoAssignTaxonomy`/`generateThumbnailBestEffort`). `SetStorageConfig` bleibt bei `s.logger` (kein Request-Kontext).
|
||||||
|
|
||||||
|
**Geheimnisschutz (Abnahme-Prüfung 2):** es wird nichts aus Query-String, Headern oder Body geloggt. `normalizeRoute` maskiert numerische IDs, hash-artige Segmente und immer das Segment hinter `/share/` — Share-Tokens können damit weder in Logs noch in Metrik-Labels auftauchen. Test `TestNormalizeRouteRedactsSecrets` deckt das ab.
|
||||||
|
|
||||||
|
**Offen / auf 192.168.1.204 zu prüfen:** `go vet` + `go test ./internal/api/...` (hier kein Go-Toolchain), Scrape von `/metrics` (localhost = 200, fremde IP = 403), provozierter Panic → 500 + Logzeile mit `request_id` innerhalb einer Minute, Alarm-Schwelle in der Monitoring-Seite (Prometheus-Regel auf `archivdms_panics_total`/5xx-Rate) einmal auslösen und quittieren — Prüfung 1 und 3 der Kachel sind ohne laufende Instanz nicht abschließbar.
|
||||||
|
|
||||||
|
## 2026-08-11 – FDN-06: UI-Shell & Design-System (Abnahme)
|
||||||
|
|
||||||
|
**Zeit:** ca. 0,5 h (Code-Review Shell/ui-Komponenten/Tokens, Doku-Lücke geschlossen)
|
||||||
|
|
||||||
|
Reine Abnahme-Kachel, kein Neubau. Ergebnis der Prüfung:
|
||||||
|
|
||||||
|
- **AK1 Shell steht:** erfüllt. `src/app/(app)/layout.tsx` setzt `SidebarProvider` → `AppSidebar` + `SidebarInset` → `TopBar` + Inhaltsbereich zusammen; `src/components/shell/` enthält AppSidebar (rollenabhängige Navigation), TopBar (Sidebar-Toggle, Suche, Theme-Umschalter, Benutzermenü), SearchBar, CommandPalette (Cmd+K).
|
||||||
|
- **AK2 Basis-Komponenten:** Komponenten vollständig vorhanden (Table, Dialog, Sheet, Input/Textarea/Label/Calendar, Button, Badge, Card, Tabs, DropdownMenu/Popover/Command, Toast via sonner, Sidebar, Avatar/Progress/Skeleton); alle im Code verwendeten `@/components/ui/*`-Importe lassen sich auf existierende Dateien auflösen, keine fehlende Basiskomponente. **Lücke:** es gab keinerlei Doku dazu. Behoben durch neuen README-Abschnitt „Basis-Komponenten & Design-Tokens (FDN-06)" (Tabelle Komponente → Datei → Einsatzzweck). Kein Storybook — bewusst zu groß für diese Kachel.
|
||||||
|
- **AK3 Design-Tokens:** erfüllt. HSL-Tokens zentral in `src/app/globals.css` (`:root`/`.dark`) inkl. `--radius` und `sidebar-*`, gemappt in `tailwind.config.ts`. Keine Hex-/RGB-Farbliterale in Komponenten (Prüfung per Suche); Inline-`style` nur für berechnete Geometrie (Progress, Cropper, OCR-Overlay). Einzige nicht-tokenisierte Farben sind semantische Statusfarben (emerald/amber/red) an Badges — akzeptiert und jetzt als Regel dokumentiert.
|
||||||
|
|
||||||
|
Prüfungen vor der Abnahme:
|
||||||
|
|
||||||
|
1. **3 Breakpoints visuell:** *nicht durchführbar* (kein Dev-Server/Browser, `node_modules` nicht installiert) — manueller Check empfohlen. Ersatz-Code-Review: Sidebar wechselt über `useIsMobile` auf Sheet-Drawer, `ui/table.tsx` kapselt die Tabelle in `overflow-auto` (horizontal scrollbar statt Umbruch), Dialog/Sheet/Button/Calendar nutzen `sm:`-Varianten. Listenseiten selbst setzen kaum eigene Breakpoints — offener Punkt für `UX-01`.
|
||||||
|
2. **Tastaturbedienung:** durch Radix-Primitives abgedeckt (Dialog/Sheet, DropdownMenu, Popover, Tabs, Avatar, Progress) plus `cmdk` für die Befehlspalette — Fokus-Trap, Escape, Pfeiltasten, `aria-*` kommen aus der Bibliothek, nichts davon wurde überschrieben. Native `<select>`-Elemente (20 Fundstellen) sind ohnehin tastaturbedienbar. Empirischer Test am laufenden UI steht aus.
|
||||||
|
3. **Kontrast AA:** Plausibilitätsrechnung auf den Tokens, keine Tool-Messung. Dunkel: `foreground` 98% auf `background` 3.9% ≈ 17:1, `muted-foreground` 64.9% ≈ 7:1 — klar AA. Hell: `muted-foreground` 46.1% auf Weiß ≈ 4,6:1 — knapp über AA (4,5:1), bei weiterer Aufhellung würde es kippen. Grenzfall: `text-amber-600` auf hellem Hintergrund liegt bei Kleintext unter AA; dort ist ergänzend immer Text/Icon vorhanden, Farbe ist nie alleiniger Bedeutungsträger.
|
||||||
|
|
||||||
|
Geändert: nur `README.md` (Doku) und dieser Eintrag — kein Code angefasst.
|
||||||
|
|
||||||
|
## 2026-08-11 – FDN-03: Objekt-Storage-Abstraktion + signierte Download-URLs
|
||||||
|
|
||||||
|
**Zeit:** ca. 1,0 h (Ticket, bestehende WORM-/Share-Pfade sichten, Interface + Local-Treiber, Handler/Routen, Tests, Doku)
|
||||||
|
**Ziel:** Dateizugriff hinter ein Interface ziehen und zeitlich begrenzte signierte Downloads ergänzen — ohne Pfadschema oder WORM-Semantik anzufassen.
|
||||||
|
|
||||||
|
**Neu:** `internal/objectstore/objectstore.go` (Interface `Store`: `Archive/Open/Stat/Delete/SignedURL/VerifySignedURL`, typisierte Fehler `ErrObjectExists`/`ErrObjectNotFound`/`ErrOutsideTenant`/`ErrSignatureInvalid`/`ErrSignatureExpired`, Pfadschema im Paket-Header dokumentiert), `internal/objectstore/local.go` (`LocalStore`, einzige Implementierung — kein S3, weil WORM an `chmod 0440` hängt), `internal/objectstore/local_test.go`, `internal/api/signed_url_handlers.go`.
|
||||||
|
|
||||||
|
**Refactoring statt Umbau:** Die Schritte 4–7 aus `archiveStagedFile` (Zielordner `<yyyy>/<mm>`, Kollisionsprüfung, Rename mit Copy-Fallback, `chmod 0440`) sind 1:1 nach `LocalStore.Archive` gewandert; `copyFile` in `document_handlers.go` entfällt zugunsten der Kopie im neuen Paket. `handleGetDocumentFile` und der öffentliche Share-Download lesen jetzt über `s.objects.Open` — das prüft zusätzlich, dass der `storage_path` wirklich unter `store/<tenant_id>/` liegt (Containment gegen Traversal/IDOR). `storage.ConfirmDeleteRequest` bleibt bewusst unangetastet (DB-Layer bekommt keinen Filesystem-Treiber), `Delete` steht dafür bereit.
|
||||||
|
|
||||||
|
**Signierte URLs:** kein neues Kryptoschema — HMAC-SHA256 über `v1|tenant|dokument|exp`, Schlüssel per HKDF-SHA256 aus `api.secret` mit eigenem Info-Label `archivdms-storage-url-v1` (analog `internal/cryptutil`). Prinzip wie die Share-Links (Pflicht-Ablauf, per-IP-Rate-Limit über `shareLimiter`, Audit-Trail inkl. Fehlschlägen: neue Events `signed_url_created`/`signed_url_accessed`), aber zustandslos. Routen: `POST /api/documents/{id}/signed-url` (auth, Ownership via `GetDocument(id, tenant)`) und `GET /public/files` (ohne Auth, Signatur ist das Credential). Neuer Config-Key `storage.signed_url_ttl_minutes` (Default 15, pro Anfrage überschreibbar, Deckel 24 h); ohne `api.secret` fällt der Treiber auf einen prozess-lokalen Ephemeral-Key zurück, statt den Start zu verweigern.
|
||||||
|
|
||||||
|
**Geprüfte Symbole (kein Go-Toolchain lokal, manuell gegen die Zieldateien gelesen):** `config.StorageConfig.StorePath()/InboxPath()`, `config.APIConfig.Secret`; `Server`-Felder/Methoden `cfg`, `storageCfg`, `fqdn`, `logger`, `audlog`, `shareLimiter.allow`, `remoteIP`, `logShare(r, event, *int64, username, detail, ok)`, `sessionFromCtx(...).TenantID/UserID/Username`; `storage.Store.GetDocument(ctx, id, tenantID) (*Document, error)` mit `Document.ID/Title/StoragePath`; `storage.ErrDuplicateContentHash`; `ResolvedShare.TenantID()/StoragePath()`; `audit.Entry{EventType,Username,IPAddress,Success,Detail,TenantID}`; Helfer `writeJSON/writeError/detectMimeType/safeDownloadName`.
|
||||||
|
|
||||||
|
**Deploy 2026-08-11:** rsync+update.sh auf 192.168.1.204, Build (Go+Next.js) grün, `archivdms`/`archivdms-web` beide `active`, `/api/health`→200, `/login`→200. `storage.signed_url_ttl_minutes` nicht in Live-Config gesetzt, Code-Default (`ResolvedSignedURLTTL()`) greift.
|
||||||
|
|
||||||
|
**Verifikation auf 192.168.1.204 (durchgeführt, Scratch-Kopie unter `/tmp/fdn03-verify`, danach gelöscht — Produktivquellen `/opt/archivdms-src` und der laufende Dienst wurden nicht angefasst):**
|
||||||
|
- `CGO_ENABLED=0 go build ./...` → **Exit 0**, keine Fehler.
|
||||||
|
- `go vet ./internal/objectstore/... ./internal/api/... ./config/... ./internal/audit/...` → **Exit 0**.
|
||||||
|
- `go test ./internal/objectstore/... -v -cover` → **6/6 PASS**, 66,3 % Coverage. Damit sind die drei Ticket-Prüfungen belegt: Round-Trip Archive→Open inkl. Pfadschema `store/<tenant>/<yyyy>/<mm>/<sha256>.<ext>` und `chmod 0440` (`TestArchiveOpenRoundTrip`), abgelaufene/gefälschte/fremd signierte URL wird abgewiesen (`TestSignedURLLifecycle`, `TestSignedURLDefaultTTL`), fehlendes Objekt liefert `ErrObjectNotFound` für Open/Stat/Delete (`TestMissingObjectErrors`); zusätzlich Mandanten-Containment (`TestOpenForeignTenantRejected`) und Delete (`TestDeleteRemovesObject`).
|
||||||
|
- **Zwei bestehende Repo-Lücken, die für den Build erst überbrückt werden mussten (nicht durch FDN-03 verursacht, im Repo weiterhin offen):** (1) `go.sum` fehlt komplett (bekannt aus FDN-07) — musste im Scratch per `go mod download` erzeugt werden; (2) `github.com/go-ldap/ldap/v3` steht **nicht** in `go.mod`, obwohl `internal/ldapauth` es importiert — ohne `go get github.com/go-ldap/ldap/v3@v3.4.14` bricht `go build ./...` unabhängig von dieser Kachel ab. Beides sollte mit funktionierender Toolchain + `git` (auf 192.168.1.204 nicht installiert, deshalb scheitert `go mod tidy` dort) einmal sauber ins Repo.
|
||||||
|
|
||||||
|
**Noch offen (Laufzeit, nicht geprüft):** echter Upload gegen die produktive Instanz und ein `/public/files`-Abruf vor/nach Ablauf. Kein Commit, kein Push, kein Deploy.
|
||||||
|
|
||||||
|
## 2026-08-11 – FDN-02: Rollback-Pfad für Migrationen + Entwicklungs-Seed
|
||||||
|
|
||||||
|
**Zeit:** ca. 0,8 h (Ticket, Migrations-Muster + Store-Signaturen sichten, Down-SQL, Seed-Command, Doku)
|
||||||
|
**Ziel:** Die beiden echten Lücken der Kachel schließen — Kern-Entitäten existieren bereits, es fehlten Rollback-Pfad und ein Dev-Seed.
|
||||||
|
|
||||||
|
**Rollback-Konvention:** Ab jetzt bekommt jede neue `internal/storage/migrations/NNN_name.sql` eine separate `NNN_name.down.sql` (separate Datei statt `-- DOWN`-Abschnitt, weil Regel 4 „Migrationen werden nie verändert" sonst verletzt würde und `psql -f` so ohne Nachbearbeitung funktioniert). Anforderungen im README festgeschrieben: Precondition (welche Go-Stelle vorher raus muss, sonst legt der nächste Start das Objekt wieder an), explizite Datenverlust-Angabe, WORM/GoBD-Hinweis (Down-SQL darf nie `store/`-Dateien oder `retain_until` berühren), `DROP ... IF EXISTS` in `BEGIN; ... COMMIT;`. Ausführung immer manuell, nie beim Start.
|
||||||
|
|
||||||
|
Rückwirkend als Vorlage ergänzt: `024_ocr_words.down.sql` (regenerierbar via `documents reprocess-all`), `025_document_date_score.down.sql` (nur Konfidenz weg, `document_date` bleibt), `026_accounting_api_keys.down.sql` (alle Pull-API-Keys weg, müssen neu ausgegeben werden). 001–023 bewusst ohne Down-Datei.
|
||||||
|
|
||||||
|
**Seed:** `archivdms seed dev` (`cmd/archivdms/cmd_seed_dev.go`, Dispatch `case "seed"` in `main.go` + Hilfetext) legt Mandant "Testfirma" (slug `testfirma`) und tenant-gebundenen Benutzer `testuser@testfirma.local` (Rolle `domain_admin`, `TenantID` gesetzt) an. Idempotent: Mandant über Slug (`tenantSt.List` scannen, `tenantstore` hat keine Slug-Lookup-Methode), Benutzer über `GetByEmail`; vorhandene Objekte werden wiederverwendet, das Passwort nur mit `-reset-password` neu gesetzt. Passwort kommt aus dem bestehenden `randomPassword()` und wird einmalig im gleichen Kasten-Stil wie `seedDefaultUsers` ausgegeben — kein Secret im Code. Alle Schreibaktionen inkl. Fehlschläge über `audlog.Log(audit.Entry{EventType: "seed_dev", ...})` mit `TenantID`.
|
||||||
|
|
||||||
|
**Geprüfte Symbole (kein Go-Toolchain lokal, manuell gegen die Zieldateien gelesen):** `tenantstore.Store.Create(ctx, name, slug, domain) (*Tenant, error)`, `.List(ctx) ([]*Tenant, error)`, `Tenant.Slug/Name/ID`; `userstore.Store.Create(CreateUserRequest{Username,Email,Password,Role,TenantID *int64}) (*User, error)`, `.GetByEmail(ctx, email)`, `.Update(id, UpdateUserRequest{Password *string})`, Konstanten `RoleUser`/`RoleDomainAdmin`; `audit.New(dsn, logPath, logger) (*Logger, error)`, `Logger.Log(Entry{EventType,Username,Success,Detail,TenantID})`; `config.Load`, `cfg.Database.DSN()`, `cfg.Audit.ResolvedLogPath()`; `randomPassword()` aus `main.go`.
|
||||||
|
|
||||||
|
**Offen / auf 192.168.1.204 zu verifizieren:** echter `go build ./...`, ein Lauf `archivdms seed dev` gegen leere und gegen bestehende DB (Idempotenz), Login mit dem ausgegebenen Passwort, sowie ein Rollback-Probelauf einer Down-Datei auf einer Wegwerf-DB. Kein Commit, kein Push, kein Deploy.
|
||||||
|
|
||||||
## 2026-08-11 – FDN-07: CI-Pipeline & Testharness (Gitea Actions)
|
## 2026-08-11 – FDN-07: CI-Pipeline & Testharness (Gitea Actions)
|
||||||
|
|
||||||
**Zeit:** ca. 0,7 h (Ticket lesen, Makefile/go.mod/package.json/.eslintrc.json sichten, Workflow-Datei schreiben, README-Doku, Verifikation auf 192.168.1.204)
|
**Zeit:** ca. 0,7 h (Ticket lesen, Makefile/go.mod/package.json/.eslintrc.json sichten, Workflow-Datei schreiben, README-Doku, Verifikation auf 192.168.1.204)
|
||||||
@@ -6060,3 +6149,312 @@ Keine Commits in dieser Session.
|
|||||||
Keine Änderungen ermittelbar.
|
Keine Änderungen ermittelbar.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
## 2026-08-11 21:27 – 21:28 (1m)
|
||||||
|
**Beschreibung:** Claude Code Session
|
||||||
|
**Projekt:** archivdms
|
||||||
|
|
||||||
|
### Commits
|
||||||
|
- 9a24ea2 FDN-01: repository & projektgerüst
|
||||||
|
|
||||||
|
### Geänderte Dateien
|
||||||
|
- .claude/agent-memory/archivdms-architect/project_ollama_integration_plan.md | 22 +
|
||||||
|
- .claude/agent-memory/ocr-specialist/MEMORY.md | 4 +
|
||||||
|
- .claude/agent-memory/ocr-specialist/project_deskew_border_trick_tested_negative.md | 16 +
|
||||||
|
- .claude/agent-memory/ocr-specialist/project_deskew_disable_for_photos_tested_negative.md | 24 +
|
||||||
|
- .claude/agent-memory/ocr-specialist/project_deskew_preprocessing.md | 18 +
|
||||||
|
- .claude/agent-memory/ocr-specialist/project_title_heuristic_and_osd_zero_rotate_gap.md | 14 +
|
||||||
|
- .claude/agent-memory/retention-compliance/MEMORY.md | 1 +
|
||||||
|
- .claude/agent-memory/retention-compliance/project_gobd_verfahrensdokumentation.md | 52 +
|
||||||
|
- .claude/agents/DEVLOG.md | 68 +
|
||||||
|
- .claude/agents/archivdms-architect.md | 37 +
|
||||||
|
- .claude/agents/backend-dev.md | 78 +
|
||||||
|
- .claude/agents/code-review.md | 51 +
|
||||||
|
- .claude/agents/db-migrator.md | 52 +
|
||||||
|
- .claude/agents/devops-deploy.md | 76 +
|
||||||
|
- .claude/agents/frontend-dev.md | 68 +
|
||||||
|
- .claude/agents/manticore-performance.md | 53 +
|
||||||
|
- .claude/agents/ocr-specialist.md | 53 +
|
||||||
|
- .claude/agents/retention-compliance.md | 69 +
|
||||||
|
- .claude/agents/retention-dms-vergleich.md | 45 +
|
||||||
|
- .claude/skills/devops-deploy/SKILL.md | 76 +
|
||||||
|
- .eslintrc.json | 3 +
|
||||||
|
- .gitea/workflows/ci.yml | 122 ++
|
||||||
|
- .gitignore | 7 +
|
||||||
|
- DEVLOG.md | 6062 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||||
|
- Makefile | 18 +
|
||||||
|
- cmd/archivdms/cmd_classify_retrain.go | 189 ++
|
||||||
|
- cmd/archivdms/cmd_documents_reprocess_all.go | 249 +++
|
||||||
|
- cmd/archivdms/cmd_reindex.go | 145 ++
|
||||||
|
- cmd/archivdms/cmd_reminders_notify.go | 150 ++
|
||||||
|
- cmd/archivdms/cmd_retention_apply.go | 113 ++
|
||||||
|
- cmd/archivdms/main.go | 365 ++++
|
||||||
|
- components.json | 17 +
|
||||||
|
- config/config.go | 415 +++++
|
||||||
|
- config/config.yml.example | 80 +
|
||||||
|
- deploy/cron.d/archivdms-classify-retrain | 14 +
|
||||||
|
- deploy/cron.d/archivdms-reminders | 13 +
|
||||||
|
- dms-featureliste-prompt.md | 87 +
|
||||||
|
- features/PROJ-1-wiedervorlage.md | 77 +
|
||||||
|
- features/README.md | 11 +
|
||||||
|
- go.mod | 23 +
|
||||||
|
- install.sh | 552 ++++++
|
||||||
|
- internal/api/accounting_handlers.go | 391 ++++
|
||||||
|
- internal/api/akte_handlers.go | 280 +++
|
||||||
|
- internal/api/audit_handlers.go | 51 +
|
||||||
|
- internal/api/auth_handlers.go | 99 +
|
||||||
|
- internal/api/classification_template_handlers.go | 404 ++++
|
||||||
|
- internal/api/compliance_handlers.go | 535 ++++++
|
||||||
|
- internal/api/custom_field_handlers.go | 320 ++++
|
||||||
|
- internal/api/dashboard_handlers.go | 29 +
|
||||||
|
- internal/api/date_extraction.go | 216 +++
|
||||||
|
- internal/api/document_bulk_export_handlers.go | 426 +++++
|
||||||
|
- internal/api/document_export_handlers.go | 265 +++
|
||||||
|
- internal/api/document_handlers.go | 1675 +++++++++++++++++
|
||||||
|
- internal/api/document_note_handlers.go | 111 ++
|
||||||
|
- internal/api/ldap_handlers.go | 168 ++
|
||||||
|
- internal/api/metadata_suggestion_handlers.go | 154 ++
|
||||||
|
- internal/api/ocr_word_handlers.go | 75 +
|
||||||
|
- internal/api/ollama_config_handlers.go | 134 ++
|
||||||
|
- internal/api/permission_handlers.go | 458 +++++
|
||||||
|
- internal/api/processing_job_handlers.go | 122 ++
|
||||||
|
- internal/api/public_share_handlers.go | 281 +++
|
||||||
|
- internal/api/reminder_handlers.go | 171 ++
|
||||||
|
- internal/api/retention_rule_handlers.go | 247 +++
|
||||||
|
- internal/api/saved_view_handlers.go | 140 ++
|
||||||
|
- internal/api/search_handlers.go | 122 ++
|
||||||
|
- internal/api/server.go | 554 ++++++
|
||||||
|
- internal/api/sftp_handlers.go | 116 ++
|
||||||
|
- internal/api/share_handlers.go | 159 ++
|
||||||
|
- internal/api/taxonomy_handlers.go | 291 +++
|
||||||
|
- internal/api/tenant_handlers.go | 50 +
|
||||||
|
- internal/api/tenant_settings_handlers.go | 233 +++
|
||||||
|
- internal/api/trash_handlers.go | 237 +++
|
||||||
|
- internal/api/user_handlers.go | 129 ++
|
||||||
|
- internal/api/workflow_handlers.go | 327 ++++
|
||||||
|
- internal/audit/audit.go | 507 +++++
|
||||||
|
- internal/auth/auth.go | 380 ++++
|
||||||
|
- internal/auth/ratelimit.go | 70 +
|
||||||
|
- internal/barcode/barcode.go | 64 +
|
||||||
|
- internal/classifier/naivebayes.go | 617 +++++++
|
||||||
|
- internal/cryptutil/secretbox.go | 73 +
|
||||||
|
- internal/dateformat/dateformat.go | 61 +
|
||||||
|
- internal/index/index.go | 94 +
|
||||||
|
- internal/index/manticore.go | 319 ++++
|
||||||
|
- internal/jobqueue/jobqueue.go | 258 +++
|
||||||
|
- internal/ldapauth/ldapauth.go | 239 +++
|
||||||
|
- internal/ldapstore/ldapstore.go | 263 +++
|
||||||
|
- internal/llm/ollama.go | 166 ++
|
||||||
|
- internal/mailer/mailer.go | 162 ++
|
||||||
|
- internal/mailer/templates.go | 20 +
|
||||||
|
- internal/matching/matching.go | 219 +++
|
||||||
|
- internal/ocr/convert.go | 230 +++
|
||||||
|
- internal/ocr/coords.go | 359 ++++
|
||||||
|
- internal/ocr/exif.go | 224 +++
|
||||||
|
- internal/ocr/ocr.go | 1060 +++++++++++
|
||||||
|
- internal/ocr/scripts/hough_deskew.py | 134 ++
|
||||||
|
- internal/pagesplit/pagesplit.go | 514 ++++++
|
||||||
|
- internal/sftpserver/server.go | 398 ++++
|
||||||
|
- internal/sftpserver/tenantfs.go | 133 ++
|
||||||
|
- internal/storage/accounting_api_keys.go | 173 ++
|
||||||
|
- internal/storage/accounting_pull.go | 218 +++
|
||||||
|
- internal/storage/akten.go | 230 +++
|
||||||
|
- internal/storage/classification_templates.go | 453 +++++
|
||||||
|
- internal/storage/classification_templates_apply.go | 306 +++
|
||||||
|
- internal/storage/classification_templates_title.go | 198 ++
|
||||||
|
- internal/storage/compliance.go | 74 +
|
||||||
|
- internal/storage/custom_fields.go | 646 +++++++
|
||||||
|
- internal/storage/dashboard.go | 114 ++
|
||||||
|
- internal/storage/document_date.go | 163 ++
|
||||||
|
- internal/storage/document_notes.go | 132 ++
|
||||||
|
- internal/storage/documents.go | 588 ++++++
|
||||||
|
- internal/storage/index_sync.go | 217 +++
|
||||||
|
- internal/storage/metadata_suggestions.go | 320 ++++
|
||||||
|
- internal/storage/metadata_suggestions_naivebayes.go | 128 ++
|
||||||
|
- internal/storage/metadata_suggestions_ollama.go | 215 +++
|
||||||
|
- internal/storage/migrations/001_initial.sql | 60 +
|
||||||
|
- internal/storage/migrations/002_reminders.sql | 17 +
|
||||||
|
- internal/storage/migrations/003_documents_unique_hash.sql | 8 +
|
||||||
|
- internal/storage/migrations/004_sftp_credentials.sql | 17 +
|
||||||
|
- internal/storage/migrations/005_taxonomy.sql | 68 +
|
||||||
|
- internal/storage/migrations/006_custom_fields.sql | 44 +
|
||||||
|
- internal/storage/migrations/007_trash.sql | 36 +
|
||||||
|
- internal/storage/migrations/008_permissions.sql | 78 +
|
||||||
|
- internal/storage/migrations/009_shares.sql | 53 +
|
||||||
|
- internal/storage/migrations/010_search_index.sql | 42 +
|
||||||
|
- internal/storage/migrations/011_classification_templates.sql | 52 +
|
||||||
|
- internal/storage/migrations/012_metadata_suggestions.sql | 33 +
|
||||||
|
- internal/storage/migrations/012_workflows.sql | 57 +
|
||||||
|
- internal/storage/migrations/013_document_notes.sql | 21 +
|
||||||
|
- internal/storage/migrations/014_saved_views.sql | 25 +
|
||||||
|
- internal/storage/migrations/015_tenant_scan_title_format.sql | 20 +
|
||||||
|
- internal/storage/migrations/016_tenant_scan_title_prefix.sql | 19 +
|
||||||
|
- internal/storage/migrations/017_tenant_ollama_config.sql | 26 +
|
||||||
|
- internal/storage/migrations/018_akten.sql | 31 +
|
||||||
|
- internal/storage/migrations/019_document_date.sql | 23 +
|
||||||
|
- internal/storage/migrations/020_ml_classifier.sql | 55 +
|
||||||
|
- internal/storage/migrations/021_title_template.sql | 31 +
|
||||||
|
- internal/storage/migrations/022_retention_rules.sql | 53 +
|
||||||
|
- internal/storage/migrations/023_processing_jobs.sql | 59 +
|
||||||
|
- internal/storage/migrations/024_ocr_words.sql | 46 +
|
||||||
|
- internal/storage/migrations/025_document_date_score.sql | 31 +
|
||||||
|
- internal/storage/migrations/026_accounting_api_keys.sql | 46 +
|
||||||
|
- internal/storage/migrations/README.md | 42 +
|
||||||
|
- internal/storage/ml_classifier.go | 64 +
|
||||||
|
- internal/storage/ml_classifier_train.go | 59 +
|
||||||
|
- internal/storage/ocr_words.go | 160 ++
|
||||||
|
- internal/storage/ollama_config.go | 122 ++
|
||||||
|
- internal/storage/permissions.go | 721 ++++++++
|
||||||
|
- internal/storage/processing_jobs.go | 401 ++++
|
||||||
|
- internal/storage/reminders.go | 170 ++
|
||||||
|
- internal/storage/retention_rules.go | 533 ++++++
|
||||||
|
- internal/storage/saved_views.go | 158 ++
|
||||||
|
- internal/storage/search.go | 115 ++
|
||||||
|
- internal/storage/sftp_credentials.go | 155 ++
|
||||||
|
- internal/storage/shares.go | 389 ++++
|
||||||
|
- internal/storage/storage.go | 122 ++
|
||||||
|
- internal/storage/taxonomy.go | 387 ++++
|
||||||
|
- internal/storage/trash.go | 443 +++++
|
||||||
|
- internal/storage/workflows.go | 1028 +++++++++++
|
||||||
|
- internal/tenantstore/store.go | 217 +++
|
||||||
|
- internal/thumbnail/thumbnail.go | 160 ++
|
||||||
|
- internal/userstore/userstore.go | 442 +++++
|
||||||
|
- middleware.ts | 52 +
|
||||||
|
- next-env.d.ts | 5 +
|
||||||
|
- next.config.ts | 28 +
|
||||||
|
- package.json | 53 +
|
||||||
|
- postcss.config.mjs | 9 +
|
||||||
|
- src/app/(app)/akten/[id]/page.tsx | 53 +
|
||||||
|
- src/app/(app)/akten/page.tsx | 39 +
|
||||||
|
- src/app/(app)/documents/[id]/page.tsx | 75 +
|
||||||
|
- src/app/(app)/documents/loading.tsx | 15 +
|
||||||
|
- src/app/(app)/documents/page.tsx | 66 +
|
||||||
|
- src/app/(app)/layout.tsx | 31 +
|
||||||
|
- src/app/(app)/loading.tsx | 20 +
|
||||||
|
- src/app/(app)/page.tsx | 220 +++
|
||||||
|
- src/app/(app)/reminders/actions.ts | 25 +
|
||||||
|
- src/app/(app)/reminders/loading.tsx | 15 +
|
||||||
|
- src/app/(app)/reminders/page.tsx | 52 +
|
||||||
|
- src/app/(app)/scan/page.tsx | 16 +
|
||||||
|
- src/app/(app)/search/loading.tsx | 19 +
|
||||||
|
- src/app/(app)/search/page.tsx | 170 ++
|
||||||
|
- src/app/(app)/settings/accounting-keys/page.tsx | 51 +
|
||||||
|
- src/app/(app)/settings/classification-templates/page.tsx | 69 +
|
||||||
|
- src/app/(app)/settings/correspondents/page.tsx | 22 +
|
||||||
|
- src/app/(app)/settings/custom-fields/page.tsx | 50 +
|
||||||
|
- src/app/(app)/settings/document-types/page.tsx | 48 +
|
||||||
|
- src/app/(app)/settings/ollama-config/page.tsx | 66 +
|
||||||
|
- src/app/(app)/settings/page.tsx | 222 +++
|
||||||
|
- src/app/(app)/settings/permission-groups/page.tsx | 56 +
|
||||||
|
- src/app/(app)/settings/retention-rules/page.tsx | 73 +
|
||||||
|
- src/app/(app)/settings/shares/page.tsx | 52 +
|
||||||
|
- src/app/(app)/settings/tags/page.tsx | 42 +
|
||||||
|
- src/app/(app)/settings/tenant-settings/page.tsx | 66 +
|
||||||
|
- src/app/(app)/settings/tenants/page.tsx | 46 +
|
||||||
|
- src/app/(app)/settings/users/page.tsx | 60 +
|
||||||
|
- src/app/(app)/trash/loading.tsx | 15 +
|
||||||
|
- src/app/(app)/trash/page.tsx | 56 +
|
||||||
|
- src/app/globals.css | 69 +
|
||||||
|
- src/app/layout.tsx | 33 +
|
||||||
|
- src/app/login/page.tsx | 27 +
|
||||||
|
- src/app/public/share/[token]/page.tsx | 143 ++
|
||||||
|
- src/components/DEVLOG.md | 596 ++++++
|
||||||
|
- src/components/accounting/AccountingApiKeyManager.tsx | 322 ++++
|
||||||
|
- src/components/akten/AkteDetail.tsx | 361 ++++
|
||||||
|
- src/components/akten/AkteStatusBadge.tsx | 22 +
|
||||||
|
- src/components/akten/AktenTable.tsx | 383 ++++
|
||||||
|
- src/components/auth/LoginForm.tsx | 123 ++
|
||||||
|
- src/components/classification-templates/ApplyTemplateDialog.tsx | 354 ++++
|
||||||
|
- src/components/classification-templates/ClassificationTemplateManager.tsx | 670 +++++++
|
||||||
|
- src/components/compliance/ProcedureDocumentationDownload.tsx | 49 +
|
||||||
|
- src/components/custom-fields/CustomFieldManager.tsx | 455 +++++
|
||||||
|
- src/components/custom-fields/DocumentFieldsDialog.tsx | 314 ++++
|
||||||
|
- src/components/custom-fields/DocumentTypeFieldsSection.tsx | 215 +++
|
||||||
|
- src/components/documents/DocumentDetailsTab.tsx | 897 +++++++++
|
||||||
|
- src/components/documents/DocumentImagePreview.tsx | 301 +++
|
||||||
|
- src/components/documents/DocumentNotesTab.tsx | 166 ++
|
||||||
|
- src/components/documents/DocumentPreview.tsx | 460 +++++
|
||||||
|
- src/components/documents/DocumentUploadForm.tsx | 129 ++
|
||||||
|
- src/components/documents/DocumentsTable.tsx | 555 ++++++
|
||||||
|
- src/components/documents/DocumentsWidthLayout.tsx | 81 +
|
||||||
|
- src/components/documents/ProcessingStatusBadge.tsx | 182 ++
|
||||||
|
- src/components/permissions/DocumentGrantsDialog.tsx | 231 +++
|
||||||
|
- src/components/permissions/DocumentTypeGrantsSection.tsx | 38 +
|
||||||
|
- src/components/permissions/GrantsEditor.tsx | 234 +++
|
||||||
|
- src/components/permissions/PermissionGroupManager.tsx | 370 ++++
|
||||||
|
- src/components/permissions/TagGrantsSection.tsx | 36 +
|
||||||
|
- src/components/reminders/CreateReminderButton.tsx | 111 ++
|
||||||
|
- src/components/reminders/ReminderBadge.tsx | 25 +
|
||||||
|
- src/components/reminders/ReminderList.tsx | 65 +
|
||||||
|
- src/components/reminders/RemindersTable.tsx | 110 ++
|
||||||
|
- src/components/retention-rules/RetentionRuleManager.tsx | 493 +++++
|
||||||
|
- src/components/scan/ImageCropper.tsx | 298 +++
|
||||||
|
- src/components/scan/MobileScanCapture.tsx | 193 ++
|
||||||
|
- src/components/search/SavedViewsMenu.tsx | 188 ++
|
||||||
|
- src/components/search/SearchResults.tsx | 193 ++
|
||||||
|
- src/components/settings/OllamaConfigForm.tsx | 230 +++
|
||||||
|
- src/components/settings/ScanTitleDateFormatForm.tsx | 214 +++
|
||||||
|
- src/components/shares/ShareDialog.tsx | 263 +++
|
||||||
|
- src/components/shares/ShareStatusBadge.tsx | 42 +
|
||||||
|
- src/components/shares/TenantSharesManager.tsx | 100 +
|
||||||
|
- src/components/shell/AppSidebar.tsx | 87 +
|
||||||
|
- src/components/shell/CommandPalette.tsx | 81 +
|
||||||
|
- src/components/shell/SearchBar.tsx | 45 +
|
||||||
|
- src/components/shell/TopBar.tsx | 77 +
|
||||||
|
- src/components/taxonomy/TaxonomyManager.tsx | 254 +++
|
||||||
|
- src/components/tenants/TenantManager.tsx | 131 ++
|
||||||
|
- src/components/theme-provider.tsx | 16 +
|
||||||
|
- src/components/trash/TrashManager.tsx | 364 ++++
|
||||||
|
- src/components/ui/avatar.tsx | 50 +
|
||||||
|
- src/components/ui/badge.tsx | 36 +
|
||||||
|
- src/components/ui/button.tsx | 56 +
|
||||||
|
- src/components/ui/calendar.tsx | 70 +
|
||||||
|
- src/components/ui/card.tsx | 71 +
|
||||||
|
- src/components/ui/command.tsx | 163 ++
|
||||||
|
- src/components/ui/dialog.tsx | 122 ++
|
||||||
|
- src/components/ui/dropdown-menu.tsx | 194 ++
|
||||||
|
- src/components/ui/input.tsx | 22 +
|
||||||
|
- src/components/ui/label.tsx | 26 +
|
||||||
|
- src/components/ui/popover.tsx | 31 +
|
||||||
|
- src/components/ui/progress.tsx | 28 +
|
||||||
|
- src/components/ui/sheet.tsx | 137 ++
|
||||||
|
- src/components/ui/sidebar.tsx | 237 +++
|
||||||
|
- src/components/ui/skeleton.tsx | 15 +
|
||||||
|
- src/components/ui/sonner.tsx | 31 +
|
||||||
|
- src/components/ui/table.tsx | 117 ++
|
||||||
|
- src/components/ui/tabs.tsx | 55 +
|
||||||
|
- src/components/ui/textarea.tsx | 22 +
|
||||||
|
- src/components/users/UserManager.tsx | 485 +++++
|
||||||
|
- src/hooks/use-mobile.tsx | 19 +
|
||||||
|
- src/lib/api.ts | 1577 ++++++++++++++++
|
||||||
|
- src/lib/session.ts | 51 +
|
||||||
|
- src/lib/utils.ts | 6 +
|
||||||
|
- tailwind.config.ts | 77 +
|
||||||
|
- tsconfig.json | 41 +
|
||||||
|
- update.sh | 379 ++++
|
||||||
|
|
||||||
|
## 2026-08-11 22:22 – 22:22 (0m)
|
||||||
|
**Beschreibung:** Deploy FDN-08 (Korrelations-ID-Middleware, Metrics-Endpoint, Panic-Recovery) auf 192.168.1.204
|
||||||
|
**Projekt:** archivdms
|
||||||
|
|
||||||
|
### Ergebnis
|
||||||
|
- Build (Backend + Frontend via update.sh): grün
|
||||||
|
- Dienste archivdms + archivdms-web: aktiv
|
||||||
|
- /metrics (localhost:8080): HTTP 200, Prometheus-Textformat (archivdms_build_info, archivdms_uptime_seconds, archivdms_panics_total, archivdms_http_requests_total u.a.)
|
||||||
|
- api.metrics_allowed_ips in /etc/archivdms/config.yml: nicht gesetzt, läuft auf Loopback-Default — bei Bedarf Monitoring-IP ergänzen
|
||||||
|
- request_id im Log: Middleware erzeugt/propagiert request_id korrekt (observability.go), aber handleLogin nutzt reqLog(ctx) nicht — Testanfragen gegen /api/auth/login (401) erzeugten keine sichtbare Logzeile mit request_id. Kein Fix vorgenommen (nur Feststellung).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-08-11 (Deploy)
|
||||||
|
**Beschreibung:** Deploy FDN-08-Nachbesserung: metricsMiddleware loggt jetzt jede Anfrage (nicht nur 5xx, observability.go), Login/Logout/Me loggen über s.reqLog(ctx) mit request_id (auth_handlers.go) — schließt die oben notierte Lücke.
|
||||||
|
**Projekt:** archivdms
|
||||||
|
|
||||||
|
### Ergebnis
|
||||||
|
- Build (update.sh, Backend + Frontend): grün
|
||||||
|
- Dienste archivdms + archivdms-web: aktiv
|
||||||
|
- Verifikation: falscher Login-Versuch (POST /api/auth/login, username=nonexistent) erzeugte zwei Logzeilen mit identischer request_id `877f455ddd038d6a`:
|
||||||
|
- `msg="login failed" request_id=877f455ddd038d6a username=nonexistent reason=invalid_credentials`
|
||||||
|
- `msg="request rejected" request_id=877f455ddd038d6a method=POST route=/api/auth/login status=401`
|
||||||
|
- Korrelations-ID zwischen fachlichem Log und Access-Log funktioniert wie vorgesehen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|||||||
@@ -20,10 +20,67 @@ Konzeption basiert auf Recherche zu Paperless-ngx und ecoDMS (siehe `dms-feature
|
|||||||
|
|
||||||
Läuft bei jedem Push/PR; ein fehlschlagender Schritt bricht den jeweiligen Job ab (kein `continue-on-error`), das blockiert den Merge sobald in Gitea die Branch-Protection-Regel "Require status checks to pass" auf diese Jobs gesetzt ist. **Wichtig:** archivdms hat aktuell noch keinen Gitea-Remote (nur lokales `git init`), die Pipeline greift daher erst, sobald das Repo zu einer Gitea-Instanz mit aktivierten Actions und registriertem Runner gepusht wird.
|
Läuft bei jedem Push/PR; ein fehlschlagender Schritt bricht den jeweiligen Job ab (kein `continue-on-error`), das blockiert den Merge sobald in Gitea die Branch-Protection-Regel "Require status checks to pass" auf diese Jobs gesetzt ist. **Wichtig:** archivdms hat aktuell noch keinen Gitea-Remote (nur lokales `git init`), die Pipeline greift daher erst, sobald das Repo zu einer Gitea-Instanz mit aktivierten Actions und registriertem Runner gepusht wird.
|
||||||
|
|
||||||
|
## Betrieb: Logging, Metriken & Fehler-Tracking (FDN-08)
|
||||||
|
|
||||||
|
Jede HTTP-Anfrage läuft durch eine Basis-Middleware-Kette in `internal/api/observability.go`
|
||||||
|
(`requestID -> metrics -> recover -> ServeMux`, gebaut in `api.New`):
|
||||||
|
|
||||||
|
- **Korrelations-ID:** pro Anfrage wird eine Request-ID erzeugt oder ein mitgelieferter
|
||||||
|
`X-Request-ID`-Header übernommen (nur alphanumerisch/`-_.`, max. 64 Zeichen — sonst verworfen,
|
||||||
|
Schutz gegen Log-Injection). Die ID steht im `context.Context` und im Response-Header
|
||||||
|
`X-Request-ID`. Handler loggen über `s.reqLog(r.Context())` bzw. `s.reqLog(ctx)`; jede Zeile
|
||||||
|
bekommt dadurch automatisch `request_id=...` — auch in der asynchronen Verarbeitung
|
||||||
|
(`ProcessDocumentJob`, `ReprocessDocument`), soweit der Request-Context durchgereicht wird.
|
||||||
|
- **Panic-Recovery:** `net/http` hat kein zentrales Recovery. Die `recoverMiddleware` fängt jedes
|
||||||
|
Panic ab, loggt es mit Korrelations-ID und gekürztem Stacktrace, zählt `archivdms_panics_total`
|
||||||
|
hoch und antwortet mit einem sauberen HTTP 500 statt einer abgebrochenen Verbindung.
|
||||||
|
- **Keine Geheimnisse in Logs:** geloggt werden nur Methode, normalisierter Pfad, Status, Dauer und
|
||||||
|
Client-IP. Query-Strings, Header und Bodies werden nie ausgegeben (dort stehen Signed-URL-Signaturen,
|
||||||
|
Share-Tokens, Bearer-Keys, Passwörter). `normalizeRoute` ersetzt IDs durch `{id}` und das Segment
|
||||||
|
hinter `/share/` immer durch `{token}`.
|
||||||
|
|
||||||
|
### `GET /metrics`
|
||||||
|
|
||||||
|
Prometheus-Textformat, ohne Fremdabhängigkeit (`internal/api/metrics_handlers.go`, reines `fmt.Fprintf`).
|
||||||
|
Bewusst **ohne Login** (ein Scraper hat keine Session), dafür **IP-beschränkt**: Loopback ist immer
|
||||||
|
erlaubt, weitere Scraper werden über den Config-Key `api.metrics_allowed_ips` (Liste aus IPs oder
|
||||||
|
CIDR-Bereichen, Default leer = nur localhost) freigeschaltet; alles andere bekommt 403. Der Endpunkt
|
||||||
|
gehört nicht ins öffentliche Reverse-Proxy-Mapping.
|
||||||
|
|
||||||
|
Ausgegeben werden: `archivdms_http_requests_total{method,route,status}` (Fehlerrate = Anteil
|
||||||
|
`status=~"5.."`), `archivdms_http_request_duration_seconds_{bucket,sum,count}` (Latenz-Histogramm),
|
||||||
|
`archivdms_http_requests_in_flight`, `archivdms_goroutines`, `archivdms_uptime_seconds`,
|
||||||
|
`archivdms_panics_total`, `archivdms_build_info{version}` sowie die Queue-Länge
|
||||||
|
`archivdms_processing_jobs{status}` (aus `processing_jobs`, aggregiert über alle Mandanten — es
|
||||||
|
werden keine mandantenbezogenen Daten ausgegeben). Die Label-Kardinalität ist auf 500 Serien
|
||||||
|
gedeckelt, danach landet alles unter `route="/other"`.
|
||||||
|
|
||||||
## Frontend: Login-Pflicht & UI-Struktur
|
## Frontend: Login-Pflicht & UI-Struktur
|
||||||
|
|
||||||
Alle Seiten außer `/login` sind hinter einer Session-Cookie-Prüfung (`middleware.ts`, Cookie `archivdms_session`). Ohne gültiges Cookie wird sofort (ohne sichtbares Flackern) zu `/login` umgeleitet; die vollständige JWT-Prüfung bleibt Aufgabe des Go-Backends bei jedem echten API-Call. Eingeloggte Bereiche laufen unter der Routegruppe `src/app/(app)/` mit gemeinsamer App-Shell (Sidebar, TopBar, Cmd+K-Befehlspalette, Dark Mode). `/documents` und `/reminders` sind Server Components mit serverseitigem Datenfetch (`src/lib/session.ts` reicht das Session-Cookie manuell an die Go-API weiter) statt der bisherigen "use client" + `useEffect`-Ladeschleife. Die Dokumenten-Detail-/Vorschauseite `/documents/{id}` (Server Component `src/app/(app)/documents/[id]/page.tsx` + Client-Island `DocumentPreview`) zeigt die Datei-Vorschau in einem `<iframe src="/api/documents/{id}/file">` (same-origin, Cookie-Auth greift automatisch, Browser-natives PDF/Bild-Rendering). Die Metadaten-Seitenleiste (Reiter Details/Inhalt/Verlauf) ist in die Unterkomponente `DocumentDetailsTab` ausgelagert und in einem kompakten 2-Spalten-Raster angeordnet (kurze Felder Belegdatum/Dokumenttyp/Korrespondent/Erstellt nebeneinander, Titel/Akte/Tags in voller Breite). Editierbar: Inline-Titel, Belegdatum (`<input type="date">`, `PUT /api/documents/{id}/document-date`), Dokumenttyp/Korrespondent/Akte (Command-Popover, Auswahl wird direkt beim Klick gespeichert) sowie Tags hinzufügen (`POST /api/documents/{id}/tags/{tagId}`) / entfernen (X am Badge, `DELETE ...`). Heuristische Vorschlags-Chips (inkl. erkanntem Belegdatum aus dem OCR-Text) lassen sich per Klick übernehmen. In der Kopfzeile der Detailseite liegen die Aktionen "Export" (Download-Icon, `<a href="/api/documents/{id}/export" download>` — ZIP mit Originaldatei, `metadata.json` und `ocr_text.txt`, same-origin per Session-Cookie, kein Blob-Umweg) und "Neu verarbeiten". Erreichbar über den "Vorschau"-Button bzw. das Auge-Icon in der Dokumentenliste.
|
Alle Seiten außer `/login` sind hinter einer Session-Cookie-Prüfung (`middleware.ts`, Cookie `archivdms_session`). Ohne gültiges Cookie wird sofort (ohne sichtbares Flackern) zu `/login` umgeleitet; die vollständige JWT-Prüfung bleibt Aufgabe des Go-Backends bei jedem echten API-Call. Eingeloggte Bereiche laufen unter der Routegruppe `src/app/(app)/` mit gemeinsamer App-Shell (Sidebar, TopBar, Cmd+K-Befehlspalette, Dark Mode). `/documents` und `/reminders` sind Server Components mit serverseitigem Datenfetch (`src/lib/session.ts` reicht das Session-Cookie manuell an die Go-API weiter) statt der bisherigen "use client" + `useEffect`-Ladeschleife. Die Dokumenten-Detail-/Vorschauseite `/documents/{id}` (Server Component `src/app/(app)/documents/[id]/page.tsx` + Client-Island `DocumentPreview`) zeigt die Datei-Vorschau in einem `<iframe src="/api/documents/{id}/file">` (same-origin, Cookie-Auth greift automatisch, Browser-natives PDF/Bild-Rendering). Die Metadaten-Seitenleiste (Reiter Details/Inhalt/Verlauf) ist in die Unterkomponente `DocumentDetailsTab` ausgelagert und in einem kompakten 2-Spalten-Raster angeordnet (kurze Felder Belegdatum/Dokumenttyp/Korrespondent/Erstellt nebeneinander, Titel/Akte/Tags in voller Breite). Editierbar: Inline-Titel, Belegdatum (`<input type="date">`, `PUT /api/documents/{id}/document-date`), Dokumenttyp/Korrespondent/Akte (Command-Popover, Auswahl wird direkt beim Klick gespeichert) sowie Tags hinzufügen (`POST /api/documents/{id}/tags/{tagId}`) / entfernen (X am Badge, `DELETE ...`). Heuristische Vorschlags-Chips (inkl. erkanntem Belegdatum aus dem OCR-Text) lassen sich per Klick übernehmen. In der Kopfzeile der Detailseite liegen die Aktionen "Export" (Download-Icon, `<a href="/api/documents/{id}/export" download>` — ZIP mit Originaldatei, `metadata.json` und `ocr_text.txt`, same-origin per Session-Cookie, kein Blob-Umweg) und "Neu verarbeiten". Erreichbar über den "Vorschau"-Button bzw. das Auge-Icon in der Dokumentenliste.
|
||||||
|
|
||||||
|
### Basis-Komponenten & Design-Tokens (FDN-06)
|
||||||
|
|
||||||
|
Alle wiederverwendbaren UI-Bausteine liegen ausschließlich unter `src/components/ui/` (shadcn/ui-Stand, Radix-basiert). Neue Features benutzen diese Komponenten, statt eigene Varianten zu bauen; Anpassungen erfolgen in der Datei selbst (nicht per Kopie), Dateinamen werden nie umbenannt.
|
||||||
|
|
||||||
|
| Komponente | Datei | Einsatzzweck |
|
||||||
|
|---|---|---|
|
||||||
|
| Table | `ui/table.tsx` | Standard-Listenansicht (Dokumente, Wiedervorlage, Admin-Tabellen). Wrapper hat `overflow-auto` → horizontal scrollbar auf schmalen Viewports |
|
||||||
|
| Dialog / Sheet | `ui/dialog.tsx`, `ui/sheet.tsx` | Modale Formulare bzw. mobile Drawer (Sidebar auf < md) |
|
||||||
|
| Formularelemente | `ui/input.tsx`, `ui/textarea.tsx`, `ui/label.tsx`, `ui/calendar.tsx` | Eingaben; Datumsauswahl über Calendar + Popover |
|
||||||
|
| Button | `ui/button.tsx` | Varianten (`default`/`ghost`/`outline`/`destructive`) + Größen über `cva` |
|
||||||
|
| Badge | `ui/badge.tsx` | Status-Anzeigen (Wiedervorlage, Verarbeitungsstatus, Share-Status) |
|
||||||
|
| Card | `ui/card.tsx` | Dashboard-Kacheln, Sektionen in Einstellungen |
|
||||||
|
| Tabs | `ui/tabs.tsx` | Reiter in der Dokument-Detailansicht (Details/Inhalt/Verlauf) |
|
||||||
|
| DropdownMenu / Popover / Command | `ui/dropdown-menu.tsx`, `ui/popover.tsx`, `ui/command.tsx` | Benutzermenü, Auswahl-Popover, Cmd+K-Befehlspalette |
|
||||||
|
| Toast | `ui/sonner.tsx` | Rückmeldungen nach Mutationen (`toast()` aus `sonner`); `<Toaster />` global in `src/app/layout.tsx` |
|
||||||
|
| Sidebar | `ui/sidebar.tsx` | Shell-Navigation inkl. Collapse-State und Mobile-Drawer |
|
||||||
|
| Weitere | `ui/avatar.tsx`, `ui/progress.tsx`, `ui/skeleton.tsx` | Benutzer-Avatar, Upload-Fortschritt, Skeleton-Loading in `loading.tsx` |
|
||||||
|
|
||||||
|
Die App-Shell selbst liegt in `src/components/shell/` (`AppSidebar`, `TopBar`, `SearchBar`, `CommandPalette`) und wird von `src/app/(app)/layout.tsx` zusammengesetzt (`SidebarProvider` → `AppSidebar` + `SidebarInset` → `TopBar` + Inhaltsbereich).
|
||||||
|
|
||||||
|
**Design-Tokens** sind zentral gepflegt: `src/app/globals.css` definiert die HSL-Werte je Theme (`:root` = hell, `.dark` = dunkel) für `background/foreground`, `card`, `popover`, `primary`, `secondary`, `muted`, `accent`, `destructive`, `border`, `input`, `ring`, die `sidebar-*`-Familie sowie `--radius`. `tailwind.config.ts` mappt sie auf Tailwind-Utilities (`bg-background`, `text-muted-foreground`, `border-border`, `rounded-lg` …). Abstände und Typografie kommen unverändert aus der Tailwind-Standardskala — bewusst kein eigener Satz, um Sonderwege zu vermeiden. Regel: keine Hex-/RGB-Literale und keine Inline-Styles für Farben in Komponenten. Einzige Ausnahme sind semantische Statusfarben (emerald/amber/red) an Badges und Farbbalken; Inline-`style` ist nur für berechnete Geometrie erlaubt (Progress-Balken, Crop-Rechtecke, OCR-Overlay-Boxen).
|
||||||
|
|
||||||
## Projektstatus
|
## Projektstatus
|
||||||
|
|
||||||
Scaffold + erstes fachliches Feature (Wiedervorlage/Reminder, PROJ-1) als Code geschrieben, **noch nicht gebaut/getestet** (kein Go/Node-Toolchain in dieser Umgebung ausgeführt). Vor Inbetriebnahme: `go build ./...`, `npm install && npm run build`, Migrationen gegen echte PostgreSQL-Instanz prüfen.
|
Scaffold + erstes fachliches Feature (Wiedervorlage/Reminder, PROJ-1) als Code geschrieben, **noch nicht gebaut/getestet** (kein Go/Node-Toolchain in dieser Umgebung ausgeführt). Vor Inbetriebnahme: `go build ./...`, `npm install && npm run build`, Migrationen gegen echte PostgreSQL-Instanz prüfen.
|
||||||
@@ -73,6 +130,12 @@ Ablauf beim Upload (`POST /api/documents/upload`, multipart, Feld `file` + Pflic
|
|||||||
|
|
||||||
Weil OCR erst nachgelagert läuft, richtet sich der Archivordner `<yyyy>/<mm>` nach dem Upload-Zeitpunkt; das erkannte Belegdatum landet danach in `documents.document_date`, die archivierte Datei wird dabei nie verschoben (WORM).
|
Weil OCR erst nachgelagert läuft, richtet sich der Archivordner `<yyyy>/<mm>` nach dem Upload-Zeitpunkt; das erkannte Belegdatum landet danach in `documents.document_date`, die archivierte Datei wird dabei nie verschoben (WORM).
|
||||||
|
|
||||||
|
### Storage-Abstraktion & signierte Download-Links (FDN-03)
|
||||||
|
|
||||||
|
Der Dateizugriff liegt hinter dem Interface `objectstore.Store` (`internal/objectstore`): `Archive` (Scratch-Datei → WORM-Ablage inkl. `chmod 0440`), `Open`, `Stat`, `Delete`, `SignedURL`, `VerifySignedURL`. Einzige Implementierung ist `LocalStore` (lokales Dateisystem) — **bewusst kein S3-Treiber**, weil die WORM-/GoBD-Garantie an POSIX-Rechten (0440) hängt. Das oben beschriebene Pfadschema bleibt unverändert; zusätzlich prüfen `Open`/`Stat`/`Delete` jetzt, dass der übergebene `storage_path` tatsächlich unter `store/<tenant_id>/` liegt (Mandanten-Containment gegen Pfad-Traversal/IDOR). Verdrahtet wird der Treiber in `Server.SetStorageConfig` (Signierschlüssel per HKDF-SHA256 aus `api.secret`, Link-Basis aus `server.fqdn`).
|
||||||
|
|
||||||
|
**Signierte, zeitlich begrenzte Download-URLs:** `POST /api/documents/{id}/signed-url` (authentifiziert, mandantengeprüft, Body optional `{"ttl_minutes": 15}`) liefert `{url, expires_at}`. Eingelöst wird der Link unter `GET /public/files?t=&d=&exp=&sig=` — ohne Session, die HMAC-SHA256-Signatur über `tenant|dokument|ablauf` ist das Credential. Gleiche Schutzmechanik wie die Share-Links (harter Pflicht-Ablauf, per-IP-Rate-Limit, Audit-Trail `signed_url_created`/`signed_url_accessed` inkl. Fehlschlägen), aber zustandslos, ohne DB-Zeile und ohne Widerruf — für kurzlebigen Maschinenzugriff. Für Weitergabe an Dritte mit eigenem Lebenszyklus (Widerruf, Passwort, Zugriffslimit) bleiben die Share-Links das Mittel der Wahl. Abgelaufener Link → HTTP 410, gefälschter → 403, fehlende Datei → 404. Gültigkeit: `storage.signed_url_ttl_minutes` (Default 15), pro Anfrage überschreibbar, hart gedeckelt auf 24 h.
|
||||||
|
|
||||||
### Trennseiten-Split (Barcode-Trennblätter)
|
### Trennseiten-Split (Barcode-Trennblätter)
|
||||||
|
|
||||||
`internal/pagesplit` zerlegt mehrseitige **PDF**-Scans beim Ingest an Barcode-Trennblättern in Einzeldokumente (Vorbild: Paperless-ngx, an die archivdms-Pipeline angepasst). Ablauf im synchronen Staging-Schritt, **bevor** irgendetwas archiviert wird (Schritt 1a in `storeUploadedFile`): `pdfinfo` liefert die Seitenzahl, `pdftoppm -r 150` rastert jede Seite einmal, `zbarimg` (via `internal/barcode`) dekodiert die Barcodes; Seiten mit dem konfigurierten Marker (`pagesplit.marker`, Default `ARCHIVDMS-SPLIT`, Vergleich case-insensitiv, optional Präfix-Match) gelten als Trennblatt. Die Segmente dazwischen werden mit `pdfseparate` + `pdfunite` (poppler-utils, kein qpdf/pdftk) zu je einem Teil-PDF zusammengesetzt; **das Trennblatt selbst wird verworfen**. Jedes Teildokument durchläuft danach exakt denselben Pfad wie ein normaler Einzel-Upload (eigener Hash + Duplikatprüfung, eigene WORM-Ablage mit `chmod 0440`, eigener Verarbeitungsjob mit OCR/Taxonomie/Workflows).
|
`internal/pagesplit` zerlegt mehrseitige **PDF**-Scans beim Ingest an Barcode-Trennblättern in Einzeldokumente (Vorbild: Paperless-ngx, an die archivdms-Pipeline angepasst). Ablauf im synchronen Staging-Schritt, **bevor** irgendetwas archiviert wird (Schritt 1a in `storeUploadedFile`): `pdfinfo` liefert die Seitenzahl, `pdftoppm -r 150` rastert jede Seite einmal, `zbarimg` (via `internal/barcode`) dekodiert die Barcodes; Seiten mit dem konfigurierten Marker (`pagesplit.marker`, Default `ARCHIVDMS-SPLIT`, Vergleich case-insensitiv, optional Präfix-Match) gelten als Trennblatt. Die Segmente dazwischen werden mit `pdfseparate` + `pdfunite` (poppler-utils, kein qpdf/pdftk) zu je einem Teil-PDF zusammengesetzt; **das Trennblatt selbst wird verworfen**. Jedes Teildokument durchläuft danach exakt denselben Pfad wie ein normaler Einzel-Upload (eigener Hash + Duplikatprüfung, eigene WORM-Ablage mit `chmod 0440`, eigener Verarbeitungsjob mit OCR/Taxonomie/Workflows).
|
||||||
@@ -96,6 +159,7 @@ storage:
|
|||||||
base_path: "/var/lib/archivdms" # enthält inbox/, store/, ocr-tmp/
|
base_path: "/var/lib/archivdms" # enthält inbox/, store/, ocr-tmp/
|
||||||
retention_days: 3650
|
retention_days: 3650
|
||||||
max_upload_size_mb: 50
|
max_upload_size_mb: 50
|
||||||
|
signed_url_ttl_minutes: 15 # Default-Gültigkeit signierter Download-Links
|
||||||
|
|
||||||
ocr:
|
ocr:
|
||||||
tesseract_path: "tesseract"
|
tesseract_path: "tesseract"
|
||||||
@@ -124,11 +188,31 @@ index:
|
|||||||
manticore_dsn: "" # z.B. "archivdms@tcp(127.0.0.1:9306)/?charset=utf8mb4"
|
manticore_dsn: "" # z.B. "archivdms@tcp(127.0.0.1:9306)/?charset=utf8mb4"
|
||||||
```
|
```
|
||||||
|
|
||||||
### Volltext-Index (Manticore, Phase 1)
|
### Volltext-Index (Manticore)
|
||||||
|
|
||||||
Optionaler sekundärer Volltext-Index (`internal/index`) über Manticore Search (MySQL-Protokoll, Port 9306, `github.com/go-sql-driver/mysql`, CGO-frei). **PostgreSQL bleibt Source of Truth** — der Index wird nur best-effort synchron gehalten: jeder Sync-Fehler wird geloggt, blockiert aber nie den auslösenden Request. Ist `index.manticore_dsn` leer, ist der Indexer `nil` und alle Sync-Aufrufe sind No-ops.
|
Sekundärer Volltext-Index (`internal/index`) über Manticore Search (MySQL-Protokoll, Port 9306, nur `127.0.0.1` gebunden, `github.com/go-sql-driver/mysql`, CGO-frei). **PostgreSQL bleibt Source of Truth** — der Index wird nur best-effort synchron gehalten: jeder Sync-Fehler wird geloggt, blockiert aber nie den auslösenden Request. Ist `index.manticore_dsn` leer, ist der Indexer `nil` und alle Sync-/Suchaufrufe sind No-ops (reproduzierbarer Start ohne manuelle Zusatzschritte: DSN in `/etc/archivdms/config.yml` setzen, Backend/Systemd-Dienst startet den Rest selbst — RT-Tabellen werden lazy beim ersten Tenant-Zugriff angelegt).
|
||||||
|
|
||||||
Pro Mandant existiert eine RT-Tabelle `documents_tenant_<tenant_id>` (Tabellenname gegen Injection validiert). Synchronisiert wird nach Upload/Create, ACL-/Tag-/Dokumenttyp-Änderung (RecomputeVisibility), Korrespondent- und Custom-Field-Änderung; beim Verschieben in den Papierkorb sowie bei finaler Löschung (Vier-Augen-bestätigt) wird der Eintrag aus dem Index entfernt (GoBD: endgültig gelöschte Dokumente dürfen nicht mehr auffindbar sein), beim Wiederherstellen neu indexiert. Der Such-Endpunkt und ein Reindex-CLI folgen in Phase 2/3.
|
Pro Mandant existiert eine eigene RT-Tabelle `documents_tenant_<tenant_id>` (Tabellenname gegen Injection validiert, Mandantentrennung über getrennte Tabellen statt Row-Filter). Synchronisiert wird nach Upload/Create, ACL-/Tag-/Dokumenttyp-Änderung (RecomputeVisibility), Korrespondent- und Custom-Field-Änderung; beim Verschieben in den Papierkorb sowie bei finaler Löschung (Vier-Augen-bestätigt) wird der Eintrag aus dem Index entfernt (GoBD: endgültig gelöschte Dokumente dürfen nicht mehr auffindbar sein), beim Wiederherstellen neu indexiert. Ein manueller Neuaufbau des kompletten Index aus dem Postgres-Bestand ist jederzeit über `archivdms reindex [-tenant N]` möglich (streamt in Batches von 500 Dokumenten, bricht laut ohne exit(1) ab statt einen fehlenden `manticore_dsn` als stillen No-op zu behandeln). Such-Endpunkt: `GET /api/documents/search` (ACL-gefiltert; Manticore liefert nur IDs+Score, die vollständigen Dokumentzeilen kommen aus Postgres).
|
||||||
|
|
||||||
|
**Index-Schema** (`internal/index/manticore.go`, `ensureTable`):
|
||||||
|
|
||||||
|
| Feld | Typ | Zweck |
|
||||||
|
|---|---|---|
|
||||||
|
| `doc_id` | string | Anzeige-ID (String-Form von `id`) |
|
||||||
|
| `title` | text | Volltext-durchsucht |
|
||||||
|
| `doc_type` | text | Volltext-durchsucht (Legacy-Freitext) |
|
||||||
|
| `correspondent` | text | Volltext-durchsucht (Legacy-Freitext) |
|
||||||
|
| `ocr_text` | text | Volltext-durchsucht, OCR-Ergebnis |
|
||||||
|
| `tags` | text | Volltext-durchsucht, Tag-Namen als Leerzeichen-separierter String |
|
||||||
|
| `tag_ids` | multi (MVA) | Attribut-Filter (Tag-Facette) |
|
||||||
|
| `doc_type_id` / `correspondent_id` | bigint | Attribut-Filter (Taxonomie-IDs) |
|
||||||
|
| `acl_group_ids` | multi (MVA) | Attribut-Filter für ACL (Sichtbarkeitsgruppen) |
|
||||||
|
| `retain_until_ts` / `created_ts` / `updated_ts` | bigint (Unix) | Zeitstempel-Attribute |
|
||||||
|
| `deleted` | uint | Soft-Delete-Flag, `Search` filtert immer `deleted = 0` |
|
||||||
|
|
||||||
|
**Gewichtung**: Die Volltext-Suche matcht gleichgewichtet über `@(title,ocr_text,tags,correspondent,doc_type)` (kein `field_weights`-Boost) — Ranking erfolgt rein über Manticores BM25 (`WEIGHT()`), sortiert nach `WEIGHT() DESC, created_ts DESC`. Bewusst MVP: keine Feldgewichtung, um keine Suchsyntax-Erwartungshaltung/Tuning-Aufwand vor dem ersten echten Nutzungs-Feedback aufzubauen. Eine spätere Gewichtung (z. B. `title` stärker als `ocr_text`) ist eine reine Query-Änderung in `Search`, kein Schema-Umbau.
|
||||||
|
|
||||||
|
**Deutsche Sprachbehandlung**: Die Tabelle wird mit `morphology='lemmatize_de_all,stem_en'` angelegt — deutsche Lemmatisierung (u. a. Kompositazerlegung, Umlaut-Normalisierung über Manticores eingebautes de-Wörterbuch) plus englisches Stemming für Mischtexte. Keine explizite Stoppwortliste konfiguriert (Manticore filtert ohne `stopwords=`-Option keine Füllwörter heraus) — bei BM25-Ranking wirkt sich das nur moderat auf die Relevanz aus, da seltene Terme ohnehin höher gewichtet werden; bei Bedarf lässt sich eine deutsche Stoppwortliste (`stopwords = de`) nachrüsten, das erfordert danach einen vollen Reindex (`archivdms reindex`).
|
||||||
|
|
||||||
### OCR-Wortkoordinaten & Vorschau-Overlay
|
### OCR-Wortkoordinaten & Vorschau-Overlay
|
||||||
|
|
||||||
@@ -211,10 +295,29 @@ bash update.sh
|
|||||||
|
|
||||||
Route `/scan` (in der Sidebar als "Beleg erfassen") öffnet auf mobilen Browsern direkt die Rückkamera (`<input type="file" capture="environment">`), kein natives App nötig. Ein Foto genügt, Titel wird automatisch aus Datum/Uhrzeit gesetzt, Upload läuft über dieselbe Pipeline wie `/documents` (Hash, WORM-Ablage, OCR, automatisches Tag-/Korrespondent-Matching). Dokumenttyp/Korrespondent lassen sich danach unter `/documents` nachträglich ergänzen.
|
Route `/scan` (in der Sidebar als "Beleg erfassen") öffnet auf mobilen Browsern direkt die Rückkamera (`<input type="file" capture="environment">`), kein natives App nötig. Ein Foto genügt, Titel wird automatisch aus Datum/Uhrzeit gesetzt, Upload läuft über dieselbe Pipeline wie `/documents` (Hash, WORM-Ablage, OCR, automatisches Tag-/Korrespondent-Matching). Dokumenttyp/Korrespondent lassen sich danach unter `/documents` nachträglich ergänzen.
|
||||||
|
|
||||||
|
## Migrationen & Entwicklungs-Seed (FDN-02)
|
||||||
|
|
||||||
|
Schemaänderungen laufen weiterhin über die idempotenten `initSchema()`-Funktionen
|
||||||
|
der Store-Pakete (kein Migrationstool). Ergänzend gilt: zu jeder neuen
|
||||||
|
`internal/storage/migrations/NNN_name.sql` gehört eine `NNN_name.down.sql` mit
|
||||||
|
reviewtem Rückbau-SQL (manuell per `psql -f` auszuführen, nie automatisch).
|
||||||
|
Details und Anforderungen: `internal/storage/migrations/README.md`.
|
||||||
|
|
||||||
|
Für eine frische lokale Datenbank legt
|
||||||
|
|
||||||
|
```
|
||||||
|
archivdms seed dev [-config PATH] [-name NAME] [-slug SLUG] [-email EMAIL] [-role ROLE] [-reset-password]
|
||||||
|
```
|
||||||
|
|
||||||
|
einen Test-Mandanten ("Testfirma") und einen tenant-gebundenen Test-Benutzer
|
||||||
|
(`testuser@testfirma.local`, Rolle `domain_admin`) an. Der Befehl ist idempotent
|
||||||
|
(Mandant über Slug, Benutzer über E-Mail); das Passwort wird zufällig erzeugt und
|
||||||
|
einmalig auf der Konsole ausgegeben — es steht nirgends im Code oder in der Config.
|
||||||
|
|
||||||
## Struktur
|
## Struktur
|
||||||
|
|
||||||
```
|
```
|
||||||
cmd/archivdms/ CLI-Einstiegspunkt (serve, reminders notify)
|
cmd/archivdms/ CLI-Einstiegspunkt (serve, reminders notify, seed dev)
|
||||||
config/ YAML-Konfiguration
|
config/ YAML-Konfiguration
|
||||||
internal/api/ HTTP-Handler (auth, user, audit, document, reminder, sftp-credentials)
|
internal/api/ HTTP-Handler (auth, user, audit, document, reminder, sftp-credentials)
|
||||||
internal/audit/ Append-only Audit-Log (GoBD-Nachvollziehbarkeit)
|
internal/audit/ Append-only Audit-Log (GoBD-Nachvollziehbarkeit)
|
||||||
|
|||||||
@@ -0,0 +1,253 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"archivdms/config"
|
||||||
|
"archivdms/internal/audit"
|
||||||
|
"archivdms/internal/tenantstore"
|
||||||
|
"archivdms/internal/userstore"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Defaults for the development seed (FDN-02). Deliberately NO password
|
||||||
|
// constant here: the seed password is generated per run (crypto/rand via
|
||||||
|
// randomPassword) and printed exactly once, mirroring seedDefaultUsers.
|
||||||
|
const (
|
||||||
|
seedTenantName = "Testfirma"
|
||||||
|
seedTenantSlug = "testfirma"
|
||||||
|
seedUserName = "testuser"
|
||||||
|
seedUserEmail = "testuser@testfirma.local"
|
||||||
|
seedDomainToken = "" // no mail domain by default — domain login stays off
|
||||||
|
)
|
||||||
|
|
||||||
|
// runSeed dispatches `archivdms seed <subcommand>`.
|
||||||
|
//
|
||||||
|
// Usage: archivdms seed dev [-config PATH] [-name N] [-slug S] [-email E]
|
||||||
|
//
|
||||||
|
// [-role R] [-reset-password]
|
||||||
|
func runSeed(args []string) {
|
||||||
|
if len(args) == 0 || args[0] != "dev" {
|
||||||
|
fmt.Println("usage: archivdms seed dev [-config PATH] [-name NAME] [-slug SLUG] [-email EMAIL] [-role ROLE] [-reset-password]")
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
runSeedDev(args[1:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// runSeedDev creates a development test tenant plus one tenant-bound user so a
|
||||||
|
// fresh local/staging database is usable without clicking through the admin UI.
|
||||||
|
//
|
||||||
|
// Idempotent by design: an existing tenant (matched by slug) and an existing
|
||||||
|
// user (matched by e-mail) are reused, never duplicated and never silently
|
||||||
|
// overwritten. Only with -reset-password is an existing user's password
|
||||||
|
// replaced by a freshly generated one.
|
||||||
|
//
|
||||||
|
// The generated password is printed to the console exactly once and is never
|
||||||
|
// persisted in plaintext (userstore.Create bcrypt-hashes it, cost 12) — same
|
||||||
|
// convention as the first-start superadmin seed in main.go. Nothing about this
|
||||||
|
// command is safe to run against production data, hence the explicit warning.
|
||||||
|
func runSeedDev(args []string) {
|
||||||
|
fs := flag.NewFlagSet("seed dev", flag.ExitOnError)
|
||||||
|
configPath := fs.String("config", "/etc/archivdms/config.yml", "path to config file")
|
||||||
|
tenantName := fs.String("name", seedTenantName, "Name des Test-Mandanten")
|
||||||
|
tenantSlug := fs.String("slug", seedTenantSlug, "Slug des Test-Mandanten")
|
||||||
|
email := fs.String("email", seedUserEmail, "E-Mail des Test-Benutzers (Login-Kennung)")
|
||||||
|
role := fs.String("role", userstore.RoleDomainAdmin, "Rolle des Test-Benutzers (user|domain_admin)")
|
||||||
|
resetPassword := fs.Bool("reset-password", false, "Passwort eines bereits vorhandenen Test-Benutzers neu setzen")
|
||||||
|
fs.Parse(args)
|
||||||
|
|
||||||
|
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||||
|
|
||||||
|
switch *role {
|
||||||
|
case userstore.RoleUser, userstore.RoleDomainAdmin:
|
||||||
|
default:
|
||||||
|
logger.Error("ungültige Rolle — erlaubt sind user oder domain_admin", "role", *role)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := config.Load(*configPath)
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("failed to load config", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
tenantSt, err := tenantstore.New(cfg.Database.DSN())
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("tenant store init failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer tenantSt.Close()
|
||||||
|
|
||||||
|
users, err := userstore.New(cfg.Database.DSN())
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("userstore init failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer users.Close()
|
||||||
|
|
||||||
|
audlog, err := audit.New(cfg.Database.DSN(), cfg.Audit.ResolvedLogPath(), logger)
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("audit init failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer audlog.Close()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// --- Tenant -------------------------------------------------------
|
||||||
|
tenant, err := findTenantBySlug(ctx, tenantSt, *tenantSlug)
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("list tenants failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
tenantCreated := false
|
||||||
|
if tenant == nil {
|
||||||
|
tenant, err = tenantSt.Create(ctx, *tenantName, *tenantSlug, seedDomainToken)
|
||||||
|
if err != nil {
|
||||||
|
audlog.Log(audit.Entry{
|
||||||
|
EventType: "seed_dev",
|
||||||
|
Username: "cli",
|
||||||
|
Success: false,
|
||||||
|
Detail: fmt.Sprintf("Mandant %q konnte nicht angelegt werden: %v", *tenantSlug, err),
|
||||||
|
})
|
||||||
|
logger.Error("create tenant failed", "slug", *tenantSlug, "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
tenantCreated = true
|
||||||
|
audlog.Log(audit.Entry{
|
||||||
|
EventType: "seed_dev",
|
||||||
|
Username: "cli",
|
||||||
|
Success: true,
|
||||||
|
TenantID: &tenant.ID,
|
||||||
|
Detail: fmt.Sprintf("Test-Mandant angelegt: %s (slug=%s)", tenant.Name, tenant.Slug),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- User ---------------------------------------------------------
|
||||||
|
// GetByEmail returns an error (not a sentinel) when nothing matches, so a
|
||||||
|
// nil user is treated as "does not exist yet".
|
||||||
|
existing, _ := users.GetByEmail(ctx, *email)
|
||||||
|
|
||||||
|
password, err := randomPassword()
|
||||||
|
if err != nil {
|
||||||
|
logger.Error("generate password failed", "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
var userCreated, passwordShown bool
|
||||||
|
switch {
|
||||||
|
case existing == nil:
|
||||||
|
username := usernameFromEmail(*email)
|
||||||
|
created, err := users.Create(userstore.CreateUserRequest{
|
||||||
|
Username: username,
|
||||||
|
Email: *email,
|
||||||
|
Password: password,
|
||||||
|
Role: *role,
|
||||||
|
TenantID: &tenant.ID,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
audlog.Log(audit.Entry{
|
||||||
|
EventType: "seed_dev",
|
||||||
|
Username: "cli",
|
||||||
|
Success: false,
|
||||||
|
TenantID: &tenant.ID,
|
||||||
|
Detail: fmt.Sprintf("Test-Benutzer %q konnte nicht angelegt werden: %v", *email, err),
|
||||||
|
})
|
||||||
|
logger.Error("create user failed", "email", *email, "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
existing = created
|
||||||
|
userCreated, passwordShown = true, true
|
||||||
|
audlog.Log(audit.Entry{
|
||||||
|
EventType: "seed_dev",
|
||||||
|
Username: "cli",
|
||||||
|
Success: true,
|
||||||
|
TenantID: &tenant.ID,
|
||||||
|
Detail: fmt.Sprintf("Test-Benutzer angelegt: %s (rolle=%s, mandant=%d)", created.Email, created.Role, tenant.ID),
|
||||||
|
})
|
||||||
|
|
||||||
|
case *resetPassword:
|
||||||
|
if _, err := users.Update(existing.ID, userstore.UpdateUserRequest{Password: &password}); err != nil {
|
||||||
|
audlog.Log(audit.Entry{
|
||||||
|
EventType: "seed_dev",
|
||||||
|
Username: "cli",
|
||||||
|
Success: false,
|
||||||
|
TenantID: &tenant.ID,
|
||||||
|
Detail: fmt.Sprintf("Passwort-Reset für %q fehlgeschlagen: %v", *email, err),
|
||||||
|
})
|
||||||
|
logger.Error("reset password failed", "email", *email, "err", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
passwordShown = true
|
||||||
|
audlog.Log(audit.Entry{
|
||||||
|
EventType: "seed_dev",
|
||||||
|
Username: "cli",
|
||||||
|
Success: true,
|
||||||
|
TenantID: &tenant.ID,
|
||||||
|
Detail: fmt.Sprintf("Passwort des Test-Benutzers %s neu gesetzt", existing.Email),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Ausgabe ------------------------------------------------------
|
||||||
|
fmt.Println()
|
||||||
|
fmt.Println("╔══════════════════════════════════════════════════════════════╗")
|
||||||
|
fmt.Println("║ ARCHIVDMS — ENTWICKLUNGS-SEED (NICHT PRODUKTIV) ║")
|
||||||
|
fmt.Printf("║ Mandant : %-24s %-22s ║\n", tenant.Name, statusWord(tenantCreated))
|
||||||
|
fmt.Printf("║ Slug : %-47s ║\n", tenant.Slug)
|
||||||
|
fmt.Printf("║ Login : %-24s %-22s ║\n", existing.Email, statusWord(userCreated))
|
||||||
|
fmt.Printf("║ Rolle : %-47s ║\n", existing.Role)
|
||||||
|
if passwordShown {
|
||||||
|
fmt.Printf("║ Passwort : %-47s ║\n", password)
|
||||||
|
fmt.Println("║ Wird nur EINMAL angezeigt — nicht in Produktion nutzen! ║")
|
||||||
|
} else {
|
||||||
|
fmt.Println("║ Passwort : unverändert (mit -reset-password neu setzen) ║")
|
||||||
|
}
|
||||||
|
fmt.Println("╚══════════════════════════════════════════════════════════════╝")
|
||||||
|
fmt.Println()
|
||||||
|
|
||||||
|
logger.Info("seed dev complete",
|
||||||
|
"tenant_id", tenant.ID,
|
||||||
|
"tenant_created", tenantCreated,
|
||||||
|
"user_id", existing.ID,
|
||||||
|
"user_created", userCreated,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// findTenantBySlug returns the tenant with the given slug, or (nil, nil) when
|
||||||
|
// no such tenant exists. tenantstore has no slug lookup, and the tenant list is
|
||||||
|
// small (one row per Mandant), so scanning List is adequate here.
|
||||||
|
func findTenantBySlug(ctx context.Context, st *tenantstore.Store, slug string) (*tenantstore.Tenant, error) {
|
||||||
|
all, err := st.List(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("list tenants: %w", err)
|
||||||
|
}
|
||||||
|
for _, t := range all {
|
||||||
|
if t.Slug == slug {
|
||||||
|
return t, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// usernameFromEmail derives the users.username value (UNIQUE) from the login
|
||||||
|
// e-mail: local part, or the seed default when the address has no "@".
|
||||||
|
func usernameFromEmail(email string) string {
|
||||||
|
if i := strings.IndexByte(email, '@'); i > 0 {
|
||||||
|
return email[:i]
|
||||||
|
}
|
||||||
|
if email == "" {
|
||||||
|
return seedUserName
|
||||||
|
}
|
||||||
|
return email
|
||||||
|
}
|
||||||
|
|
||||||
|
func statusWord(created bool) string {
|
||||||
|
if created {
|
||||||
|
return "(neu angelegt)"
|
||||||
|
}
|
||||||
|
return "(bereits vorhanden)"
|
||||||
|
}
|
||||||
@@ -61,6 +61,9 @@ func main() {
|
|||||||
case "documents":
|
case "documents":
|
||||||
runDocuments(os.Args[2:])
|
runDocuments(os.Args[2:])
|
||||||
return
|
return
|
||||||
|
case "seed":
|
||||||
|
runSeed(os.Args[2:])
|
||||||
|
return
|
||||||
case "version":
|
case "version":
|
||||||
fmt.Printf("archivdms %s\n", AppVersion)
|
fmt.Printf("archivdms %s\n", AppVersion)
|
||||||
return
|
return
|
||||||
@@ -174,6 +177,8 @@ func main() {
|
|||||||
Secret: jwtSecret,
|
Secret: jwtSecret,
|
||||||
SecureCookies: cfg.API.SecureCookies,
|
SecureCookies: cfg.API.SecureCookies,
|
||||||
TrustedProxies: cfg.API.TrustedProxies,
|
TrustedProxies: cfg.API.TrustedProxies,
|
||||||
|
// FDN-08: zusätzliche Scrape-Quellen für GET /metrics (loopback immer).
|
||||||
|
MetricsAllowedIPs: cfg.API.MetricsAllowedIPs,
|
||||||
}
|
}
|
||||||
srv := api.New(apiCfg, docStore, authMgr, users, audlog, logger)
|
srv := api.New(apiCfg, docStore, authMgr, users, audlog, logger)
|
||||||
srv.SetTenants(tenantSt)
|
srv.SetTenants(tenantSt)
|
||||||
@@ -360,6 +365,8 @@ Usage:
|
|||||||
archivdms classify retrain [-config PATH] [-tenant N] [-dry-run] Naive-Bayes-Klassifikator neu trainieren (Cron)
|
archivdms classify retrain [-config PATH] [-tenant N] [-dry-run] Naive-Bayes-Klassifikator neu trainieren (Cron)
|
||||||
archivdms retention apply [-config PATH] [-tenant N] [-dry-run] GoBD-Aufbewahrungsfristen berechnen und retain_until setzen (Cron)
|
archivdms retention apply [-config PATH] [-tenant N] [-dry-run] GoBD-Aufbewahrungsfristen berechnen und retain_until setzen (Cron)
|
||||||
archivdms documents reprocess-all [-config PATH] [-tenant N] [-dry-run] [-delay-ms N] Alle Dokumente sequenziell neu OCR-verarbeiten (Altbestand)
|
archivdms documents reprocess-all [-config PATH] [-tenant N] [-dry-run] [-delay-ms N] Alle Dokumente sequenziell neu OCR-verarbeiten (Altbestand)
|
||||||
|
archivdms seed dev [-config PATH] [-name NAME] [-slug SLUG] [-email EMAIL] [-role ROLE] [-reset-password]
|
||||||
|
Test-Mandant + Test-Benutzer für die lokale Entwicklung anlegen (idempotent, Passwort wird einmalig ausgegeben)
|
||||||
archivdms version Version anzeigen
|
archivdms version Version anzeigen
|
||||||
archivdms help Diese Hilfe anzeigen`)
|
archivdms help Diese Hilfe anzeigen`)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -25,6 +25,10 @@ type APIConfig struct {
|
|||||||
// TrustedProxies is a list of IP addresses or CIDR ranges whose
|
// TrustedProxies is a list of IP addresses or CIDR ranges whose
|
||||||
// X-Forwarded-For header is trusted. Empty = trust no proxy.
|
// X-Forwarded-For header is trusted. Empty = trust no proxy.
|
||||||
TrustedProxies []string `yaml:"trusted_proxies"`
|
TrustedProxies []string `yaml:"trusted_proxies"`
|
||||||
|
// MetricsAllowedIPs sind zusätzliche Quell-IPs oder CIDR-Bereiche, die den
|
||||||
|
// unauthentifizierten Prometheus-Endpunkt GET /metrics scrapen dürfen
|
||||||
|
// (FDN-08). Loopback ist immer erlaubt, alles andere per Default gesperrt.
|
||||||
|
MetricsAllowedIPs []string `yaml:"metrics_allowed_ips"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ServerConfig holds general server settings.
|
// ServerConfig holds general server settings.
|
||||||
@@ -101,6 +105,18 @@ type StorageConfig struct {
|
|||||||
RetentionDays int `yaml:"retention_days"`
|
RetentionDays int `yaml:"retention_days"`
|
||||||
// MaxUploadSizeMB caps the accepted multipart upload size. 0 = default 50.
|
// MaxUploadSizeMB caps the accepted multipart upload size. 0 = default 50.
|
||||||
MaxUploadSizeMB int `yaml:"max_upload_size_mb"`
|
MaxUploadSizeMB int `yaml:"max_upload_size_mb"`
|
||||||
|
// SignedURLTTLMinutes is the default validity of signed download URLs
|
||||||
|
// (internal/objectstore). 0 = default 15 minutes.
|
||||||
|
SignedURLTTLMinutes int `yaml:"signed_url_ttl_minutes"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolvedSignedURLTTL returns the configured signed-download-URL validity,
|
||||||
|
// falling back to 15 minutes when unset (<= 0).
|
||||||
|
func (s StorageConfig) ResolvedSignedURLTTL() time.Duration {
|
||||||
|
if s.SignedURLTTLMinutes <= 0 {
|
||||||
|
return 15 * time.Minute
|
||||||
|
}
|
||||||
|
return time.Duration(s.SignedURLTTLMinutes) * time.Minute
|
||||||
}
|
}
|
||||||
|
|
||||||
// InboxPath returns the directory raw uploads are written to before hashing
|
// InboxPath returns the directory raw uploads are written to before hashing
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ storage:
|
|||||||
base_path: "/var/lib/archivdms" # enthält inbox/, store/, ocr-tmp/ (siehe README)
|
base_path: "/var/lib/archivdms" # enthält inbox/, store/, ocr-tmp/ (siehe README)
|
||||||
retention_days: 3650 # GoBD default: 10 Jahre
|
retention_days: 3650 # GoBD default: 10 Jahre
|
||||||
max_upload_size_mb: 50
|
max_upload_size_mb: 50
|
||||||
|
signed_url_ttl_minutes: 15 # Gültigkeit signierter Download-Links (Default 15)
|
||||||
|
|
||||||
ocr:
|
ocr:
|
||||||
tesseract_path: "tesseract" # muss im PATH liegen, apt install tesseract-ocr tesseract-ocr-deu
|
tesseract_path: "tesseract" # muss im PATH liegen, apt install tesseract-ocr tesseract-ocr-deu
|
||||||
@@ -70,6 +71,10 @@ api:
|
|||||||
secret: "CHANGE_ME_TO_A_LONG_RANDOM_SECRET"
|
secret: "CHANGE_ME_TO_A_LONG_RANDOM_SECRET"
|
||||||
secure_cookies: true
|
secure_cookies: true
|
||||||
trusted_proxies: []
|
trusted_proxies: []
|
||||||
|
# FDN-08: GET /metrics (Prometheus-Textformat) läuft ohne Login, dafür
|
||||||
|
# IP-beschränkt. Loopback ist immer erlaubt; hier zusätzliche Scraper
|
||||||
|
# freischalten (einzelne IPs oder CIDR). Leer = nur localhost.
|
||||||
|
metrics_allowed_ips: [] # z.B. ["192.168.1.0/24"]
|
||||||
|
|
||||||
audit:
|
audit:
|
||||||
log_path: "/var/log/archivdms/audit.log"
|
log_path: "/var/log/archivdms/audit.log"
|
||||||
|
|||||||
@@ -5,19 +5,27 @@ go 1.26.0
|
|||||||
toolchain go1.26.5
|
toolchain go1.26.5
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
github.com/go-ldap/ldap/v3 v3.4.14
|
||||||
github.com/go-sql-driver/mysql v1.8.1
|
github.com/go-sql-driver/mysql v1.8.1
|
||||||
github.com/golang-jwt/jwt/v5 v5.2.1
|
github.com/golang-jwt/jwt/v5 v5.2.1
|
||||||
github.com/jackc/pgx/v5 v5.6.0
|
github.com/jackc/pgx/v5 v5.6.0
|
||||||
github.com/pkg/sftp v1.13.7
|
github.com/pkg/sftp v1.13.7
|
||||||
golang.org/x/crypto v0.48.0
|
golang.org/x/crypto v0.54.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
filippo.io/edwards25519 v1.1.0 // indirect
|
filippo.io/edwards25519 v1.1.0 // indirect
|
||||||
|
github.com/Azure/go-ntlmssp v0.1.1 // indirect
|
||||||
|
github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
|
||||||
|
github.com/google/uuid v1.6.0 // indirect
|
||||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||||
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
|
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
|
||||||
github.com/jackc/puddle/v2 v2.2.1 // indirect
|
github.com/jackc/puddle/v2 v2.2.1 // indirect
|
||||||
golang.org/x/sync v0.19.0 // indirect
|
github.com/kr/fs v0.1.0 // indirect
|
||||||
golang.org/x/text v0.34.0 // indirect
|
github.com/kr/text v0.2.0 // indirect
|
||||||
|
github.com/rogpeppe/go-internal v1.16.0 // indirect
|
||||||
|
golang.org/x/sync v0.22.0 // indirect
|
||||||
|
golang.org/x/sys v0.47.0 // indirect
|
||||||
|
golang.org/x/text v0.40.0 // indirect
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -0,0 +1,118 @@
|
|||||||
|
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
|
||||||
|
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
|
||||||
|
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
|
||||||
|
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
|
||||||
|
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktpoUAgOJK3OTFc/xug0PCXYCqU0FgDKI=
|
||||||
|
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
|
||||||
|
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||||
|
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
|
||||||
|
github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
|
||||||
|
github.com/go-ldap/ldap/v3 v3.4.14 h1:D6PYdEgsaVzsXyr6w/yDC06Ria4uUhWm+Rb+er8lfAs=
|
||||||
|
github.com/go-ldap/ldap/v3 v3.4.14/go.mod h1:S4eJUMUNjDkE0ZJtIZdybwyb03sGGLW6gxXT1Hs8VKA=
|
||||||
|
github.com/go-sql-driver/mysql v1.8.1 h1:LedoTUt/eveggdHS9qUFC1EFSa8bU2+1pZjSRpvNJ1Y=
|
||||||
|
github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg=
|
||||||
|
github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
|
||||||
|
github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
|
||||||
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8=
|
||||||
|
github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
|
||||||
|
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||||
|
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk=
|
||||||
|
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||||
|
github.com/jackc/pgx/v5 v5.6.0 h1:SWJzexBzPL5jb0GEsrPMLIsi/3jOo7RHlzTjcAeDrPY=
|
||||||
|
github.com/jackc/pgx/v5 v5.6.0/go.mod h1:DNZ/vlrUnhWCoFGxHAG8U2ljioxukquj7utPDgtQdTw=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.1 h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk=
|
||||||
|
github.com/jackc/puddle/v2 v2.2.1/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
|
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
|
||||||
|
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
|
||||||
|
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
|
||||||
|
github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM=
|
||||||
|
github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg=
|
||||||
|
github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo=
|
||||||
|
github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o=
|
||||||
|
github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg=
|
||||||
|
github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8=
|
||||||
|
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
|
||||||
|
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
|
||||||
|
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
|
||||||
|
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
|
||||||
|
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
|
||||||
|
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
|
||||||
|
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
|
||||||
|
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||||
|
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||||
|
github.com/pkg/sftp v1.13.7 h1:uv+I3nNJvlKZIQGSr8JVQLNHFU9YhhNpvC14Y6KgmSM=
|
||||||
|
github.com/pkg/sftp v1.13.7/go.mod h1:KMKI0t3T6hfA+lTR/ssZdunHo+uwq7ghoN09/FSu3DY=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/rogpeppe/go-internal v1.16.0 h1:O9DK+vNMDVGLr2BeZqmpLeMjiMNkuXfcqntWbZV6S5g=
|
||||||
|
github.com/rogpeppe/go-internal v1.16.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
|
||||||
|
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||||
|
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||||
|
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||||
|
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
|
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||||
|
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||||
|
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
||||||
|
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||||
|
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||||
|
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||||
|
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||||
|
golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
|
||||||
|
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
|
||||||
|
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
|
||||||
|
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||||
|
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||||
|
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||||
|
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
||||||
|
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||||
|
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||||
|
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
|
||||||
|
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||||
|
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||||
|
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||||
|
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||||
|
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||||
|
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||||
|
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||||
|
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
|
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||||
|
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||||
|
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
|
||||||
|
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
|
||||||
|
golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0=
|
||||||
|
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
||||||
|
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
||||||
|
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||||
|
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||||
|
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||||
|
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||||
|
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
|
||||||
|
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||||
|
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||||
|
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||||
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
|
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||||
|
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||||
|
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
|
||||||
|
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||||
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||||
|
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
@@ -190,7 +190,7 @@ func (s *Server) accountingAuth(h http.HandlerFunc) http.HandlerFunc {
|
|||||||
tenantID, keyID, err := s.store.ResolveAccountingAPIKey(r.Context(), rawKey)
|
tenantID, keyID, err := s.store.ResolveAccountingAPIKey(r.Context(), rawKey)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if !errors.Is(err, storage.ErrAccountingKeyNotFound) {
|
if !errors.Is(err, storage.ErrAccountingKeyNotFound) {
|
||||||
s.logger.Error("accounting api key resolve failed", "err", err)
|
s.reqLog(r.Context()).Error("accounting api key resolve failed", "err", err)
|
||||||
}
|
}
|
||||||
// Unknown, revoked and broken keys are indistinguishable.
|
// Unknown, revoked and broken keys are indistinguishable.
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
@@ -290,7 +290,7 @@ func (s *Server) handleAccountingListDocuments(w http.ResponseWriter, r *http.Re
|
|||||||
writeError(w, http.StatusBadRequest, "invalid cursor")
|
writeError(w, http.StatusBadRequest, "invalid cursor")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.logger.Error("accounting list failed", "tenant_id", tenantID, "err", err)
|
s.reqLog(r.Context()).Error("accounting list failed", "tenant_id", tenantID, "err", err)
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
|
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
|
||||||
TenantID: &tenantID, Success: false,
|
TenantID: &tenantID, Success: false,
|
||||||
@@ -342,7 +342,7 @@ func (s *Server) handleAccountingDocumentFile(w http.ResponseWriter, r *http.Req
|
|||||||
|
|
||||||
f, err := os.Open(ref.StoragePath())
|
f, err := os.Open(ref.StoragePath())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("accounting file open failed", "document_id", ref.DocumentID, "tenant_id", tenantID, "err", err)
|
s.reqLog(r.Context()).Error("accounting file open failed", "document_id", ref.DocumentID, "tenant_id", tenantID, "err", err)
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
|
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
|
||||||
TenantID: &tenantID, DocumentID: strconv.FormatInt(id, 10), Success: false,
|
TenantID: &tenantID, DocumentID: strconv.FormatInt(id, 10), Success: false,
|
||||||
@@ -364,7 +364,7 @@ func (s *Server) handleAccountingDocumentFile(w http.ResponseWriter, r *http.Req
|
|||||||
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(ref.Title, ext)+"\"")
|
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(ref.Title, ext)+"\"")
|
||||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
if _, err := io.Copy(w, f); err != nil {
|
if _, err := io.Copy(w, f); err != nil {
|
||||||
s.logger.Warn("accounting file stream interrupted", "document_id", ref.DocumentID, "err", err)
|
s.reqLog(r.Context()).Warn("accounting file stream interrupted", "document_id", ref.DocumentID, "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -13,15 +13,22 @@ type loginRequest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleLogin(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleLogin(w http.ResponseWriter, r *http.Request) {
|
||||||
|
ctx := r.Context()
|
||||||
var req loginRequest
|
var req loginRequest
|
||||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||||
|
s.reqLog(ctx).Warn("login: invalid request body",
|
||||||
|
"remote_ip", s.remoteIP(r), "err", err)
|
||||||
writeError(w, http.StatusBadRequest, "invalid request body")
|
writeError(w, http.StatusBadRequest, "invalid request body")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
ip := s.remoteIP(r)
|
ip := s.remoteIP(r)
|
||||||
token, user, err := s.authMgr.LoginFrom(r.Context(), req.Username, req.Password, ip)
|
token, user, err := s.authMgr.LoginFrom(ctx, req.Username, req.Password, ip)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
// Kein Passwort, keine Fehlerdetails vom Auth-Manager im Klartext:
|
||||||
|
// nur Benutzername + IP zur Korrelation von Brute-Force-Versuchen.
|
||||||
|
s.reqLog(ctx).Warn("login failed",
|
||||||
|
"username", req.Username, "remote_ip", ip, "reason", "invalid_credentials")
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventLogin,
|
EventType: audit.EventLogin,
|
||||||
Username: req.Username,
|
Username: req.Username,
|
||||||
@@ -43,7 +50,14 @@ func (s *Server) handleLogin(w http.ResponseWriter, r *http.Request) {
|
|||||||
MaxAge: 8 * 60 * 60,
|
MaxAge: 8 * 60 * 60,
|
||||||
})
|
})
|
||||||
|
|
||||||
_ = s.users.UpdateLastLogin(user.ID)
|
if err := s.users.UpdateLastLogin(user.ID); err != nil {
|
||||||
|
s.reqLog(ctx).Warn("login: last_login update failed",
|
||||||
|
"user_id", user.ID, "err", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
s.reqLog(ctx).Info("login succeeded",
|
||||||
|
"user_id", user.ID, "username", user.Username,
|
||||||
|
"tenant_id", user.TenantID, "remote_ip", ip)
|
||||||
|
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventLogin,
|
EventType: audit.EventLogin,
|
||||||
@@ -57,9 +71,12 @@ func (s *Server) handleLogin(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleMe(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleMe(w http.ResponseWriter, r *http.Request) {
|
||||||
sess := sessionFromCtx(r.Context())
|
ctx := r.Context()
|
||||||
|
sess := sessionFromCtx(ctx)
|
||||||
user, err := s.users.GetByID(sess.UserID)
|
user, err := s.users.GetByID(sess.UserID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
s.reqLog(ctx).Warn("me: user lookup failed",
|
||||||
|
"user_id", sess.UserID, "err", err)
|
||||||
writeError(w, http.StatusNotFound, "user not found")
|
writeError(w, http.StatusNotFound, "user not found")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -67,7 +84,8 @@ func (s *Server) handleMe(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) handleLogout(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleLogout(w http.ResponseWriter, r *http.Request) {
|
||||||
sess := sessionFromCtx(r.Context())
|
ctx := r.Context()
|
||||||
|
sess := sessionFromCtx(ctx)
|
||||||
token := ""
|
token := ""
|
||||||
if c, err := r.Cookie(sessionCookieName); err == nil {
|
if c, err := r.Cookie(sessionCookieName); err == nil {
|
||||||
token = c.Value
|
token = c.Value
|
||||||
@@ -76,9 +94,16 @@ func (s *Server) handleLogout(w http.ResponseWriter, r *http.Request) {
|
|||||||
token = extractBearerToken(r)
|
token = extractBearerToken(r)
|
||||||
}
|
}
|
||||||
if token != "" {
|
if token != "" {
|
||||||
_ = s.authMgr.Logout(token)
|
if err := s.authMgr.Logout(token); err != nil {
|
||||||
|
s.reqLog(ctx).Warn("logout: session invalidation failed",
|
||||||
|
"user_id", sess.UserID, "err", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
s.reqLog(ctx).Info("logout",
|
||||||
|
"user_id", sess.UserID, "username", sess.Username,
|
||||||
|
"tenant_id", sess.TenantID, "remote_ip", s.remoteIP(r))
|
||||||
|
|
||||||
http.SetCookie(w, &http.Cookie{
|
http.SetCookie(w, &http.Cookie{
|
||||||
Name: sessionCookieName,
|
Name: sessionCookieName,
|
||||||
Value: "",
|
Value: "",
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
|
|||||||
|
|
||||||
stats, err := s.store.GetDashboardStats(r.Context(), *sess.TenantID, sess.UserID)
|
stats, err := s.store.GetDashboardStats(r.Context(), *sess.TenantID, sess.UserID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("dashboard stats failed", "err", err)
|
s.reqLog(r.Context()).Error("dashboard stats failed", "err", err)
|
||||||
writeError(w, http.StatusInternalServerError, "dashboard stats failed")
|
writeError(w, http.StatusInternalServerError, "dashboard stats failed")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -197,7 +197,7 @@ func (s *Server) handleBulkExportDocuments(w http.ResponseWriter, r *http.Reques
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
// A single unreadable document must not kill the archive — unless the
|
// A single unreadable document must not kill the archive — unless the
|
||||||
// ZIP writer itself failed, which we detect on Close below.
|
// ZIP writer itself failed, which we detect on Close below.
|
||||||
s.logger.Warn("bulk export: document skipped", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(r.Context()).Warn("bulk export: document skipped", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
skipped = append(skipped, fmt.Sprintf("%d: %v", doc.ID, err))
|
skipped = append(skipped, fmt.Sprintf("%d: %v", doc.ID, err))
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -225,7 +225,7 @@ func (s *Server) handleBulkExportDocuments(w http.ResponseWriter, r *http.Reques
|
|||||||
detail := fmt.Sprintf("zip_bulk_export: exported=%d skipped=%d", exported, len(skipped))
|
detail := fmt.Sprintf("zip_bulk_export: exported=%d skipped=%d", exported, len(skipped))
|
||||||
if streamErr != nil {
|
if streamErr != nil {
|
||||||
// Headers are already out — audit the partial export, no HTTP error.
|
// Headers are already out — audit the partial export, no HTTP error.
|
||||||
s.logger.Warn("bulk document export stream failed", "tenant_id", tenantID, "err", streamErr)
|
s.reqLog(r.Context()).Warn("bulk document export stream failed", "tenant_id", tenantID, "err", streamErr)
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventDocumentBulkExport, Username: sess.Username, TenantID: sess.TenantID,
|
EventType: audit.EventDocumentBulkExport, Username: sess.Username, TenantID: sess.TenantID,
|
||||||
Success: false, Detail: detail + " stream_failed: " + streamErr.Error(),
|
Success: false, Detail: detail + " stream_failed: " + streamErr.Error(),
|
||||||
|
|||||||
@@ -143,7 +143,7 @@ func (s *Server) handleExportDocument(w http.ResponseWriter, r *http.Request) {
|
|||||||
|
|
||||||
f, err := os.Open(doc.StoragePath)
|
f, err := os.Open(doc.StoragePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("export: document file open failed", "document_id", doc.ID, "tenant_id", *sess.TenantID, "err", err)
|
s.reqLog(r.Context()).Error("export: document file open failed", "document_id", doc.ID, "tenant_id", *sess.TenantID, "err", err)
|
||||||
fail(http.StatusInternalServerError, "file unavailable", "file_open_failed: "+err.Error())
|
fail(http.StatusInternalServerError, "file unavailable", "file_open_failed: "+err.Error())
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -212,7 +212,7 @@ func (s *Server) handleExportDocument(w http.ResponseWriter, r *http.Request) {
|
|||||||
|
|
||||||
if streamErr != nil {
|
if streamErr != nil {
|
||||||
// Headers are already out — log + audit the partial export, no HTTP error.
|
// Headers are already out — log + audit the partial export, no HTTP error.
|
||||||
s.logger.Warn("document export stream failed", "document_id", doc.ID, "err", streamErr)
|
s.reqLog(r.Context()).Warn("document export stream failed", "document_id", doc.ID, "err", streamErr)
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventDocumentExport, Username: sess.Username, TenantID: sess.TenantID,
|
EventType: audit.EventDocumentExport, Username: sess.Username, TenantID: sess.TenantID,
|
||||||
DocumentID: idStr, Success: false, Detail: "stream_failed: " + streamErr.Error(),
|
DocumentID: idStr, Success: false, Detail: "stream_failed: " + streamErr.Error(),
|
||||||
|
|||||||
@@ -21,6 +21,7 @@ import (
|
|||||||
"archivdms/internal/auth"
|
"archivdms/internal/auth"
|
||||||
"archivdms/internal/dateformat"
|
"archivdms/internal/dateformat"
|
||||||
"archivdms/internal/matching"
|
"archivdms/internal/matching"
|
||||||
|
"archivdms/internal/objectstore"
|
||||||
"archivdms/internal/ocr"
|
"archivdms/internal/ocr"
|
||||||
"archivdms/internal/storage"
|
"archivdms/internal/storage"
|
||||||
"archivdms/internal/userstore"
|
"archivdms/internal/userstore"
|
||||||
@@ -138,10 +139,12 @@ func (s *Server) handleGetDocumentFile(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
f, err := os.Open(doc.StoragePath)
|
// Read through the storage abstraction: it re-checks that the stored path
|
||||||
|
// really lies inside this tenant's store subtree before opening.
|
||||||
|
f, err := s.objects.Open(r.Context(), *sess.TenantID, doc.StoragePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// WORM store should always hold the file, but never trust the disk.
|
// WORM store should always hold the file, but never trust the disk.
|
||||||
s.logger.Error("document file open failed", "document_id", doc.ID, "tenant_id", *sess.TenantID, "storage_path", doc.StoragePath, "err", err)
|
s.reqLog(r.Context()).Error("document file open failed", "document_id", doc.ID, "tenant_id", *sess.TenantID, "storage_path", doc.StoragePath, "err", err)
|
||||||
writeError(w, http.StatusInternalServerError, "file unavailable")
|
writeError(w, http.StatusInternalServerError, "file unavailable")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -152,7 +155,7 @@ func (s *Server) handleGetDocumentFile(w http.ResponseWriter, r *http.Request) {
|
|||||||
w.Header().Set("Content-Disposition", "inline; filename=\""+safeDownloadName(doc.Title, ext)+"\"")
|
w.Header().Set("Content-Disposition", "inline; filename=\""+safeDownloadName(doc.Title, ext)+"\"")
|
||||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
if _, err := io.Copy(w, f); err != nil {
|
if _, err := io.Copy(w, f); err != nil {
|
||||||
s.logger.Warn("document file stream interrupted", "document_id", doc.ID, "err", err)
|
s.reqLog(r.Context()).Warn("document file stream interrupted", "document_id", doc.ID, "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -209,7 +212,7 @@ func (s *Server) handleGetDocumentThumbnail(w http.ResponseWriter, r *http.Reque
|
|||||||
w.Header().Set("Cache-Control", "private, max-age=86400")
|
w.Header().Set("Cache-Control", "private, max-age=86400")
|
||||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
if _, err := io.Copy(w, f); err != nil {
|
if _, err := io.Copy(w, f); err != nil {
|
||||||
s.logger.Warn("thumbnail stream interrupted", "document_id", doc.ID, "err", err)
|
s.reqLog(r.Context()).Warn("thumbnail stream interrupted", "document_id", doc.ID, "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -453,7 +456,7 @@ func (s *Server) ReprocessDocument(ctx context.Context, tenantID, id int64, acto
|
|||||||
mimeType := detectMimeType("", ext, doc.StoragePath)
|
mimeType := detectMimeType("", ext, doc.StoragePath)
|
||||||
result, err := s.ocr.Extract(ctx, doc.StoragePath, mimeType)
|
result, err := s.ocr.Extract(ctx, doc.StoragePath, mimeType)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Warn("reprocess ocr extraction failed", "document_id", doc.ID, "tenant_id", tenantID, "storage_path", doc.StoragePath, "err", err)
|
s.reqLog(ctx).Warn("reprocess ocr extraction failed", "document_id", doc.ID, "tenant_id", tenantID, "storage_path", doc.StoragePath, "err", err)
|
||||||
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentReprocessed, Username: actor, TenantID: &tenantID, DocumentID: docIDStr, Success: false, Detail: "ocr_failed: " + err.Error()})
|
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentReprocessed, Username: actor, TenantID: &tenantID, DocumentID: docIDStr, Success: false, Detail: "ocr_failed: " + err.Error()})
|
||||||
return nil, fmt.Errorf("reprocess ocr extract: %w", err)
|
return nil, fmt.Errorf("reprocess ocr extract: %w", err)
|
||||||
}
|
}
|
||||||
@@ -466,7 +469,7 @@ func (s *Server) ReprocessDocument(ctx context.Context, tenantID, id int64, acto
|
|||||||
// alongside the new ones — best-effort, never fails the reprocess since
|
// alongside the new ones — best-effort, never fails the reprocess since
|
||||||
// ocr_text is already the authoritative persisted result.
|
// ocr_text is already the authoritative persisted result.
|
||||||
if err := s.store.ReplaceOCRWords(ctx, id, ocrWordsFromResult(id, result.Words)); err != nil {
|
if err := s.store.ReplaceOCRWords(ctx, id, ocrWordsFromResult(id, result.Words)); err != nil {
|
||||||
s.logger.Warn("reprocess replace ocr_words failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("reprocess replace ocr_words failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.store.UpdateDocumentOCRText(ctx, id, tenantID, ocrText); err != nil {
|
if err := s.store.UpdateDocumentOCRText(ctx, id, tenantID, ocrText); err != nil {
|
||||||
@@ -491,7 +494,7 @@ func (s *Server) ReprocessDocument(ctx context.Context, tenantID, id int64, acto
|
|||||||
prefix, dateLayout := s.tenantScanTitleParams(ctx, tenantID)
|
prefix, dateLayout := s.tenantScanTitleParams(ctx, tenantID)
|
||||||
if newTitle := titleFromOCRText(ocrText, prefix, dateLayout); newTitle != "" && newTitle != doc.Title {
|
if newTitle := titleFromOCRText(ocrText, prefix, dateLayout); newTitle != "" && newTitle != doc.Title {
|
||||||
if err := s.store.UpdateDocumentTitleAuto(ctx, id, tenantID, newTitle); err != nil {
|
if err := s.store.UpdateDocumentTitleAuto(ctx, id, tenantID, newTitle); err != nil {
|
||||||
s.logger.Warn("reprocess title update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("reprocess title update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
} else {
|
} else {
|
||||||
doc.Title = newTitle
|
doc.Title = newTitle
|
||||||
}
|
}
|
||||||
@@ -513,7 +516,7 @@ func (s *Server) ReprocessDocument(ctx context.Context, tenantID, id int64, acto
|
|||||||
}
|
}
|
||||||
if !sameDate(datePtr, doc.DocumentDate) {
|
if !sameDate(datePtr, doc.DocumentDate) {
|
||||||
if err := s.store.UpdateDocumentDate(ctx, id, tenantID, datePtr, scorePtr); err != nil {
|
if err := s.store.UpdateDocumentDate(ctx, id, tenantID, datePtr, scorePtr); err != nil {
|
||||||
s.logger.Warn("reprocess document_date update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("reprocess document_date update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
} else {
|
} else {
|
||||||
doc.DocumentDate = datePtr
|
doc.DocumentDate = datePtr
|
||||||
doc.DocumentDateScore = scorePtr
|
doc.DocumentDateScore = scorePtr
|
||||||
@@ -524,12 +527,12 @@ func (s *Server) ReprocessDocument(ctx context.Context, tenantID, id int64, acto
|
|||||||
// Additive auto-assignment (only fills unset doc_type/correspondent, only
|
// Additive auto-assignment (only fills unset doc_type/correspondent, only
|
||||||
// attaches tags) — best-effort, never fails the request.
|
// attaches tags) — best-effort, never fails the request.
|
||||||
if assignWarn := s.autoAssignTaxonomy(ctx, tenantID, doc, barcodes, ocrText); assignWarn != "" {
|
if assignWarn := s.autoAssignTaxonomy(ctx, tenantID, doc, barcodes, ocrText); assignWarn != "" {
|
||||||
s.logger.Info("reprocess auto-assignment", "document_id", doc.ID, "tenant_id", tenantID, "detail", assignWarn)
|
s.reqLog(ctx).Info("reprocess auto-assignment", "document_id", doc.ID, "tenant_id", tenantID, "detail", assignWarn)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Re-evaluate on_upload workflows — best-effort, never fails the request.
|
// Re-evaluate on_upload workflows — best-effort, never fails the request.
|
||||||
if err := s.store.RunWorkflowsForDocument(ctx, tenantID, doc, storage.WorkflowTriggerOnUpload); err != nil {
|
if err := s.store.RunWorkflowsForDocument(ctx, tenantID, doc, storage.WorkflowTriggerOnUpload); err != nil {
|
||||||
s.logger.Warn("reprocess workflow execution failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("reprocess workflow execution failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Re-read so the response reflects any doc_type/correspondent set by
|
// Re-read so the response reflects any doc_type/correspondent set by
|
||||||
@@ -726,11 +729,11 @@ func (s *Server) generateThumbnailBestEffort(ctx context.Context, tenantID, docu
|
|||||||
}
|
}
|
||||||
thumbPath := filepath.Join(s.storageCfg.ThumbnailPath(), strconv.FormatInt(tenantID, 10), contentHash+".png")
|
thumbPath := filepath.Join(s.storageCfg.ThumbnailPath(), strconv.FormatInt(tenantID, 10), contentHash+".png")
|
||||||
if err := s.thumbs.Generate(ctx, storagePath, mimeType, thumbPath); err != nil {
|
if err := s.thumbs.Generate(ctx, storagePath, mimeType, thumbPath); err != nil {
|
||||||
s.logger.Warn("eager thumbnail generation failed", "document_id", documentID, "tenant_id", tenantID, "mime_type", mimeType, "err", err)
|
s.reqLog(ctx).Warn("eager thumbnail generation failed", "document_id", documentID, "tenant_id", tenantID, "mime_type", mimeType, "err", err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if err := s.store.SetDocumentHasThumbnail(ctx, documentID, tenantID, true); err != nil {
|
if err := s.store.SetDocumentHasThumbnail(ctx, documentID, tenantID, true); err != nil {
|
||||||
s.logger.Warn("has_thumbnail flag update failed", "document_id", documentID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("has_thumbnail flag update failed", "document_id", documentID, "tenant_id", tenantID, "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -753,7 +756,7 @@ func (s *Server) handleUploadDocument(w http.ResponseWriter, r *http.Request) {
|
|||||||
tenantID := *sess.TenantID
|
tenantID := *sess.TenantID
|
||||||
|
|
||||||
maxBytes := int64(s.storageCfg.ResolvedMaxUploadSizeMB()) * 1024 * 1024
|
maxBytes := int64(s.storageCfg.ResolvedMaxUploadSizeMB()) * 1024 * 1024
|
||||||
s.logger.Info("upload request received", "username", sess.Username, "tenant_id", tenantID,
|
s.reqLog(r.Context()).Info("upload request received", "username", sess.Username, "tenant_id", tenantID,
|
||||||
"content_length", r.ContentLength, "max_bytes", maxBytes, "remote_addr", r.RemoteAddr)
|
"content_length", r.ContentLength, "max_bytes", maxBytes, "remote_addr", r.RemoteAddr)
|
||||||
|
|
||||||
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
|
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
|
||||||
@@ -763,7 +766,7 @@ func (s *Server) handleUploadDocument(w http.ResponseWriter, r *http.Request) {
|
|||||||
// hitting MaxBytesReader — both close the connection before any
|
// hitting MaxBytesReader — both close the connection before any
|
||||||
// later log/audit call would otherwise run, which previously left
|
// later log/audit call would otherwise run, which previously left
|
||||||
// zero trace of the failure in the backend logs (see DEVLOG 2026-07-15).
|
// zero trace of the failure in the backend logs (see DEVLOG 2026-07-15).
|
||||||
s.logger.Warn("upload parse failed", "username", sess.Username, "tenant_id", tenantID,
|
s.reqLog(r.Context()).Warn("upload parse failed", "username", sess.Username, "tenant_id", tenantID,
|
||||||
"content_length", r.ContentLength, "max_bytes", maxBytes, "err", err)
|
"content_length", r.ContentLength, "max_bytes", maxBytes, "err", err)
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventDocumentCreate, Username: sess.Username, TenantID: sess.TenantID,
|
EventType: audit.EventDocumentCreate, Username: sess.Username, TenantID: sess.TenantID,
|
||||||
@@ -956,41 +959,19 @@ func (s *Server) archiveStagedFile(
|
|||||||
}
|
}
|
||||||
pathDate := time.Now()
|
pathDate := time.Now()
|
||||||
|
|
||||||
// 4. Build the WORM target path store/<tenant>/<yyyy>/<mm>/<hash>.<ext>.
|
// 4./5./6./7. WORM archival — delegated unchanged to the storage
|
||||||
storeDir := filepath.Join(s.storageCfg.StorePath(), strconv.FormatInt(tenantID, 10),
|
// abstraction (internal/objectstore): build the target path
|
||||||
fmt.Sprintf("%04d", pathDate.Year()), fmt.Sprintf("%02d", pathDate.Month()))
|
// store/<tenant>/<yyyy>/<mm>/<hash>.<ext>, reject an already-stored hash as
|
||||||
if err := os.MkdirAll(storeDir, 0o750); err != nil {
|
// duplicate (filesystem half of the duplicate protection; the DB unique
|
||||||
os.Remove(inboxPath)
|
// index is the other half), move inbox -> store (rename, copy+remove
|
||||||
return nil, "", fmt.Errorf("create store dir: %w", err)
|
// fallback across mounts) and finally chmod 0440. Path scheme and
|
||||||
}
|
// semantics are identical to the previous inline implementation.
|
||||||
storePath := filepath.Join(storeDir, contentHash+ext)
|
storePath, err := s.objects.Archive(ctx, tenantID, inboxPath, ext, contentHash, pathDate)
|
||||||
|
if err != nil {
|
||||||
// 5. Collision check: identical hash already stored -> reject as
|
if errors.Is(err, objectstore.ErrObjectExists) {
|
||||||
// duplicate before touching the DB (filesystem-level half of the
|
return nil, "", storage.ErrDuplicateContentHash
|
||||||
// duplicate protection; the DB unique index is the other half).
|
|
||||||
if _, err := os.Stat(storePath); err == nil {
|
|
||||||
os.Remove(inboxPath)
|
|
||||||
return nil, "", storage.ErrDuplicateContentHash
|
|
||||||
} else if !os.IsNotExist(err) {
|
|
||||||
os.Remove(inboxPath)
|
|
||||||
return nil, "", fmt.Errorf("stat store path: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// 6. Move inbox -> store. Prefer atomic rename; fall back to copy+remove
|
|
||||||
// if inbox/store ever end up on different filesystems/mounts.
|
|
||||||
if err := os.Rename(inboxPath, storePath); err != nil {
|
|
||||||
if copyErr := copyFile(inboxPath, storePath); copyErr != nil {
|
|
||||||
os.Remove(inboxPath)
|
|
||||||
return nil, "", fmt.Errorf("move file to store: rename failed (%v), copy fallback failed: %w", err, copyErr)
|
|
||||||
}
|
}
|
||||||
os.Remove(inboxPath)
|
return nil, "", err
|
||||||
}
|
|
||||||
|
|
||||||
// 7. WORM lock: read-only, no write access for anyone once archived. This is
|
|
||||||
// the ONLY chmod and it happens exactly once, after the file reaches its
|
|
||||||
// final path — the file is never moved or renamed again afterwards.
|
|
||||||
if err := os.Chmod(storePath, 0o440); err != nil {
|
|
||||||
return nil, "", fmt.Errorf("chmod store file: %w", err)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// 8. Record the document AND its processing job in ONE transaction. If no
|
// 8. Record the document AND its processing job in ONE transaction. If no
|
||||||
@@ -1018,7 +999,7 @@ func (s *Server) archiveStagedFile(
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, "", err
|
return nil, "", err
|
||||||
}
|
}
|
||||||
s.logger.Info("document staged, processing job queued",
|
s.reqLog(ctx).Info("document staged, processing job queued",
|
||||||
"document_id", doc.ID, "tenant_id", tenantID, "job_id", job.ID, "derive_title", deriveTitle)
|
"document_id", doc.ID, "tenant_id", tenantID, "job_id", job.ID, "derive_title", deriveTitle)
|
||||||
|
|
||||||
// 7b. Eager thumbnail generation: same pipeline/cache path as the lazy
|
// 7b. Eager thumbnail generation: same pipeline/cache path as the lazy
|
||||||
@@ -1084,7 +1065,7 @@ func (s *Server) trySplitStagedUpload(
|
|||||||
|
|
||||||
res, split, err := s.pagesplitter.Split(ctx, inboxPath)
|
res, split, err := s.pagesplitter.Split(ctx, inboxPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Warn("separator-page split failed, archiving document unsplit",
|
s.reqLog(ctx).Warn("separator-page split failed, archiving document unsplit",
|
||||||
"tenant_id", tenantID, "filename", filename, "err", err)
|
"tenant_id", tenantID, "filename", filename, "err", err)
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
EventType: audit.EventDocumentSplit, Username: splitActor, TenantID: &tenantID, Success: false,
|
EventType: audit.EventDocumentSplit, Username: splitActor, TenantID: &tenantID, Success: false,
|
||||||
@@ -1167,7 +1148,7 @@ func (s *Server) trySplitStagedUpload(
|
|||||||
EventType: audit.EventDocumentSplit, Username: splitActor, TenantID: &tenantID,
|
EventType: audit.EventDocumentSplit, Username: splitActor, TenantID: &tenantID,
|
||||||
DocumentID: strconv.FormatInt(docs[0].ID, 10), Success: true, Detail: detail,
|
DocumentID: strconv.FormatInt(docs[0].ID, 10), Success: true, Detail: detail,
|
||||||
})
|
})
|
||||||
s.logger.Info("upload split at barcode separator pages",
|
s.reqLog(ctx).Info("upload split at barcode separator pages",
|
||||||
"tenant_id", tenantID, "filename", filename, "pages", res.PageCount,
|
"tenant_id", tenantID, "filename", filename, "pages", res.PageCount,
|
||||||
"separator_pages", res.SeparatorPages, "parts", len(res.Parts),
|
"separator_pages", res.SeparatorPages, "parts", len(res.Parts),
|
||||||
"documents_created", len(docs), "duplicates_skipped", dupCount)
|
"documents_created", len(docs), "duplicates_skipped", dupCount)
|
||||||
@@ -1238,7 +1219,7 @@ func (s *Server) ProcessDocumentJob(ctx context.Context, tenantID, documentID in
|
|||||||
mimeType := detectMimeType("", ext, doc.StoragePath)
|
mimeType := detectMimeType("", ext, doc.StoragePath)
|
||||||
result, err := s.ocr.Extract(ctx, doc.StoragePath, mimeType)
|
result, err := s.ocr.Extract(ctx, doc.StoragePath, mimeType)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Warn("jobqueue ocr extraction failed", "document_id", doc.ID, "tenant_id", tenantID,
|
s.reqLog(ctx).Warn("jobqueue ocr extraction failed", "document_id", doc.ID, "tenant_id", tenantID,
|
||||||
"storage_path", doc.StoragePath, "resolved_mime", mimeType, "err", err)
|
"storage_path", doc.StoragePath, "resolved_mime", mimeType, "err", err)
|
||||||
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentProcessed, Username: actor, TenantID: &tenantID, DocumentID: docIDStr, Success: false, Detail: "ocr_failed: " + err.Error()})
|
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentProcessed, Username: actor, TenantID: &tenantID, DocumentID: docIDStr, Success: false, Detail: "ocr_failed: " + err.Error()})
|
||||||
return fmt.Errorf("jobqueue ocr extract: %w", err)
|
return fmt.Errorf("jobqueue ocr extract: %w", err)
|
||||||
@@ -1251,7 +1232,7 @@ func (s *Server) ProcessDocumentJob(ctx context.Context, tenantID, documentID in
|
|||||||
// retry/backoff) never accumulates duplicates. Best-effort, never fails
|
// retry/backoff) never accumulates duplicates. Best-effort, never fails
|
||||||
// the job since ocr_text is already the authoritative persisted result.
|
// the job since ocr_text is already the authoritative persisted result.
|
||||||
if err := s.store.ReplaceOCRWords(ctx, documentID, ocrWordsFromResult(documentID, result.Words)); err != nil {
|
if err := s.store.ReplaceOCRWords(ctx, documentID, ocrWordsFromResult(documentID, result.Words)); err != nil {
|
||||||
s.logger.Warn("jobqueue replace ocr_words failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("jobqueue replace ocr_words failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.store.UpdateDocumentOCRText(ctx, documentID, tenantID, ocrText); err != nil {
|
if err := s.store.UpdateDocumentOCRText(ctx, documentID, tenantID, ocrText); err != nil {
|
||||||
@@ -1267,7 +1248,7 @@ func (s *Server) ProcessDocumentJob(ctx context.Context, tenantID, documentID in
|
|||||||
prefix, dateLayout := s.tenantScanTitleParams(ctx, tenantID)
|
prefix, dateLayout := s.tenantScanTitleParams(ctx, tenantID)
|
||||||
if newTitle := titleFromOCRText(ocrText, prefix, dateLayout); newTitle != "" && newTitle != doc.Title {
|
if newTitle := titleFromOCRText(ocrText, prefix, dateLayout); newTitle != "" && newTitle != doc.Title {
|
||||||
if err := s.store.UpdateDocumentTitleAuto(ctx, documentID, tenantID, newTitle); err != nil {
|
if err := s.store.UpdateDocumentTitleAuto(ctx, documentID, tenantID, newTitle); err != nil {
|
||||||
s.logger.Warn("jobqueue title update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("jobqueue title update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
} else {
|
} else {
|
||||||
doc.Title = newTitle
|
doc.Title = newTitle
|
||||||
}
|
}
|
||||||
@@ -1286,7 +1267,7 @@ func (s *Server) ProcessDocumentJob(ctx context.Context, tenantID, documentID in
|
|||||||
}
|
}
|
||||||
if !sameDate(datePtr, doc.DocumentDate) {
|
if !sameDate(datePtr, doc.DocumentDate) {
|
||||||
if err := s.store.UpdateDocumentDate(ctx, documentID, tenantID, datePtr, scorePtr); err != nil {
|
if err := s.store.UpdateDocumentDate(ctx, documentID, tenantID, datePtr, scorePtr); err != nil {
|
||||||
s.logger.Warn("jobqueue document_date update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("jobqueue document_date update failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
} else {
|
} else {
|
||||||
doc.DocumentDate = datePtr
|
doc.DocumentDate = datePtr
|
||||||
doc.DocumentDateScore = scorePtr
|
doc.DocumentDateScore = scorePtr
|
||||||
@@ -1295,11 +1276,11 @@ func (s *Server) ProcessDocumentJob(ctx context.Context, tenantID, documentID in
|
|||||||
}
|
}
|
||||||
|
|
||||||
if assignWarn := s.autoAssignTaxonomy(ctx, tenantID, doc, barcodes, ocrText); assignWarn != "" {
|
if assignWarn := s.autoAssignTaxonomy(ctx, tenantID, doc, barcodes, ocrText); assignWarn != "" {
|
||||||
s.logger.Info("jobqueue auto-assignment", "document_id", doc.ID, "tenant_id", tenantID, "detail", assignWarn)
|
s.reqLog(ctx).Info("jobqueue auto-assignment", "document_id", doc.ID, "tenant_id", tenantID, "detail", assignWarn)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := s.store.RunWorkflowsForDocument(ctx, tenantID, doc, storage.WorkflowTriggerOnUpload); err != nil {
|
if err := s.store.RunWorkflowsForDocument(ctx, tenantID, doc, storage.WorkflowTriggerOnUpload); err != nil {
|
||||||
s.logger.Warn("jobqueue workflow execution failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
s.reqLog(ctx).Warn("jobqueue workflow execution failed", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Final index sync: auto-assignment/workflows may have re-indexed already,
|
// Final index sync: auto-assignment/workflows may have re-indexed already,
|
||||||
@@ -1323,7 +1304,7 @@ func (s *Server) autoAssignTaxonomy(ctx context.Context, tenantID int64, doc *st
|
|||||||
// Nachvollziehbarkeit even when nothing matched.
|
// Nachvollziehbarkeit even when nothing matched.
|
||||||
if len(barcodes) > 0 {
|
if len(barcodes) > 0 {
|
||||||
if err := s.store.SetDocumentBarcodeValues(ctx, doc.ID, tenantID, barcodes); err != nil {
|
if err := s.store.SetDocumentBarcodeValues(ctx, doc.ID, tenantID, barcodes); err != nil {
|
||||||
s.logger.Warn("failed to store barcode values", "document_id", doc.ID, "err", err)
|
s.reqLog(ctx).Warn("failed to store barcode values", "document_id", doc.ID, "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1371,7 +1352,7 @@ func (s *Server) autoAssignTaxonomy(ctx context.Context, tenantID int64, doc *st
|
|||||||
for _, kind := range taxonomyKinds {
|
for _, kind := range taxonomyKinds {
|
||||||
entities, err := s.store.ListActiveMatchers(ctx, kind, tenantID)
|
entities, err := s.store.ListActiveMatchers(ctx, kind, tenantID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Warn("failed to list active matchers", "kind", kind, "err", err)
|
s.reqLog(ctx).Warn("failed to list active matchers", "kind", kind, "err", err)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
for _, entity := range entities {
|
for _, entity := range entities {
|
||||||
@@ -1537,31 +1518,9 @@ func randomUploadID() string {
|
|||||||
return hex.EncodeToString(b)
|
return hex.EncodeToString(b)
|
||||||
}
|
}
|
||||||
|
|
||||||
// copyFile is the cross-device fallback for os.Rename (EXDEV): copy + fsync
|
// The cross-device copy fallback for the WORM move lives in
|
||||||
// + remove the source.
|
// internal/objectstore (copyFile there) since FDN-03 moved the archival step
|
||||||
func copyFile(src, dst string) error {
|
// behind the storage abstraction.
|
||||||
in, err := os.Open(src)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
defer in.Close()
|
|
||||||
|
|
||||||
out, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o640)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if _, err := io.Copy(out, in); err != nil {
|
|
||||||
out.Close()
|
|
||||||
os.Remove(dst)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := out.Sync(); err != nil {
|
|
||||||
out.Close()
|
|
||||||
os.Remove(dst)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return out.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
// ocrSupportedMimeTypes is the whitelist of MIME types the OCR pipeline
|
// ocrSupportedMimeTypes is the whitelist of MIME types the OCR pipeline
|
||||||
// (internal/ocr.Extract) actually dispatches on. A declared Content-Type is
|
// (internal/ocr.Extract) actually dispatches on. A declared Content-Type is
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
// FDN-08 — /metrics im Prometheus-Textformat, ohne Fremdabhängigkeit.
|
||||||
|
//
|
||||||
|
// Zugriff: bewusst OHNE Login (Scrape-Clients haben keine Session), dafür
|
||||||
|
// IP-beschränkt. Default ist loopback-only; weitere Scraper werden über
|
||||||
|
// config api.metrics_allowed_ips (IP oder CIDR) freigeschaltet. Es werden
|
||||||
|
// ausschließlich aggregierte Zähler ausgegeben — keine Tenant-Daten, keine
|
||||||
|
// Pfadsegmente mit IDs oder Tokens (siehe normalizeRoute).
|
||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"runtime"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// handleMetrics rendert die Registry im Prometheus-Textformat.
|
||||||
|
func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !s.metricsAllowed(r) {
|
||||||
|
writeError(w, http.StatusForbidden, "metrics endpoint not allowed from this address")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
routes, inFlight, panics := s.metrics.snapshot()
|
||||||
|
|
||||||
|
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
|
||||||
|
var b strings.Builder
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_build_info Statische Build-Information.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_build_info gauge\n")
|
||||||
|
fmt.Fprintf(&b, "archivdms_build_info{version=\"%s\"} 1\n", escapeLabel(s.appVersion))
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_uptime_seconds Laufzeit des Prozesses in Sekunden.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_uptime_seconds gauge\n")
|
||||||
|
fmt.Fprintf(&b, "archivdms_uptime_seconds %.3f\n", time.Since(s.startTime).Seconds())
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_goroutines Aktuelle Anzahl Goroutinen.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_goroutines gauge\n")
|
||||||
|
fmt.Fprintf(&b, "archivdms_goroutines %d\n", runtime.NumGoroutine())
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_http_requests_in_flight Aktuell laufende HTTP-Anfragen.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_http_requests_in_flight gauge\n")
|
||||||
|
fmt.Fprintf(&b, "archivdms_http_requests_in_flight %d\n", inFlight)
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_panics_total Zentral abgefangene Panics (unbehandelte Fehler).\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_panics_total counter\n")
|
||||||
|
fmt.Fprintf(&b, "archivdms_panics_total %d\n", panics)
|
||||||
|
|
||||||
|
// Requests + Latenz je Route/Status.
|
||||||
|
keys := make([]routeKey, 0, len(routes))
|
||||||
|
for k := range routes {
|
||||||
|
keys = append(keys, k)
|
||||||
|
}
|
||||||
|
sort.Slice(keys, func(i, j int) bool {
|
||||||
|
if keys[i].route != keys[j].route {
|
||||||
|
return keys[i].route < keys[j].route
|
||||||
|
}
|
||||||
|
if keys[i].method != keys[j].method {
|
||||||
|
return keys[i].method < keys[j].method
|
||||||
|
}
|
||||||
|
return keys[i].status < keys[j].status
|
||||||
|
})
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_http_requests_total Anzahl HTTP-Anfragen je Route und Status.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_http_requests_total counter\n")
|
||||||
|
for _, k := range keys {
|
||||||
|
st := routes[k]
|
||||||
|
fmt.Fprintf(&b, "archivdms_http_requests_total{method=\"%s\",route=\"%s\",status=\"%d\"} %d\n",
|
||||||
|
escapeLabel(k.method), escapeLabel(k.route), k.status, st.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_http_request_duration_seconds Latenz der HTTP-Anfragen.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_http_request_duration_seconds histogram\n")
|
||||||
|
for _, k := range keys {
|
||||||
|
st := routes[k]
|
||||||
|
labels := fmt.Sprintf("method=\"%s\",route=\"%s\",status=\"%d\"",
|
||||||
|
escapeLabel(k.method), escapeLabel(k.route), k.status)
|
||||||
|
for i, ub := range latencyBuckets {
|
||||||
|
fmt.Fprintf(&b, "archivdms_http_request_duration_seconds_bucket{%s,le=\"%g\"} %d\n",
|
||||||
|
labels, ub, st.bucketCount[i])
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, "archivdms_http_request_duration_seconds_bucket{%s,le=\"+Inf\"} %d\n", labels, st.count)
|
||||||
|
fmt.Fprintf(&b, "archivdms_http_request_duration_seconds_sum{%s} %.6f\n", labels, st.sumSeconds)
|
||||||
|
fmt.Fprintf(&b, "archivdms_http_request_duration_seconds_count{%s} %d\n", labels, st.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Queue-Länge (Akzeptanzkriterium 2). Fehler hier dürfen den Scrape nicht
|
||||||
|
// scheitern lassen — dann fehlt die Metrik einfach für diesen Durchlauf.
|
||||||
|
if s.store != nil {
|
||||||
|
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
counts, err := s.store.CountProcessingJobsByStatus(ctx)
|
||||||
|
if err != nil {
|
||||||
|
s.reqLog(r.Context()).Warn("metrics: queue length query failed", "err", err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(&b, "# HELP archivdms_processing_jobs Länge der Verarbeitungswarteschlange je Status.\n")
|
||||||
|
fmt.Fprintf(&b, "# TYPE archivdms_processing_jobs gauge\n")
|
||||||
|
statuses := make([]string, 0, len(counts))
|
||||||
|
for st := range counts {
|
||||||
|
statuses = append(statuses, st)
|
||||||
|
}
|
||||||
|
sort.Strings(statuses)
|
||||||
|
for _, st := range statuses {
|
||||||
|
fmt.Fprintf(&b, "archivdms_processing_jobs{status=\"%s\"} %d\n", escapeLabel(st), counts[st])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Write([]byte(b.String()))
|
||||||
|
}
|
||||||
|
|
||||||
|
// metricsAllowed prüft die Herkunft des Scrape-Requests: loopback immer,
|
||||||
|
// sonst nur konfigurierte IPs/CIDRs (config api.metrics_allowed_ips).
|
||||||
|
func (s *Server) metricsAllowed(r *http.Request) bool {
|
||||||
|
host, _, err := net.SplitHostPort(r.RemoteAddr)
|
||||||
|
if err != nil {
|
||||||
|
host = r.RemoteAddr
|
||||||
|
}
|
||||||
|
ip := net.ParseIP(strings.TrimSpace(host))
|
||||||
|
if ip == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if ip.IsLoopback() {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, entry := range s.cfg.MetricsAllowedIPs {
|
||||||
|
entry = strings.TrimSpace(entry)
|
||||||
|
if entry == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if strings.Contains(entry, "/") {
|
||||||
|
if _, cidr, err := net.ParseCIDR(entry); err == nil && cidr.Contains(ip) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if parsed := net.ParseIP(entry); parsed != nil && parsed.Equal(ip) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// escapeLabel entschärft Anführungszeichen/Backslashes/Zeilenumbrüche in
|
||||||
|
// Prometheus-Labelwerten.
|
||||||
|
func escapeLabel(v string) string {
|
||||||
|
r := strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`)
|
||||||
|
return r.Replace(v)
|
||||||
|
}
|
||||||
@@ -0,0 +1,404 @@
|
|||||||
|
// FDN-08 — Logging, Metriken & Fehler-Tracking.
|
||||||
|
//
|
||||||
|
// Dieses File enthält die drei Bausteine, die bisher gefehlt haben:
|
||||||
|
//
|
||||||
|
// 1. requestIDMiddleware: erzeugt (oder übernimmt aus X-Request-ID) eine
|
||||||
|
// Korrelations-ID, hängt sie an den context.Context und gibt sie im
|
||||||
|
// Response-Header zurück. Über loggerFromCtx(ctx) bekommt jede Log-Zeile
|
||||||
|
// im Request-Lebenszyklus das Feld request_id, ohne dass jede Call-Site
|
||||||
|
// umgeschrieben werden muss.
|
||||||
|
// 2. metricsMiddleware: zählt Requests je (Methode, normalisierter Pfad,
|
||||||
|
// Status) und summiert die Latenz in Histogramm-Buckets. Ausgabe über
|
||||||
|
// GET /metrics im Prometheus-Textformat (kein prometheus/client_golang).
|
||||||
|
// 3. recoverMiddleware: zentrales Panic-Recovery. net/http's ServeMux hat
|
||||||
|
// keins; ohne das reißt ein Panic in einem Handler die Verbindung ab und
|
||||||
|
// der Fehler taucht nirgends auf.
|
||||||
|
//
|
||||||
|
// Bewusst KEIN Logging von Query-Strings, Headern oder Request-Bodies:
|
||||||
|
// dort stecken Tokens (Signed-URL-Signatur, Share-Token, Bearer-Keys) und
|
||||||
|
// potenziell Passwörter. Geloggt werden ausschließlich Methode, normalisierter
|
||||||
|
// Pfad (IDs/Tokens durch Platzhalter ersetzt), Status, Dauer und Client-IP.
|
||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"runtime/debug"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stackTrace liefert einen gekürzten Stacktrace für das Panic-Log.
|
||||||
|
func stackTrace() string {
|
||||||
|
const maxStack = 4096
|
||||||
|
buf := debug.Stack()
|
||||||
|
if len(buf) > maxStack {
|
||||||
|
buf = buf[:maxStack]
|
||||||
|
}
|
||||||
|
return string(buf)
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
requestIDKey contextKey = "request_id"
|
||||||
|
loggerKey contextKey = "logger"
|
||||||
|
requestIDHeader = "X-Request-ID"
|
||||||
|
maxRequestIDLen = 64
|
||||||
|
)
|
||||||
|
|
||||||
|
// newRequestID erzeugt eine zufällige 16-stellige Hex-ID. Fällt bei einem
|
||||||
|
// (praktisch unmöglichen) Fehler der Entropiequelle auf einen Zeitstempel
|
||||||
|
// zurück — eine Anfrage darf daran nie scheitern.
|
||||||
|
func newRequestID() string {
|
||||||
|
buf := make([]byte, 8)
|
||||||
|
if _, err := rand.Read(buf); err != nil {
|
||||||
|
return strconv.FormatInt(time.Now().UnixNano(), 36)
|
||||||
|
}
|
||||||
|
return hex.EncodeToString(buf)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sanitizeRequestID übernimmt eine vom Client/Proxy gelieferte ID nur, wenn
|
||||||
|
// sie kurz und druckbar-alphanumerisch ist. Verhindert Log-Injection über
|
||||||
|
// Zeilenumbrüche im Header.
|
||||||
|
func sanitizeRequestID(v string) string {
|
||||||
|
v = strings.TrimSpace(v)
|
||||||
|
if v == "" || len(v) > maxRequestIDLen {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
for _, r := range v {
|
||||||
|
switch {
|
||||||
|
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
|
||||||
|
case r == '-' || r == '_' || r == '.':
|
||||||
|
default:
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestIDFromCtx liefert die Korrelations-ID der laufenden Anfrage oder "".
|
||||||
|
func requestIDFromCtx(ctx context.Context) string {
|
||||||
|
if ctx == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
v, _ := ctx.Value(requestIDKey).(string)
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// loggerFromCtx liefert den Request-Logger (inkl. request_id-Feld). Außerhalb
|
||||||
|
// eines HTTP-Requests — oder wenn kein Logger hinterlegt wurde — kommt ein
|
||||||
|
// no-op-freier Fallback zurück, damit Aufrufer nie auf nil prüfen müssen.
|
||||||
|
func loggerFromCtx(ctx context.Context) *slog.Logger {
|
||||||
|
if ctx != nil {
|
||||||
|
if l, ok := ctx.Value(loggerKey).(*slog.Logger); ok && l != nil {
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return slog.Default()
|
||||||
|
}
|
||||||
|
|
||||||
|
// reqLog ist der Einstieg für Handler: nutzt den Request-Logger aus dem
|
||||||
|
// Context, fällt aber auf den Server-Logger zurück, wenn die Middleware nicht
|
||||||
|
// durchlaufen wurde (z.B. in Tests, die Handler direkt aufrufen).
|
||||||
|
func (s *Server) reqLog(ctx context.Context) *slog.Logger {
|
||||||
|
if ctx != nil {
|
||||||
|
if l, ok := ctx.Value(loggerKey).(*slog.Logger); ok && l != nil {
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.logger != nil {
|
||||||
|
return s.logger
|
||||||
|
}
|
||||||
|
return slog.Default()
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestIDMiddleware setzt die Korrelations-ID und den Request-Logger.
|
||||||
|
func (s *Server) requestIDMiddleware(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
rid := sanitizeRequestID(r.Header.Get(requestIDHeader))
|
||||||
|
if rid == "" {
|
||||||
|
rid = newRequestID()
|
||||||
|
}
|
||||||
|
base := s.logger
|
||||||
|
if base == nil {
|
||||||
|
base = slog.Default()
|
||||||
|
}
|
||||||
|
reqLogger := base.With("request_id", rid)
|
||||||
|
|
||||||
|
ctx := context.WithValue(r.Context(), requestIDKey, rid)
|
||||||
|
ctx = context.WithValue(ctx, loggerKey, reqLogger)
|
||||||
|
|
||||||
|
w.Header().Set(requestIDHeader, rid)
|
||||||
|
next.ServeHTTP(w, r.WithContext(ctx))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// statusRecorder merkt sich Statuscode und geschriebene Bytes, damit die
|
||||||
|
// Metrik-Middleware nach dem Handler auswerten kann.
|
||||||
|
type statusRecorder struct {
|
||||||
|
http.ResponseWriter
|
||||||
|
status int
|
||||||
|
written int64
|
||||||
|
wrote bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (rec *statusRecorder) WriteHeader(code int) {
|
||||||
|
if !rec.wrote {
|
||||||
|
rec.status = code
|
||||||
|
rec.wrote = true
|
||||||
|
}
|
||||||
|
rec.ResponseWriter.WriteHeader(code)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (rec *statusRecorder) Write(b []byte) (int, error) {
|
||||||
|
if !rec.wrote {
|
||||||
|
rec.status = http.StatusOK
|
||||||
|
rec.wrote = true
|
||||||
|
}
|
||||||
|
n, err := rec.ResponseWriter.Write(b)
|
||||||
|
rec.written += int64(n)
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flush reicht http.Flusher durch (Downloads/Streaming-Endpunkte).
|
||||||
|
func (rec *statusRecorder) Flush() {
|
||||||
|
if f, ok := rec.ResponseWriter.(http.Flusher); ok {
|
||||||
|
f.Flush()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// recoverMiddleware fängt Panics zentral ab: Log mit Korrelations-ID +
|
||||||
|
// Stacktrace, sauberer 500 an den Client. Ohne das stürzt zwar nicht der
|
||||||
|
// Prozess (net/http fängt pro Verbindung ab), der Fehler bleibt aber
|
||||||
|
// unsichtbar und der Client bekommt einen abgebrochenen Stream.
|
||||||
|
func (s *Server) recoverMiddleware(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer func() {
|
||||||
|
rec := recover()
|
||||||
|
if rec == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if rec == http.ErrAbortHandler {
|
||||||
|
panic(rec)
|
||||||
|
}
|
||||||
|
s.metrics.incPanic()
|
||||||
|
loggerFromCtx(r.Context()).Error("panic in http handler",
|
||||||
|
"method", r.Method,
|
||||||
|
"path", normalizeRoute(r.URL.Path),
|
||||||
|
"remote_ip", s.remoteIP(r),
|
||||||
|
"panic", rec,
|
||||||
|
"stack", stackTrace(),
|
||||||
|
)
|
||||||
|
if sr, ok := w.(*statusRecorder); ok && sr.wrote {
|
||||||
|
return // Header sind raus, mehr geht nicht
|
||||||
|
}
|
||||||
|
writeError(w, http.StatusInternalServerError, "interner Serverfehler")
|
||||||
|
}()
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// metricsMiddleware misst Dauer und Status jeder Anfrage.
|
||||||
|
func (s *Server) metricsMiddleware(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}
|
||||||
|
start := time.Now()
|
||||||
|
s.metrics.incInFlight()
|
||||||
|
defer func() {
|
||||||
|
s.metrics.decInFlight()
|
||||||
|
d := time.Since(start)
|
||||||
|
route := normalizeRoute(r.URL.Path)
|
||||||
|
s.metrics.observe(r.Method, route, rec.status, d)
|
||||||
|
// Zugriffs-Log: JEDE Anfrage erzeugt genau eine Zeile mit
|
||||||
|
// request_id (AK1). Level nach Status gestaffelt, damit im
|
||||||
|
// Normalbetrieb (Info) nur Auffälligkeiten sichtbar sind:
|
||||||
|
// 5xx=Error, 4xx=Warn, Rest=Debug.
|
||||||
|
lvl := slog.LevelDebug
|
||||||
|
msg := "request completed"
|
||||||
|
switch {
|
||||||
|
case rec.status >= 500:
|
||||||
|
lvl, msg = slog.LevelError, "request failed"
|
||||||
|
case rec.status >= 400:
|
||||||
|
lvl, msg = slog.LevelWarn, "request rejected"
|
||||||
|
}
|
||||||
|
loggerFromCtx(r.Context()).Log(r.Context(), lvl, msg,
|
||||||
|
"method", r.Method, "route", route,
|
||||||
|
"status", rec.status, "duration_ms", d.Milliseconds(),
|
||||||
|
"bytes", rec.written,
|
||||||
|
"remote_ip", s.remoteIP(r))
|
||||||
|
}()
|
||||||
|
next.ServeHTTP(rec, r)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Pfad-Normalisierung ---
|
||||||
|
|
||||||
|
// tokenSegments sind Pfadabschnitte, deren FOLGENDES Segment ein Geheimnis
|
||||||
|
// ist (Share-Token). Die dürfen niemals in Logs oder Metrik-Labels landen.
|
||||||
|
var tokenSegments = map[string]bool{"share": true}
|
||||||
|
|
||||||
|
// normalizeRoute ersetzt variable Pfadsegmente durch Platzhalter. Das hält
|
||||||
|
// die Label-Kardinalität der Metriken klein UND verhindert, dass IDs oder
|
||||||
|
// Share-Tokens in Logs/Metriken auftauchen. Query-Strings werden nie
|
||||||
|
// betrachtet (dort stehen Signaturen und API-Keys).
|
||||||
|
func normalizeRoute(path string) string {
|
||||||
|
if path == "" {
|
||||||
|
return "/"
|
||||||
|
}
|
||||||
|
parts := strings.Split(path, "/")
|
||||||
|
prevSecret := false
|
||||||
|
for i, p := range parts {
|
||||||
|
if p == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if prevSecret {
|
||||||
|
parts[i] = "{token}"
|
||||||
|
prevSecret = false
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
prevSecret = tokenSegments[p]
|
||||||
|
if isVariableSegment(p) {
|
||||||
|
parts[i] = "{id}"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out := strings.Join(parts, "/")
|
||||||
|
if len(out) > 120 {
|
||||||
|
return "/other"
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// isVariableSegment erkennt IDs, Hashes und sonstige nicht-statische
|
||||||
|
// Pfadbestandteile. Konservativ: im Zweifel maskieren.
|
||||||
|
func isVariableSegment(seg string) bool {
|
||||||
|
if _, err := strconv.ParseInt(seg, 10, 64); err == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if len(seg) > 40 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
hasDigit := false
|
||||||
|
for _, r := range seg {
|
||||||
|
switch {
|
||||||
|
case r >= '0' && r <= '9':
|
||||||
|
hasDigit = true
|
||||||
|
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r == '-', r == '_', r == '.':
|
||||||
|
default:
|
||||||
|
// Alles Ungewöhnliche (Sonderzeichen, Umlaute, %-Encoding) ist mit
|
||||||
|
// Sicherheit kein statisches Routensegment.
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Statische Routensegmente sind reine Wörter ("delete-requests",
|
||||||
|
// "classification-templates"). Längere Mischungen aus Buchstaben und
|
||||||
|
// Ziffern sind Hashes/Tokens/Dateinamen -> maskieren.
|
||||||
|
return hasDigit && len(seg) >= 8
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Metrik-Registry ---
|
||||||
|
|
||||||
|
// latencyBuckets sind die oberen Grenzen (Sekunden) des Latenz-Histogramms.
|
||||||
|
var latencyBuckets = []float64{0.005, 0.025, 0.1, 0.5, 1, 2.5, 5, 10, 30}
|
||||||
|
|
||||||
|
type routeKey struct {
|
||||||
|
method string
|
||||||
|
route string
|
||||||
|
status int
|
||||||
|
}
|
||||||
|
|
||||||
|
type routeStat struct {
|
||||||
|
count uint64
|
||||||
|
sumSeconds float64
|
||||||
|
bucketCount []uint64 // len(latencyBuckets), kumulativ erst beim Rendern
|
||||||
|
}
|
||||||
|
|
||||||
|
// metricsRegistry ist eine minimale, prozesslokale Metrik-Sammlung. Keine
|
||||||
|
// globale Variable: hängt als Feld am Server (Dependency Injection).
|
||||||
|
type metricsRegistry struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
routes map[routeKey]*routeStat
|
||||||
|
inFlight int64
|
||||||
|
panics uint64
|
||||||
|
started time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMetricsRegistry() *metricsRegistry {
|
||||||
|
return &metricsRegistry{
|
||||||
|
routes: make(map[routeKey]*routeStat),
|
||||||
|
started: time.Now(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metricsRegistry) observe(method, route string, status int, d time.Duration) {
|
||||||
|
if m == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
k := routeKey{method: method, route: route, status: status}
|
||||||
|
secs := d.Seconds()
|
||||||
|
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
// Kardinalitätsbremse: unbekannte Pfade nicht unbegrenzt sammeln.
|
||||||
|
st := m.routes[k]
|
||||||
|
if st == nil {
|
||||||
|
if len(m.routes) >= 500 {
|
||||||
|
k = routeKey{method: method, route: "/other", status: status}
|
||||||
|
st = m.routes[k]
|
||||||
|
}
|
||||||
|
if st == nil {
|
||||||
|
st = &routeStat{bucketCount: make([]uint64, len(latencyBuckets))}
|
||||||
|
m.routes[k] = st
|
||||||
|
}
|
||||||
|
}
|
||||||
|
st.count++
|
||||||
|
st.sumSeconds += secs
|
||||||
|
for i, ub := range latencyBuckets {
|
||||||
|
if secs <= ub {
|
||||||
|
st.bucketCount[i]++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metricsRegistry) incInFlight() {
|
||||||
|
if m == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.mu.Lock()
|
||||||
|
m.inFlight++
|
||||||
|
m.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metricsRegistry) decInFlight() {
|
||||||
|
if m == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.mu.Lock()
|
||||||
|
m.inFlight--
|
||||||
|
m.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metricsRegistry) incPanic() {
|
||||||
|
if m == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
m.mu.Lock()
|
||||||
|
m.panics++
|
||||||
|
m.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// snapshot liefert eine Kopie der Zähler für das Rendern.
|
||||||
|
func (m *metricsRegistry) snapshot() (map[routeKey]routeStat, int64, uint64) {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
out := make(map[routeKey]routeStat, len(m.routes))
|
||||||
|
for k, v := range m.routes {
|
||||||
|
cp := routeStat{count: v.count, sumSeconds: v.sumSeconds, bucketCount: make([]uint64, len(v.bucketCount))}
|
||||||
|
copy(cp.bucketCount, v.bucketCount)
|
||||||
|
out[k] = cp
|
||||||
|
}
|
||||||
|
return out, m.inFlight, m.panics
|
||||||
|
}
|
||||||
@@ -0,0 +1,174 @@
|
|||||||
|
// FDN-08 — Tests zu den drei Akzeptanzkriterien:
|
||||||
|
// 1. Korrelations-ID über alle Schichten
|
||||||
|
// 2. Metriken (Latenz, Fehlerrate, Queue-Länge)
|
||||||
|
// 3. Unbehandelte Fehler werden zentral gemeldet
|
||||||
|
//
|
||||||
|
// Zusätzlich Prüfung 2 der Abnahme: es dürfen keine Tokens/Passwörter in Logs
|
||||||
|
// oder Metrik-Labels landen (TestNormalizeRouteRedactsSecrets).
|
||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"archivdms/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newTestServer(buf *bytes.Buffer) *Server {
|
||||||
|
logger := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
|
||||||
|
return New(config.APIConfig{}, nil, nil, nil, nil, logger)
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 1: jede Anfrage bekommt eine Korrelations-ID, eine vom Client gelieferte
|
||||||
|
// wird übernommen und im Response-Header zurückgegeben.
|
||||||
|
func TestRequestIDGeneratedAndEchoed(t *testing.T) {
|
||||||
|
srv := newTestServer(&bytes.Buffer{})
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/health", nil))
|
||||||
|
got := rec.Header().Get(requestIDHeader)
|
||||||
|
if got == "" {
|
||||||
|
t.Fatalf("erwartete generierte Request-ID im Header %s", requestIDHeader)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/health", nil)
|
||||||
|
req.Header.Set(requestIDHeader, "abc-123")
|
||||||
|
rec2 := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec2, req)
|
||||||
|
if rec2.Header().Get(requestIDHeader) != "abc-123" {
|
||||||
|
t.Fatalf("Client-Request-ID nicht übernommen: %q", rec2.Header().Get(requestIDHeader))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log-Injection: unsaubere IDs werden verworfen, nicht durchgereicht.
|
||||||
|
bad := httptest.NewRequest(http.MethodGet, "/api/health", nil)
|
||||||
|
bad.Header.Set(requestIDHeader, "evil\nid")
|
||||||
|
rec3 := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec3, bad)
|
||||||
|
if strings.Contains(rec3.Header().Get(requestIDHeader), "evil") {
|
||||||
|
t.Fatalf("ungültige Request-ID wurde übernommen: %q", rec3.Header().Get(requestIDHeader))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 1: die ID landet im Logger, den Handler über den Context ziehen.
|
||||||
|
func TestLoggerFromCtxCarriesRequestID(t *testing.T) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
srv := newTestServer(&buf)
|
||||||
|
|
||||||
|
h := srv.requestIDMiddleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if requestIDFromCtx(r.Context()) == "" {
|
||||||
|
t.Errorf("keine Request-ID im Context")
|
||||||
|
}
|
||||||
|
srv.reqLog(r.Context()).Info("testereignis")
|
||||||
|
}))
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/health", nil))
|
||||||
|
|
||||||
|
rid := rec.Header().Get(requestIDHeader)
|
||||||
|
if !strings.Contains(buf.String(), "request_id="+rid) {
|
||||||
|
t.Fatalf("Log-Zeile ohne request_id=%s: %s", rid, buf.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 2: Zähler und Latenz-Histogramm werden im Prometheus-Textformat geliefert.
|
||||||
|
func TestMetricsEndpoint(t *testing.T) {
|
||||||
|
srv := newTestServer(&bytes.Buffer{})
|
||||||
|
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
srv.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodGet, "/api/health", nil))
|
||||||
|
}
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/metrics", nil)
|
||||||
|
req.RemoteAddr = "127.0.0.1:54321"
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("erwartet 200, bekam %d", rec.Code)
|
||||||
|
}
|
||||||
|
body := rec.Body.String()
|
||||||
|
want := `archivdms_http_requests_total{method="GET",route="/api/health",status="200"} 2`
|
||||||
|
if !strings.Contains(body, want) {
|
||||||
|
t.Fatalf("Request-Zähler fehlt:\n%s", body)
|
||||||
|
}
|
||||||
|
for _, frag := range []string{
|
||||||
|
"archivdms_http_request_duration_seconds_bucket",
|
||||||
|
"archivdms_http_request_duration_seconds_sum",
|
||||||
|
"archivdms_goroutines",
|
||||||
|
"archivdms_panics_total",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(body, frag) {
|
||||||
|
t.Errorf("Metrik %q fehlt in der Ausgabe", frag)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 2 (Zugriffsschutz): fremde IPs dürfen nicht scrapen.
|
||||||
|
func TestMetricsEndpointIPRestricted(t *testing.T) {
|
||||||
|
srv := newTestServer(&bytes.Buffer{})
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/metrics", nil)
|
||||||
|
req.RemoteAddr = "203.0.113.7:5000"
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, req)
|
||||||
|
if rec.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("erwartet 403 für fremde IP, bekam %d", rec.Code)
|
||||||
|
}
|
||||||
|
|
||||||
|
srv.cfg.MetricsAllowedIPs = []string{"203.0.113.0/24"}
|
||||||
|
rec2 := httptest.NewRecorder()
|
||||||
|
req2 := httptest.NewRequest(http.MethodGet, "/metrics", nil)
|
||||||
|
req2.RemoteAddr = "203.0.113.7:5000"
|
||||||
|
srv.ServeHTTP(rec2, req2)
|
||||||
|
if rec2.Code != http.StatusOK {
|
||||||
|
t.Fatalf("erwartet 200 für freigeschaltete CIDR, bekam %d", rec2.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 3: ein Panic wird zentral abgefangen, mit Korrelations-ID geloggt und als
|
||||||
|
// sauberer 500 beantwortet — der Zähler archivdms_panics_total steigt.
|
||||||
|
func TestRecoverMiddlewareCatchesPanic(t *testing.T) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
srv := newTestServer(&buf)
|
||||||
|
|
||||||
|
boom := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
panic("kaputt")
|
||||||
|
})
|
||||||
|
h := srv.requestIDMiddleware(srv.metricsMiddleware(srv.recoverMiddleware(boom)))
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/documents/42", nil))
|
||||||
|
|
||||||
|
if rec.Code != http.StatusInternalServerError {
|
||||||
|
t.Fatalf("erwartet 500, bekam %d", rec.Code)
|
||||||
|
}
|
||||||
|
rid := rec.Header().Get(requestIDHeader)
|
||||||
|
logged := buf.String()
|
||||||
|
if !strings.Contains(logged, "panic in http handler") || !strings.Contains(logged, "request_id="+rid) {
|
||||||
|
t.Fatalf("Panic nicht mit Korrelations-ID geloggt: %s", logged)
|
||||||
|
}
|
||||||
|
if _, _, panics := srv.metrics.snapshot(); panics != 1 {
|
||||||
|
t.Fatalf("erwartet 1 gezähltes Panic, bekam %d", panics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Abnahme-Prüfung 2: keine Tokens/IDs in Logs oder Metrik-Labels.
|
||||||
|
func TestNormalizeRouteRedactsSecrets(t *testing.T) {
|
||||||
|
cases := [][2]string{
|
||||||
|
{"/api/health", "/api/health"},
|
||||||
|
{"/api/documents/42", "/api/documents/{id}"},
|
||||||
|
{"/api/documents/42/notes/7", "/api/documents/{id}/notes/{id}"},
|
||||||
|
{"/api/classification-templates", "/api/classification-templates"},
|
||||||
|
{"/api/trash/9/delete-requests", "/api/trash/{id}/delete-requests"},
|
||||||
|
{"/public/share/s3cr3tTokenXyz", "/public/share/{token}"},
|
||||||
|
{"/public/share/abc/download", "/public/share/{token}/download"},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
in, want := c[0], c[1]
|
||||||
|
if got := normalizeRoute(in); got != want {
|
||||||
|
t.Errorf("normalizeRoute(%q) = %q, erwartet %q", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -51,7 +51,7 @@ func (s *Server) handleListDocumentOCRWords(w http.ResponseWriter, r *http.Reque
|
|||||||
|
|
||||||
words, err := s.store.ListOCRWords(r.Context(), id)
|
words, err := s.store.ListOCRWords(r.Context(), id)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("list ocr words failed", "document_id", id, "tenant_id", *sess.TenantID, "err", err)
|
s.reqLog(r.Context()).Error("list ocr words failed", "document_id", id, "tenant_id", *sess.TenantID, "err", err)
|
||||||
writeError(w, http.StatusInternalServerError, "list ocr words failed")
|
writeError(w, http.StatusInternalServerError, "list ocr words failed")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,7 +15,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -107,7 +106,7 @@ func (s *Server) handlePublicShareDownload(w http.ResponseWriter, r *http.Reques
|
|||||||
// Atomically claim one access slot (closes the max_accesses race).
|
// Atomically claim one access slot (closes the max_accesses race).
|
||||||
ok, err := s.store.IncrementShareAccess(r.Context(), rs.ShareID())
|
ok, err := s.store.IncrementShareAccess(r.Context(), rs.ShareID())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("share access increment failed", "share_id", rs.ShareID(), "err", err)
|
s.reqLog(r.Context()).Error("share access increment failed", "share_id", rs.ShareID(), "err", err)
|
||||||
writeError(w, http.StatusInternalServerError, "download failed")
|
writeError(w, http.StatusInternalServerError, "download failed")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -118,9 +117,9 @@ func (s *Server) handlePublicShareDownload(w http.ResponseWriter, r *http.Reques
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
f, err := os.Open(rs.StoragePath())
|
f, err := s.objects.Open(r.Context(), rs.TenantID(), rs.StoragePath())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("share file open failed", "share_id", rs.ShareID(), "err", err)
|
s.reqLog(r.Context()).Error("share file open failed", "share_id", rs.ShareID(), "err", err)
|
||||||
writeError(w, http.StatusInternalServerError, "download failed")
|
writeError(w, http.StatusInternalServerError, "download failed")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -133,7 +132,7 @@ func (s *Server) handlePublicShareDownload(w http.ResponseWriter, r *http.Reques
|
|||||||
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(rs.DocumentTitle, ext)+"\"")
|
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(rs.DocumentTitle, ext)+"\"")
|
||||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
if _, err := io.Copy(w, f); err != nil {
|
if _, err := io.Copy(w, f); err != nil {
|
||||||
s.logger.Warn("share file stream interrupted", "share_id", rs.ShareID(), "err", err)
|
s.reqLog(r.Context()).Warn("share file stream interrupted", "share_id", rs.ShareID(), "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -141,7 +140,7 @@ func (s *Server) handlePublicShareDownload(w http.ResponseWriter, r *http.Reques
|
|||||||
// into the audit log (EventShareAccessed). Never blocks the response path.
|
// into the audit log (EventShareAccessed). Never blocks the response path.
|
||||||
func (s *Server) recordShareAccess(r *http.Request, rs *storage.ResolvedShare, ip, result string) {
|
func (s *Server) recordShareAccess(r *http.Request, rs *storage.ResolvedShare, ip, result string) {
|
||||||
if err := s.store.LogShareAccess(r.Context(), rs.ShareID(), ip, r.UserAgent(), result); err != nil {
|
if err := s.store.LogShareAccess(r.Context(), rs.ShareID(), ip, r.UserAgent(), result); err != nil {
|
||||||
s.logger.Error("share access log failed", "share_id", rs.ShareID(), "err", err)
|
s.reqLog(r.Context()).Error("share access log failed", "share_id", rs.ShareID(), "err", err)
|
||||||
}
|
}
|
||||||
tenantID := rs.TenantID()
|
tenantID := rs.TenantID()
|
||||||
s.audlog.Log(audit.Entry{
|
s.audlog.Log(audit.Entry{
|
||||||
|
|||||||
+83
-1
@@ -6,10 +6,13 @@ package api
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -19,6 +22,7 @@ import (
|
|||||||
"archivdms/internal/ldapauth"
|
"archivdms/internal/ldapauth"
|
||||||
"archivdms/internal/ldapstore"
|
"archivdms/internal/ldapstore"
|
||||||
"archivdms/internal/mailer"
|
"archivdms/internal/mailer"
|
||||||
|
"archivdms/internal/objectstore"
|
||||||
"archivdms/internal/ocr"
|
"archivdms/internal/ocr"
|
||||||
"archivdms/internal/pagesplit"
|
"archivdms/internal/pagesplit"
|
||||||
"archivdms/internal/storage"
|
"archivdms/internal/storage"
|
||||||
@@ -46,6 +50,10 @@ type Server struct {
|
|||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
mux *http.ServeMux
|
mux *http.ServeMux
|
||||||
ocr *ocr.Extractor
|
ocr *ocr.Extractor
|
||||||
|
// objects is the WORM object-storage driver (internal/objectstore): the
|
||||||
|
// only place archived files are written, read or unlinked. Wired by
|
||||||
|
// SetStorageConfig/SetObjectStore.
|
||||||
|
objects objectstore.Store
|
||||||
thumbs *thumbnail.Generator
|
thumbs *thumbnail.Generator
|
||||||
// pagesplitter performs barcode separator-page splitting of multi-page PDF
|
// pagesplitter performs barcode separator-page splitting of multi-page PDF
|
||||||
// uploads before archival (internal/pagesplit). May be nil / disabled, in
|
// uploads before archival (internal/pagesplit). May be nil / disabled, in
|
||||||
@@ -66,6 +74,15 @@ type Server struct {
|
|||||||
// (per client IP) to blunt token/password enumeration.
|
// (per client IP) to blunt token/password enumeration.
|
||||||
shareLimiter *ipRateLimiter
|
shareLimiter *ipRateLimiter
|
||||||
|
|
||||||
|
// metrics ist die prozesslokale Metrik-Registry (FDN-08,
|
||||||
|
// internal/api/observability.go). Kein globaler Zustand: hängt am Server.
|
||||||
|
metrics *metricsRegistry
|
||||||
|
|
||||||
|
// baseHandler ist die in New() gebaute Middleware-Kette um s.mux
|
||||||
|
// (requestID -> metrics -> recover). Einmal gebaut, danach nur gelesen —
|
||||||
|
// kein Lazy-Init in ServeHTTP (Data Race).
|
||||||
|
baseHandler http.Handler
|
||||||
|
|
||||||
// accountingLimiter rate-limits the Bearer-key Buchhaltungs-Pull-API
|
// accountingLimiter rate-limits the Bearer-key Buchhaltungs-Pull-API
|
||||||
// (per client IP) to blunt API-key guessing. Separate bucket set from
|
// (per client IP) to blunt API-key guessing. Separate bucket set from
|
||||||
// shareLimiter so a busy accounting client cannot starve share downloads.
|
// shareLimiter so a busy accounting client cannot starve share downloads.
|
||||||
@@ -76,8 +93,46 @@ type Server struct {
|
|||||||
// paths, max upload size) into the API server. Needed by
|
// paths, max upload size) into the API server. Needed by
|
||||||
// handleUploadDocument, which cannot rely solely on the storage.Store
|
// handleUploadDocument, which cannot rely solely on the storage.Store
|
||||||
// (that only knows its own base dir, not the inbox/ocr-tmp layout).
|
// (that only knows its own base dir, not the inbox/ocr-tmp layout).
|
||||||
|
//
|
||||||
|
// It also constructs the object-storage driver (internal/objectstore), the
|
||||||
|
// single place where archived files are written, read and unlinked. Call
|
||||||
|
// SetFQDN before this if generated signed URLs should be absolute.
|
||||||
func (s *Server) SetStorageConfig(cfg config.StorageConfig) {
|
func (s *Server) SetStorageConfig(cfg config.StorageConfig) {
|
||||||
s.storageCfg = cfg
|
s.storageCfg = cfg
|
||||||
|
secret := s.cfg.Secret
|
||||||
|
if strings.TrimSpace(secret) == "" {
|
||||||
|
// No master secret configured: fall back to an ephemeral, per-process
|
||||||
|
// key so file access keeps working; signed URLs then simply do not
|
||||||
|
// survive a restart. Never fail startup over this.
|
||||||
|
buf := make([]byte, 32)
|
||||||
|
if _, err := rand.Read(buf); err != nil {
|
||||||
|
secret = strconv.FormatInt(time.Now().UnixNano(), 36)
|
||||||
|
} else {
|
||||||
|
secret = hex.EncodeToString(buf)
|
||||||
|
}
|
||||||
|
s.logger.Warn("objectstore: api.secret unset, using ephemeral signing key")
|
||||||
|
}
|
||||||
|
driver, err := objectstore.NewLocalStore(cfg, secret, s.publicBaseURL())
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Error("objectstore init failed", "err", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.objects = driver
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetObjectStore overrides the object-storage driver (tests / alternative
|
||||||
|
// wiring). Normally set implicitly by SetStorageConfig.
|
||||||
|
func (s *Server) SetObjectStore(o objectstore.Store) {
|
||||||
|
s.objects = o
|
||||||
|
}
|
||||||
|
|
||||||
|
// publicBaseURL is the origin used for generated links. Empty FQDN yields
|
||||||
|
// site-relative URLs.
|
||||||
|
func (s *Server) publicBaseURL() string {
|
||||||
|
if strings.TrimSpace(s.fqdn) == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return "https://" + strings.TrimSuffix(strings.TrimSpace(s.fqdn), "/")
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetOCR wires the OCR extractor into the API server. May be nil, in which
|
// SetOCR wires the OCR extractor into the API server. May be nil, in which
|
||||||
@@ -150,8 +205,10 @@ func New(
|
|||||||
shareLimiter: newIPRateLimiter(20, 1.0),
|
shareLimiter: newIPRateLimiter(20, 1.0),
|
||||||
// Batch pulls are legitimate here: 60 requests burst, refilled at 5/sec.
|
// Batch pulls are legitimate here: 60 requests burst, refilled at 5/sec.
|
||||||
accountingLimiter: newIPRateLimiter(60, 5.0),
|
accountingLimiter: newIPRateLimiter(60, 5.0),
|
||||||
|
metrics: newMetricsRegistry(),
|
||||||
}
|
}
|
||||||
s.routes()
|
s.routes()
|
||||||
|
s.baseHandler = s.requestIDMiddleware(s.metricsMiddleware(s.recoverMiddleware(s.mux)))
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -168,6 +225,10 @@ func (s *Server) authAdmin(h http.HandlerFunc) http.HandlerFunc {
|
|||||||
func (s *Server) routes() {
|
func (s *Server) routes() {
|
||||||
s.mux.HandleFunc("GET /api/health", s.handleHealth)
|
s.mux.HandleFunc("GET /api/health", s.handleHealth)
|
||||||
s.mux.HandleFunc("GET /api/version", s.handleVersion)
|
s.mux.HandleFunc("GET /api/version", s.handleVersion)
|
||||||
|
// Prometheus-Scrape-Endpunkt (FDN-08, internal/api/metrics_handlers.go).
|
||||||
|
// Bewusst ohne s.auth — ein Scraper hat keine Session; der Zugriff wird
|
||||||
|
// stattdessen per Quell-IP begrenzt (loopback + api.metrics_allowed_ips).
|
||||||
|
s.mux.HandleFunc("GET /metrics", s.handleMetrics)
|
||||||
|
|
||||||
s.mux.HandleFunc("POST /api/auth/login", s.handleLogin)
|
s.mux.HandleFunc("POST /api/auth/login", s.handleLogin)
|
||||||
s.mux.HandleFunc("GET /api/auth/me", s.auth(s.handleMe))
|
s.mux.HandleFunc("GET /api/auth/me", s.auth(s.handleMe))
|
||||||
@@ -377,6 +438,12 @@ func (s *Server) routes() {
|
|||||||
s.mux.HandleFunc("GET /public/share/{token}", s.handlePublicShareMeta)
|
s.mux.HandleFunc("GET /public/share/{token}", s.handlePublicShareMeta)
|
||||||
s.mux.HandleFunc("POST /public/share/{token}/download", s.handlePublicShareDownload)
|
s.mux.HandleFunc("POST /public/share/{token}/download", s.handlePublicShareDownload)
|
||||||
|
|
||||||
|
// Signed, time-limited download URLs (internal/api/signed_url_handlers.go).
|
||||||
|
// Issuing is authenticated + tenant-scoped; redeeming runs WITHOUT s.auth
|
||||||
|
// because the HMAC signature in the query string is the credential.
|
||||||
|
s.mux.HandleFunc("POST /api/documents/{id}/signed-url", s.auth(s.handleCreateDocumentSignedURL))
|
||||||
|
s.mux.HandleFunc("GET /public/files", s.handleSignedFileDownload)
|
||||||
|
|
||||||
// Buchhaltungs-Pull-API (internal/api/accounting_handlers.go).
|
// Buchhaltungs-Pull-API (internal/api/accounting_handlers.go).
|
||||||
// Key administration runs on the normal session auth and is domain_admin-only
|
// Key administration runs on the normal session auth and is domain_admin-only
|
||||||
// (a key grants tenant-wide read access to archived documents).
|
// (a key grants tenant-wide read access to archived documents).
|
||||||
@@ -417,8 +484,23 @@ func (s *Server) routes() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ServeHTTP implements http.Handler.
|
// ServeHTTP implements http.Handler.
|
||||||
|
//
|
||||||
|
// Die Basis-Middleware-Kette (FDN-08) liegt bewusst hier und nicht an den
|
||||||
|
// einzelnen Routen, damit sie ausnahmslos für JEDE Anfrage gilt — auch für
|
||||||
|
// /public/*, /metrics und nicht gefundene Pfade:
|
||||||
|
//
|
||||||
|
// requestID -> metrics -> recover -> ServeMux
|
||||||
|
//
|
||||||
|
// Reihenfolge: requestID zuerst, damit Metrik- und Panic-Log die
|
||||||
|
// Korrelations-ID haben; recover innen, damit der 500 noch über den
|
||||||
|
// statusRecorder der Metrik-Middleware läuft und dort gezählt wird.
|
||||||
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||||
s.mux.ServeHTTP(w, r)
|
if s.baseHandler == nil {
|
||||||
|
// Server wurde nicht über New() gebaut (Tests): ohne Kette bedienen.
|
||||||
|
s.mux.ServeHTTP(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.baseHandler.ServeHTTP(w, r)
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- system handlers ---
|
// --- system handlers ---
|
||||||
|
|||||||
@@ -0,0 +1,162 @@
|
|||||||
|
// Signed, time-limited download URLs for archived documents (FDN-03,
|
||||||
|
// internal/objectstore). Same principle as the external share links
|
||||||
|
// (share_handlers.go / public_share_handlers.go) — unguessable credential in
|
||||||
|
// the link, mandatory hard expiry, IP rate limiting, audit trail — but
|
||||||
|
// stateless: the credential is an HMAC-SHA256 signature over
|
||||||
|
// tenant|document|expiry instead of a DB row.
|
||||||
|
//
|
||||||
|
// POST /api/documents/{id}/signed-url issue a link (authenticated, tenant-scoped)
|
||||||
|
// GET /public/files?t=&d=&exp=&sig= redeem it (no session, signature is the credential)
|
||||||
|
//
|
||||||
|
// Use share links when a document is handed to an external party with its own
|
||||||
|
// lifecycle (revoke, password, access cap); use signed URLs for short-lived
|
||||||
|
// machine access to the file itself (viewer, export job) without a session
|
||||||
|
// cookie. Nothing here bypasses tenant scoping: the tenant id is taken from
|
||||||
|
// the signed payload and every lookup still filters on it.
|
||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"archivdms/internal/audit"
|
||||||
|
"archivdms/internal/objectstore"
|
||||||
|
)
|
||||||
|
|
||||||
|
// createSignedURLRequest is the optional POST body. TTLMinutes overrides the
|
||||||
|
// configured default validity (storage.signed_url_ttl_minutes); it is capped
|
||||||
|
// at 24 hours so no effectively unbounded link can be minted.
|
||||||
|
type createSignedURLRequest struct {
|
||||||
|
TTLMinutes int `json:"ttl_minutes,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// createSignedURLResponse is returned to the caller.
|
||||||
|
type createSignedURLResponse struct {
|
||||||
|
URL string `json:"url"`
|
||||||
|
ExpiresAt time.Time `json:"expires_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// maxSignedURLTTL caps a caller-supplied validity.
|
||||||
|
const maxSignedURLTTL = 24 * time.Hour
|
||||||
|
|
||||||
|
// handleCreateDocumentSignedURL handles POST /api/documents/{id}/signed-url.
|
||||||
|
func (s *Server) handleCreateDocumentSignedURL(w http.ResponseWriter, r *http.Request) {
|
||||||
|
sess := sessionFromCtx(r.Context())
|
||||||
|
if sess.TenantID == nil {
|
||||||
|
writeError(w, http.StatusForbidden, "tenant context required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.objects == nil {
|
||||||
|
writeError(w, http.StatusServiceUnavailable, "Dateispeicher nicht konfiguriert")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, http.StatusBadRequest, "invalid document id")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var req createSignedURLRequest
|
||||||
|
if r.Body != nil {
|
||||||
|
_ = json.NewDecoder(r.Body).Decode(&req) // body is optional
|
||||||
|
}
|
||||||
|
ttl := time.Duration(req.TTLMinutes) * time.Minute
|
||||||
|
if ttl <= 0 {
|
||||||
|
ttl = s.storageCfg.ResolvedSignedURLTTL()
|
||||||
|
}
|
||||||
|
if ttl > maxSignedURLTTL {
|
||||||
|
ttl = maxSignedURLTTL
|
||||||
|
}
|
||||||
|
|
||||||
|
// IDOR guard: the document must belong to the caller's tenant. GetDocument
|
||||||
|
// filters WHERE id = $1 AND tenant_id = $2.
|
||||||
|
doc, err := s.store.GetDocument(r.Context(), docID, *sess.TenantID)
|
||||||
|
if err != nil {
|
||||||
|
s.logShare(r, audit.EventSignedURLCreated, sess.TenantID, sess.Username,
|
||||||
|
"signed_url doc:"+strconv.FormatInt(docID, 10)+" err:not_found", false)
|
||||||
|
writeError(w, http.StatusNotFound, "document not found")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
link, err := s.objects.SignedURL(*sess.TenantID, doc.ID, ttl)
|
||||||
|
if err != nil {
|
||||||
|
s.logShare(r, audit.EventSignedURLCreated, sess.TenantID, sess.Username,
|
||||||
|
"signed_url doc:"+strconv.FormatInt(docID, 10)+" err:"+err.Error(), false)
|
||||||
|
writeError(w, http.StatusInternalServerError, "Link konnte nicht erstellt werden")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
expiresAt := time.Now().Add(ttl)
|
||||||
|
s.logShare(r, audit.EventSignedURLCreated, sess.TenantID, sess.Username,
|
||||||
|
"signed_url doc:"+strconv.FormatInt(docID, 10)+" ttl:"+ttl.String(), true)
|
||||||
|
writeJSON(w, http.StatusOK, createSignedURLResponse{URL: link, ExpiresAt: expiresAt})
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleSignedFileDownload handles GET /public/files?t=&d=&exp=&sig=. Served
|
||||||
|
// WITHOUT the s.auth wrapper by design: the signature is the credential.
|
||||||
|
// Order: rate-limit -> verify signature -> verify expiry -> tenant-scoped
|
||||||
|
// document lookup -> stream from the WORM store. Every outcome is audit-logged
|
||||||
|
// (EventSignedURLAccessed), successes and failures alike.
|
||||||
|
func (s *Server) handleSignedFileDownload(w http.ResponseWriter, r *http.Request) {
|
||||||
|
ip := s.remoteIP(r)
|
||||||
|
if !s.shareLimiter.allow(ip) {
|
||||||
|
writeError(w, http.StatusTooManyRequests, "too many requests")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.objects == nil {
|
||||||
|
writeError(w, http.StatusServiceUnavailable, "Dateispeicher nicht konfiguriert")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ref, err := s.objects.VerifySignedURL(r.URL.Query(), time.Now())
|
||||||
|
if err != nil {
|
||||||
|
s.logSignedAccess(r, nil, "signed_url_access err:"+err.Error(), false)
|
||||||
|
if errors.Is(err, objectstore.ErrSignatureExpired) {
|
||||||
|
writeError(w, http.StatusGone, "Der Link ist abgelaufen.")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeError(w, http.StatusForbidden, "Der Link ist ungültig.")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
doc, err := s.store.GetDocument(r.Context(), ref.DocumentID, ref.TenantID)
|
||||||
|
if err != nil {
|
||||||
|
s.logSignedAccess(r, &ref.TenantID,
|
||||||
|
"signed_url_access doc:"+strconv.FormatInt(ref.DocumentID, 10)+" err:not_found", false)
|
||||||
|
writeError(w, http.StatusNotFound, "Dokument nicht gefunden.")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f, err := s.objects.Open(r.Context(), ref.TenantID, doc.StoragePath)
|
||||||
|
if err != nil {
|
||||||
|
s.reqLog(r.Context()).Error("signed url file open failed", "document_id", doc.ID, "tenant_id", ref.TenantID, "err", err)
|
||||||
|
s.logSignedAccess(r, &ref.TenantID,
|
||||||
|
"signed_url_access doc:"+strconv.FormatInt(doc.ID, 10)+" err:"+err.Error(), false)
|
||||||
|
if errors.Is(err, objectstore.ErrObjectNotFound) {
|
||||||
|
writeError(w, http.StatusNotFound, "Datei nicht im Archiv vorhanden.")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeError(w, http.StatusInternalServerError, "Download fehlgeschlagen.")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
|
||||||
|
s.logSignedAccess(r, &ref.TenantID, "signed_url_access doc:"+strconv.FormatInt(doc.ID, 10), true)
|
||||||
|
|
||||||
|
ext := filepath.Ext(doc.StoragePath)
|
||||||
|
w.Header().Set("Content-Type", detectMimeType("", ext, doc.StoragePath))
|
||||||
|
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(doc.Title, ext)+"\"")
|
||||||
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||||
|
if _, err := io.Copy(w, f); err != nil {
|
||||||
|
s.reqLog(r.Context()).Warn("signed url stream interrupted", "document_id", doc.ID, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// logSignedAccess records a redemption attempt in the audit log. There is no
|
||||||
|
// session here, so the username column carries the anonymous marker.
|
||||||
|
func (s *Server) logSignedAccess(r *http.Request, tenantID *int64, detail string, ok bool) {
|
||||||
|
s.audlog.Log(audit.Entry{
|
||||||
|
EventType: audit.EventSignedURLAccessed, Username: "anonymous", TenantID: tenantID,
|
||||||
|
IPAddress: s.remoteIP(r), Success: ok, Detail: detail,
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -87,6 +87,12 @@ const (
|
|||||||
EventShareRevoked = "share_revoked"
|
EventShareRevoked = "share_revoked"
|
||||||
EventShareAccessed = "share_accessed"
|
EventShareAccessed = "share_accessed"
|
||||||
|
|
||||||
|
// Signed, time-limited storage download URLs (internal/objectstore,
|
||||||
|
// internal/api/signed_url_handlers.go). Created is logged when a link is
|
||||||
|
// issued, Accessed on every attempt to redeem one (success and failure).
|
||||||
|
EventSignedURLCreated = "signed_url_created"
|
||||||
|
EventSignedURLAccessed = "signed_url_accessed"
|
||||||
|
|
||||||
// LDAP directory integration (internal/ldapstore, internal/ldapauth,
|
// LDAP directory integration (internal/ldapstore, internal/ldapauth,
|
||||||
// internal/api/ldap_handlers.go). ConfigChanged covers create/update/delete
|
// internal/api/ldap_handlers.go). ConfigChanged covers create/update/delete
|
||||||
// and the test action; LoginSuccess/Failed are logged on every LDAP bind
|
// and the test action; LoginSuccess/Failed are logged on every LDAP bind
|
||||||
|
|||||||
@@ -0,0 +1,247 @@
|
|||||||
|
package objectstore
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/base64"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/crypto/hkdf"
|
||||||
|
|
||||||
|
"archivdms/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// signKeyInfo domain-separates the download-URL signing key from every other
|
||||||
|
// key derived from the same master secret (JWT uses "archivdms-jwt-v1", the
|
||||||
|
// LDAP secretbox uses "archivdms-ldap-secretbox-v1").
|
||||||
|
const signKeyInfo = "archivdms-storage-url-v1"
|
||||||
|
|
||||||
|
// SignedPath is the route a signed download URL points at. It is served
|
||||||
|
// WITHOUT session auth — the signature is the credential.
|
||||||
|
const SignedPath = "/public/files"
|
||||||
|
|
||||||
|
// LocalStore is the local-filesystem WORM driver and the only implementation
|
||||||
|
// of Store. It owns no state beyond the storage configuration, the URL signing
|
||||||
|
// key and the public base URL used for link generation.
|
||||||
|
type LocalStore struct {
|
||||||
|
cfg config.StorageConfig
|
||||||
|
signKey []byte
|
||||||
|
baseURL string
|
||||||
|
}
|
||||||
|
|
||||||
|
// compile-time interface check.
|
||||||
|
var _ Store = (*LocalStore)(nil)
|
||||||
|
|
||||||
|
// NewLocalStore builds the local driver. secret is the application master
|
||||||
|
// secret (config.APIConfig.Secret) from which the URL signing key is derived
|
||||||
|
// via HKDF-SHA256; baseURL is the public origin used for generated links
|
||||||
|
// (empty = emit site-relative URLs).
|
||||||
|
func NewLocalStore(cfg config.StorageConfig, secret, baseURL string) (*LocalStore, error) {
|
||||||
|
if strings.TrimSpace(secret) == "" {
|
||||||
|
return nil, fmt.Errorf("objectstore: empty signing secret")
|
||||||
|
}
|
||||||
|
key := make([]byte, 32)
|
||||||
|
if _, err := io.ReadFull(hkdf.New(sha256.New, []byte(secret), nil, []byte(signKeyInfo)), key); err != nil {
|
||||||
|
return nil, fmt.Errorf("objectstore: derive signing key: %w", err)
|
||||||
|
}
|
||||||
|
return &LocalStore{
|
||||||
|
cfg: cfg,
|
||||||
|
signKey: key,
|
||||||
|
baseURL: strings.TrimRight(strings.TrimSpace(baseURL), "/"),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// tenantRoot is the tenant's WORM subtree: <BasePath>/store/<tenant_id>.
|
||||||
|
func (l *LocalStore) tenantRoot(tenantID int64) string {
|
||||||
|
return filepath.Join(l.cfg.StorePath(), strconv.FormatInt(tenantID, 10))
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveTenantPath cleans storagePath and verifies it lies inside the
|
||||||
|
// tenant's own store subtree. This is the filesystem-level counterpart of the
|
||||||
|
// "WHERE tenant_id = $N" rule: even a manipulated storage_path from the DB
|
||||||
|
// cannot be used to read another tenant's archive.
|
||||||
|
func (l *LocalStore) resolveTenantPath(tenantID int64, storagePath string) (string, error) {
|
||||||
|
if strings.TrimSpace(storagePath) == "" {
|
||||||
|
return "", ErrObjectNotFound
|
||||||
|
}
|
||||||
|
clean := filepath.Clean(storagePath)
|
||||||
|
root := filepath.Clean(l.tenantRoot(tenantID))
|
||||||
|
if clean != root && !strings.HasPrefix(clean, root+string(os.PathSeparator)) {
|
||||||
|
return "", ErrOutsideTenant
|
||||||
|
}
|
||||||
|
return clean, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Archive implements Store. It reproduces, unchanged, the archival steps the
|
||||||
|
// upload pipeline has always performed: build store/<tenant>/<yyyy>/<mm>,
|
||||||
|
// reject an existing target as duplicate, move (rename, copy+remove fallback
|
||||||
|
// across devices) and finally chmod 0440 — the one and only chmod, applied
|
||||||
|
// once the file sits at its final path.
|
||||||
|
func (l *LocalStore) Archive(ctx context.Context, tenantID int64, srcPath, ext, contentHash string, at time.Time) (string, error) {
|
||||||
|
if at.IsZero() {
|
||||||
|
at = time.Now()
|
||||||
|
}
|
||||||
|
storeDir := filepath.Join(l.tenantRoot(tenantID),
|
||||||
|
fmt.Sprintf("%04d", at.Year()), fmt.Sprintf("%02d", at.Month()))
|
||||||
|
if err := os.MkdirAll(storeDir, 0o750); err != nil {
|
||||||
|
os.Remove(srcPath)
|
||||||
|
return "", fmt.Errorf("objectstore: create store dir: %w", err)
|
||||||
|
}
|
||||||
|
dst := filepath.Join(storeDir, contentHash+ext)
|
||||||
|
|
||||||
|
// Collision check: identical hash already stored -> duplicate.
|
||||||
|
if _, err := os.Stat(dst); err == nil {
|
||||||
|
os.Remove(srcPath)
|
||||||
|
return "", ErrObjectExists
|
||||||
|
} else if !os.IsNotExist(err) {
|
||||||
|
os.Remove(srcPath)
|
||||||
|
return "", fmt.Errorf("objectstore: stat store path: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := os.Rename(srcPath, dst); err != nil {
|
||||||
|
if copyErr := copyFile(srcPath, dst); copyErr != nil {
|
||||||
|
os.Remove(srcPath)
|
||||||
|
return "", fmt.Errorf("objectstore: move file to store: rename failed (%v), copy fallback failed: %w", err, copyErr)
|
||||||
|
}
|
||||||
|
os.Remove(srcPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// WORM lock: read-only for everyone from now on.
|
||||||
|
if err := os.Chmod(dst, 0o440); err != nil {
|
||||||
|
return "", fmt.Errorf("objectstore: chmod store file: %w", err)
|
||||||
|
}
|
||||||
|
return dst, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Open implements Store.
|
||||||
|
func (l *LocalStore) Open(ctx context.Context, tenantID int64, storagePath string) (io.ReadSeekCloser, error) {
|
||||||
|
p, err := l.resolveTenantPath(tenantID, storagePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
f, err := os.Open(p)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, fmt.Errorf("objectstore: open %q: %w", p, ErrObjectNotFound)
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("objectstore: open object: %w", err)
|
||||||
|
}
|
||||||
|
return f, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stat implements Store.
|
||||||
|
func (l *LocalStore) Stat(ctx context.Context, tenantID int64, storagePath string) (os.FileInfo, error) {
|
||||||
|
p, err := l.resolveTenantPath(tenantID, storagePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
fi, err := os.Stat(p)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, fmt.Errorf("objectstore: stat %q: %w", p, ErrObjectNotFound)
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("objectstore: stat object: %w", err)
|
||||||
|
}
|
||||||
|
return fi, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete implements Store. Only legitimate after a confirmed deletion request
|
||||||
|
// whose retention period has expired — this layer performs no retention check
|
||||||
|
// of its own; that stays in storage.ConfirmDeleteRequest, which still unlinks
|
||||||
|
// inside its own transaction and is deliberately left untouched by FDN-03
|
||||||
|
// (moving it here would mean handing the DB layer a filesystem driver).
|
||||||
|
func (l *LocalStore) Delete(ctx context.Context, tenantID int64, storagePath string) error {
|
||||||
|
p, err := l.resolveTenantPath(tenantID, storagePath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.Remove(p); err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return fmt.Errorf("objectstore: remove %q: %w", p, ErrObjectNotFound)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("objectstore: remove object: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SignedURL implements Store. The URL carries tenant id, document id and an
|
||||||
|
// absolute expiry, authenticated by an HMAC-SHA256 over exactly those three
|
||||||
|
// values — the same "unguessable token, hard expiry, server-side check"
|
||||||
|
// principle as the external share links, only stateless (no DB row).
|
||||||
|
func (l *LocalStore) SignedURL(tenantID, documentID int64, ttl time.Duration) (string, error) {
|
||||||
|
if tenantID <= 0 || documentID <= 0 {
|
||||||
|
return "", fmt.Errorf("objectstore: invalid signed url reference")
|
||||||
|
}
|
||||||
|
if ttl <= 0 {
|
||||||
|
ttl = l.cfg.ResolvedSignedURLTTL()
|
||||||
|
}
|
||||||
|
exp := time.Now().Add(ttl).Unix()
|
||||||
|
q := url.Values{}
|
||||||
|
q.Set("t", strconv.FormatInt(tenantID, 10))
|
||||||
|
q.Set("d", strconv.FormatInt(documentID, 10))
|
||||||
|
q.Set("exp", strconv.FormatInt(exp, 10))
|
||||||
|
q.Set("sig", l.sign(tenantID, documentID, exp))
|
||||||
|
return l.baseURL + SignedPath + "?" + q.Encode(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// VerifySignedURL implements Store. Signature first, expiry second, so a
|
||||||
|
// forged link never learns anything from the expiry branch.
|
||||||
|
func (l *LocalStore) VerifySignedURL(q url.Values, now time.Time) (SignedRef, error) {
|
||||||
|
tenantID, err1 := strconv.ParseInt(q.Get("t"), 10, 64)
|
||||||
|
documentID, err2 := strconv.ParseInt(q.Get("d"), 10, 64)
|
||||||
|
exp, err3 := strconv.ParseInt(q.Get("exp"), 10, 64)
|
||||||
|
sig := q.Get("sig")
|
||||||
|
if err1 != nil || err2 != nil || err3 != nil || sig == "" || tenantID <= 0 || documentID <= 0 {
|
||||||
|
return SignedRef{}, ErrSignatureInvalid
|
||||||
|
}
|
||||||
|
want := l.sign(tenantID, documentID, exp)
|
||||||
|
if !hmac.Equal([]byte(want), []byte(sig)) {
|
||||||
|
return SignedRef{}, ErrSignatureInvalid
|
||||||
|
}
|
||||||
|
expiresAt := time.Unix(exp, 0)
|
||||||
|
if !now.Before(expiresAt) {
|
||||||
|
return SignedRef{}, ErrSignatureExpired
|
||||||
|
}
|
||||||
|
return SignedRef{TenantID: tenantID, DocumentID: documentID, ExpiresAt: expiresAt}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// sign returns the base64url HMAC-SHA256 over the canonical payload.
|
||||||
|
func (l *LocalStore) sign(tenantID, documentID, exp int64) string {
|
||||||
|
mac := hmac.New(sha256.New, l.signKey)
|
||||||
|
fmt.Fprintf(mac, "v1|%d|%d|%d", tenantID, documentID, exp)
|
||||||
|
return base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
|
||||||
|
}
|
||||||
|
|
||||||
|
// copyFile is the cross-device fallback for os.Rename (EXDEV): copy + fsync.
|
||||||
|
// The source is removed by the caller.
|
||||||
|
func copyFile(src, dst string) error {
|
||||||
|
in, err := os.Open(src)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer in.Close()
|
||||||
|
|
||||||
|
out, err := os.OpenFile(dst, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o640)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, err := io.Copy(out, in); err != nil {
|
||||||
|
out.Close()
|
||||||
|
os.Remove(dst)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := out.Sync(); err != nil {
|
||||||
|
out.Close()
|
||||||
|
os.Remove(dst)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return out.Close()
|
||||||
|
}
|
||||||
@@ -0,0 +1,204 @@
|
|||||||
|
package objectstore
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"archivdms/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newTestStore(t *testing.T) *LocalStore {
|
||||||
|
t.Helper()
|
||||||
|
l, err := NewLocalStore(config.StorageConfig{BasePath: t.TempDir()}, "test-master-secret", "https://dms.example.test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewLocalStore: %v", err)
|
||||||
|
}
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
// stage writes a scratch file and returns its path plus content hash.
|
||||||
|
func stage(t *testing.T, content string) (string, string) {
|
||||||
|
t.Helper()
|
||||||
|
p := filepath.Join(t.TempDir(), "scratch.pdf")
|
||||||
|
if err := os.WriteFile(p, []byte(content), 0o640); err != nil {
|
||||||
|
t.Fatalf("write scratch: %v", err)
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256([]byte(content))
|
||||||
|
return p, hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 1 + Prüfung 1: Round-Trip Archive -> Open, WORM path scheme and 0440.
|
||||||
|
func TestArchiveOpenRoundTrip(t *testing.T) {
|
||||||
|
l := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
src, hash := stage(t, "hello worm")
|
||||||
|
at := time.Date(2026, 3, 7, 10, 0, 0, 0, time.UTC)
|
||||||
|
|
||||||
|
dst, err := l.Archive(ctx, 42, src, ".pdf", hash, at)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Archive: %v", err)
|
||||||
|
}
|
||||||
|
want := filepath.Join(l.cfg.StorePath(), "42", "2026", "03", hash+".pdf")
|
||||||
|
if dst != want {
|
||||||
|
t.Fatalf("path scheme changed: got %q want %q", dst, want)
|
||||||
|
}
|
||||||
|
fi, err := os.Stat(dst)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stat archived: %v", err)
|
||||||
|
}
|
||||||
|
if fi.Mode().Perm() != 0o440 {
|
||||||
|
t.Fatalf("WORM permissions: got %o want 0440", fi.Mode().Perm())
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(src); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("scratch file not consumed")
|
||||||
|
}
|
||||||
|
|
||||||
|
f, err := l.Open(ctx, 42, dst)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Open: %v", err)
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
got, _ := io.ReadAll(f)
|
||||||
|
if string(got) != "hello worm" {
|
||||||
|
t.Fatalf("round-trip content mismatch: %q", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Duplicate archival of the same content is rejected.
|
||||||
|
src2, _ := stage(t, "hello worm")
|
||||||
|
if _, err := l.Archive(ctx, 42, src2, ".pdf", hash, at); !errors.Is(err, ErrObjectExists) {
|
||||||
|
t.Fatalf("duplicate: got %v want ErrObjectExists", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 1: cross-tenant access is refused even with a valid path.
|
||||||
|
func TestOpenForeignTenantRejected(t *testing.T) {
|
||||||
|
l := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
src, hash := stage(t, "tenant one")
|
||||||
|
dst, err := l.Archive(ctx, 1, src, ".pdf", hash, time.Now())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Archive: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := l.Open(ctx, 2, dst); !errors.Is(err, ErrOutsideTenant) {
|
||||||
|
t.Fatalf("foreign tenant: got %v want ErrOutsideTenant", err)
|
||||||
|
}
|
||||||
|
if _, err := l.Open(ctx, 1, filepath.Join(filepath.Dir(dst), "..", "..", "..", "2", "x.pdf")); !errors.Is(err, ErrOutsideTenant) {
|
||||||
|
t.Fatalf("traversal: got %v want ErrOutsideTenant", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prüfung 3: missing object yields a clear, typed error.
|
||||||
|
func TestMissingObjectErrors(t *testing.T) {
|
||||||
|
l := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
missing := filepath.Join(l.cfg.StorePath(), "7", "2026", "01", "deadbeef.pdf")
|
||||||
|
|
||||||
|
if _, err := l.Open(ctx, 7, missing); !errors.Is(err, ErrObjectNotFound) {
|
||||||
|
t.Fatalf("Open: got %v want ErrObjectNotFound", err)
|
||||||
|
}
|
||||||
|
if _, err := l.Stat(ctx, 7, missing); !errors.Is(err, ErrObjectNotFound) {
|
||||||
|
t.Fatalf("Stat: got %v want ErrObjectNotFound", err)
|
||||||
|
}
|
||||||
|
if err := l.Delete(ctx, 7, missing); !errors.Is(err, ErrObjectNotFound) {
|
||||||
|
t.Fatalf("Delete: got %v want ErrObjectNotFound", err)
|
||||||
|
}
|
||||||
|
if _, err := l.Open(ctx, 7, ""); !errors.Is(err, ErrObjectNotFound) {
|
||||||
|
t.Fatalf("empty path: got %v want ErrObjectNotFound", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDeleteRemovesObject(t *testing.T) {
|
||||||
|
l := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
src, hash := stage(t, "to be deleted")
|
||||||
|
dst, err := l.Archive(ctx, 5, src, ".pdf", hash, time.Now())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Archive: %v", err)
|
||||||
|
}
|
||||||
|
if err := l.Delete(ctx, 5, dst); err != nil {
|
||||||
|
t.Fatalf("Delete: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(dst); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("object still present after delete")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 2 + Prüfung 2: signed URLs verify while valid and are refused afterwards.
|
||||||
|
func TestSignedURLLifecycle(t *testing.T) {
|
||||||
|
l := newTestStore(t)
|
||||||
|
link, err := l.SignedURL(3, 99, time.Minute)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SignedURL: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(link, "https://dms.example.test"+SignedPath+"?") {
|
||||||
|
t.Fatalf("unexpected link: %s", link)
|
||||||
|
}
|
||||||
|
u, err := url.Parse(link)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse link: %v", err)
|
||||||
|
}
|
||||||
|
ref, err := l.VerifySignedURL(u.Query(), time.Now())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("verify: %v", err)
|
||||||
|
}
|
||||||
|
if ref.TenantID != 3 || ref.DocumentID != 99 {
|
||||||
|
t.Fatalf("payload mismatch: %+v", ref)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Expired.
|
||||||
|
if _, err := l.VerifySignedURL(u.Query(), time.Now().Add(2*time.Minute)); !errors.Is(err, ErrSignatureExpired) {
|
||||||
|
t.Fatalf("expired: got %v want ErrSignatureExpired", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tampered document id.
|
||||||
|
q := u.Query()
|
||||||
|
q.Set("d", "100")
|
||||||
|
if _, err := l.VerifySignedURL(q, time.Now()); !errors.Is(err, ErrSignatureInvalid) {
|
||||||
|
t.Fatalf("tampered: got %v want ErrSignatureInvalid", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Foreign key material.
|
||||||
|
other, err := NewLocalStore(config.StorageConfig{BasePath: t.TempDir()}, "different-secret", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewLocalStore: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := other.VerifySignedURL(u.Query(), time.Now()); !errors.Is(err, ErrSignatureInvalid) {
|
||||||
|
t.Fatalf("foreign key: got %v want ErrSignatureInvalid", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Missing parameters.
|
||||||
|
if _, err := l.VerifySignedURL(url.Values{}, time.Now()); !errors.Is(err, ErrSignatureInvalid) {
|
||||||
|
t.Fatalf("empty query: got %v want ErrSignatureInvalid", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AK 2: ttl <= 0 falls back to the configured default validity.
|
||||||
|
func TestSignedURLDefaultTTL(t *testing.T) {
|
||||||
|
l, err := NewLocalStore(config.StorageConfig{BasePath: t.TempDir(), SignedURLTTLMinutes: 5}, "s", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewLocalStore: %v", err)
|
||||||
|
}
|
||||||
|
link, err := l.SignedURL(1, 1, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SignedURL: %v", err)
|
||||||
|
}
|
||||||
|
u, _ := url.Parse(link)
|
||||||
|
if strings.HasPrefix(link, "http") {
|
||||||
|
t.Fatalf("empty baseURL must yield a relative link: %s", link)
|
||||||
|
}
|
||||||
|
if _, err := l.VerifySignedURL(u.Query(), time.Now().Add(4*time.Minute)); err != nil {
|
||||||
|
t.Fatalf("within default ttl: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := l.VerifySignedURL(u.Query(), time.Now().Add(6*time.Minute)); !errors.Is(err, ErrSignatureExpired) {
|
||||||
|
t.Fatalf("past default ttl: got %v want ErrSignatureExpired", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
// Package objectstore puts the existing local WORM document storage behind a
|
||||||
|
// small Go interface (FDN-03). It is a pure abstraction layer: the on-disk
|
||||||
|
// layout, the chmod 0440 WORM lock and the SHA-256 content addressing are
|
||||||
|
// exactly the ones the upload pipeline has always used — nothing about the
|
||||||
|
// path scheme or the archival semantics changes here.
|
||||||
|
//
|
||||||
|
// # Pfadschema (bestehend, NICHT verändert)
|
||||||
|
//
|
||||||
|
// All paths are rooted at config.Storage.BasePath (default /var/lib/archivdms):
|
||||||
|
//
|
||||||
|
// <BasePath>/inbox/<tenant_id>/<random>.<ext> raw upload, scratch, writable (0640)
|
||||||
|
// <BasePath>/store/<tenant_id>/<yyyy>/<mm>/<sha256>.<ext> finished archive, WORM (0440)
|
||||||
|
// <BasePath>/ocr-tmp/<random>/ OCR scratch, removed after use
|
||||||
|
// <BasePath>/thumbnails/<tenant_id>/<sha256>.png regenerable preview, not WORM
|
||||||
|
//
|
||||||
|
// Properties of the store/ layer that callers may rely on:
|
||||||
|
//
|
||||||
|
// - Tenant separation is the FIRST path segment: every object of a tenant
|
||||||
|
// lives below store/<tenant_id>/ and nowhere else. Open/Stat/Delete
|
||||||
|
// therefore verify that the given path really is inside that tenant's
|
||||||
|
// subtree (containment check) — a stored path from a foreign tenant is
|
||||||
|
// rejected with ErrOutsideTenant instead of being read.
|
||||||
|
// - <yyyy>/<mm> is derived from the archival (upload) time, not from the
|
||||||
|
// recognised Belegdatum: after the WORM move a file is never moved again.
|
||||||
|
// - The file name is the lowercase hex SHA-256 of the file content plus the
|
||||||
|
// original extension. Content addressing gives byte-identical re-uploads
|
||||||
|
// the same path, which is the filesystem half of the duplicate protection
|
||||||
|
// (the DB unique index on (tenant_id, content_hash) is the other half).
|
||||||
|
// - Archived files are chmod 0440. The directory stays writable for the
|
||||||
|
// service user, so a legally confirmed deletion (after retain_until) can
|
||||||
|
// still unlink the file — no code path ever overwrites an archived file.
|
||||||
|
// - Nothing is encrypted or container-wrapped: every object is readable with
|
||||||
|
// plain OS tools, deliberately unlike a closed vendor archive.
|
||||||
|
//
|
||||||
|
// Deliberately NO S3/object-storage driver: the WORM/GoBD guarantee rests on
|
||||||
|
// POSIX file permissions (0440) which an object store cannot provide in the
|
||||||
|
// same way. The local driver is and stays the only implementation.
|
||||||
|
package objectstore
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/url"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Storage-level errors. Callers map these onto HTTP status codes / domain
|
||||||
|
// errors (e.g. ErrObjectExists -> storage.ErrDuplicateContentHash).
|
||||||
|
var (
|
||||||
|
// ErrObjectExists is returned by Archive when the target WORM path is
|
||||||
|
// already taken, i.e. the identical content is already archived.
|
||||||
|
ErrObjectExists = errors.New("objectstore: object already exists")
|
||||||
|
// ErrObjectNotFound is returned by Open/Stat/Delete when the object does
|
||||||
|
// not exist on disk.
|
||||||
|
ErrObjectNotFound = errors.New("objectstore: object not found")
|
||||||
|
// ErrOutsideTenant is returned when a storage path does not resolve into
|
||||||
|
// the requesting tenant's store subtree (IDOR / path-traversal guard).
|
||||||
|
ErrOutsideTenant = errors.New("objectstore: path outside tenant store")
|
||||||
|
// ErrSignatureInvalid is returned when a signed URL is malformed or its
|
||||||
|
// HMAC does not verify.
|
||||||
|
ErrSignatureInvalid = errors.New("objectstore: signature invalid")
|
||||||
|
// ErrSignatureExpired is returned when a signed URL's expiry has passed.
|
||||||
|
ErrSignatureExpired = errors.New("objectstore: signature expired")
|
||||||
|
)
|
||||||
|
|
||||||
|
// SignedRef is the payload carried by a signed download URL: which document of
|
||||||
|
// which tenant may be downloaded, and until when.
|
||||||
|
type SignedRef struct {
|
||||||
|
TenantID int64
|
||||||
|
DocumentID int64
|
||||||
|
ExpiresAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// Store is the document blob storage abstraction. Every method is
|
||||||
|
// tenant-scoped; there is intentionally no "list everything" call.
|
||||||
|
type Store interface {
|
||||||
|
// Archive moves an already-hashed scratch file (inbox or split part) into
|
||||||
|
// the tenant's WORM store and locks it with chmod 0440. It returns the
|
||||||
|
// final storage path. On success the caller no longer owns srcPath; on
|
||||||
|
// failure srcPath is removed. Returns ErrObjectExists when the content is
|
||||||
|
// already archived (duplicate).
|
||||||
|
Archive(ctx context.Context, tenantID int64, srcPath, ext, contentHash string, at time.Time) (string, error)
|
||||||
|
|
||||||
|
// Open opens an archived object read-only after verifying that
|
||||||
|
// storagePath belongs to tenantID.
|
||||||
|
Open(ctx context.Context, tenantID int64, storagePath string) (io.ReadSeekCloser, error)
|
||||||
|
|
||||||
|
// Stat reports metadata of an archived object (tenant-checked).
|
||||||
|
Stat(ctx context.Context, tenantID int64, storagePath string) (os.FileInfo, error)
|
||||||
|
|
||||||
|
// Delete unlinks an archived object (tenant-checked). Only ever called
|
||||||
|
// after a confirmed, retention-cleared deletion request; a missing file is
|
||||||
|
// reported as ErrObjectNotFound.
|
||||||
|
Delete(ctx context.Context, tenantID int64, storagePath string) error
|
||||||
|
|
||||||
|
// SignedURL builds a time-limited, HMAC-signed download URL for a
|
||||||
|
// document. ttl <= 0 uses the configured default validity.
|
||||||
|
SignedURL(tenantID, documentID int64, ttl time.Duration) (string, error)
|
||||||
|
|
||||||
|
// VerifySignedURL validates the query parameters of a signed URL against
|
||||||
|
// the signing key and the current time.
|
||||||
|
VerifySignedURL(q url.Values, now time.Time) (SignedRef, error)
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
-- 024_ocr_words.down.sql
|
||||||
|
-- Rollback documentation for 024_ocr_words.sql.
|
||||||
|
-- Documentation only — archivdms has no migration runner; run manually via
|
||||||
|
-- psql after deploying a binary whose initSchema no longer creates the table.
|
||||||
|
--
|
||||||
|
-- PRECONDITION: Store.initOCRWordsSchema must be removed from
|
||||||
|
-- Store.initSchema (internal/storage/documents.go) and every caller of
|
||||||
|
-- ReplaceOCRWords / the word-box read path must be gone first — otherwise
|
||||||
|
-- the next process start recreates the table.
|
||||||
|
--
|
||||||
|
-- DATA LOSS: ocr_words is a DERIVED index over each document's OCR run, not
|
||||||
|
-- an original record. Dropping it loses no GoBD-relevant data; the word boxes
|
||||||
|
-- are fully regenerable by re-running OCR
|
||||||
|
-- (`archivdms documents reprocess-all`). Only the highlight/overlay feature
|
||||||
|
-- degrades until then.
|
||||||
|
--
|
||||||
|
-- WORM: no archived file in store/ is touched; the FK to documents is only
|
||||||
|
-- consumed here (ON DELETE CASCADE), never the other way round, so dropping
|
||||||
|
-- this table cannot cascade into documents.
|
||||||
|
|
||||||
|
BEGIN;
|
||||||
|
|
||||||
|
DROP INDEX IF EXISTS idx_ocr_words_word_text;
|
||||||
|
DROP INDEX IF EXISTS idx_ocr_words_document;
|
||||||
|
DROP TABLE IF EXISTS ocr_words;
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
-- 025_document_date_score.down.sql
|
||||||
|
-- Rollback documentation for 025_document_date_score.sql.
|
||||||
|
-- Documentation only — archivdms has no migration runner; run manually via
|
||||||
|
-- psql after deploying a binary whose initSchema no longer adds the column.
|
||||||
|
--
|
||||||
|
-- PRECONDITION: the ALTER TABLE ... ADD COLUMN IF NOT EXISTS
|
||||||
|
-- document_date_score must be removed from Store.initSchema
|
||||||
|
-- (internal/storage/documents.go) first, and every SELECT/UPDATE naming
|
||||||
|
-- document_date_score (accounting_pull.go, document date endpoint) must be
|
||||||
|
-- gone — otherwise the next start recreates the column and running queries
|
||||||
|
-- fail in between.
|
||||||
|
--
|
||||||
|
-- DATA LOSS: drops the per-document confidence values. documents.document_date
|
||||||
|
-- itself is NOT touched, so no belegdatum is lost — only the quality signal
|
||||||
|
-- the Buchhaltungs-Pull-Filter (score >= 0.75) uses. Recomputable only by
|
||||||
|
-- re-scoring the stored ocr_text.
|
||||||
|
--
|
||||||
|
-- WORM: this is a metadata column on documents; dropping it does not touch
|
||||||
|
-- any archived file in store/ and does not affect retain_until.
|
||||||
|
|
||||||
|
BEGIN;
|
||||||
|
|
||||||
|
ALTER TABLE documents DROP COLUMN IF EXISTS document_date_score;
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
-- 026_accounting_api_keys.down.sql
|
||||||
|
-- Rollback documentation for 026_accounting_api_keys.sql.
|
||||||
|
-- Documentation only — archivdms has no migration runner; this file is the
|
||||||
|
-- reviewed, copy-pasteable SQL an operator runs manually (psql) AFTER
|
||||||
|
-- deploying a binary whose initSchema no longer creates the object, and
|
||||||
|
-- inside an explicit transaction.
|
||||||
|
--
|
||||||
|
-- PRECONDITION: internal/storage/accounting_api_keys.go
|
||||||
|
-- (Store.initAccountingAPIKeysSchema) must be removed from storage.New()
|
||||||
|
-- first — otherwise the next process start recreates the table.
|
||||||
|
--
|
||||||
|
-- DATA LOSS: destroys all issued accounting API keys (hashes only, raw keys
|
||||||
|
-- were never stored). Every buchhaltung integration using the Pull-API loses
|
||||||
|
-- access and must be re-issued a new key. No GoBD document data is touched:
|
||||||
|
-- accounting_api_keys holds credentials, not archived records. The audit
|
||||||
|
-- entries referencing "key:<id>" survive in audit_log but their id becomes
|
||||||
|
-- unresolvable — accept this only when the whole feature is being withdrawn.
|
||||||
|
|
||||||
|
BEGIN;
|
||||||
|
|
||||||
|
DROP INDEX IF EXISTS idx_accounting_api_keys_tenant;
|
||||||
|
DROP TABLE IF EXISTS accounting_api_keys;
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
@@ -23,6 +23,45 @@ Migrationstool. Sie dient als:
|
|||||||
3. Der SQL-Inhalt muss exakt dem entsprechen, was `initSchema()` (oder das
|
3. Der SQL-Inhalt muss exakt dem entsprechen, was `initSchema()` (oder das
|
||||||
jeweilige Store-Paket) zur Laufzeit ausführt.
|
jeweilige Store-Paket) zur Laufzeit ausführt.
|
||||||
4. Migrationen werden nie verändert oder gelöscht, nur ergänzt.
|
4. Migrationen werden nie verändert oder gelöscht, nur ergänzt.
|
||||||
|
5. **Zu jeder neuen Migration gehört eine Down-Datei** `NNN_name.down.sql`
|
||||||
|
(siehe Abschnitt „Rollback-Pfad").
|
||||||
|
|
||||||
|
## Rollback-Pfad (`NNN_name.down.sql`)
|
||||||
|
|
||||||
|
`initSchema()` ist ausschließlich vorwärtsgerichtet — es gibt bewusst keinen
|
||||||
|
automatischen Rollback-Runner (kein Migrationstool, kein Zustandstabellen-
|
||||||
|
Tracking). Für den Ernstfall (fehlerhaftes Release, Rückbau eines Features)
|
||||||
|
braucht es trotzdem ein *reviewtes* Rückbau-SQL. Deshalb gilt ab FDN-02:
|
||||||
|
|
||||||
|
**Jede neue `NNN_name.sql` bekommt eine gleichnamige `NNN_name.down.sql`**
|
||||||
|
mit dem exakten Rückbau der Vorwärts-Migration. Separate Datei statt
|
||||||
|
`-- DOWN`-Abschnitt in derselben Datei, weil Regel 4 („Migrationen werden nie
|
||||||
|
verändert") sonst verletzt würde und weil sich eine Down-Datei fehlerfrei
|
||||||
|
per `psql -f` einspielen lässt, ohne vorher Abschnitte herauszuschneiden.
|
||||||
|
|
||||||
|
Anforderungen an eine Down-Datei:
|
||||||
|
|
||||||
|
1. Header-Kommentar mit Bezug auf die Vorwärts-Migration und der zugehörigen
|
||||||
|
PROJ-/Ticket-Nummer.
|
||||||
|
2. **Precondition** benennen: welche Go-Stelle (`initSchema`, Store-Datei,
|
||||||
|
aufrufende Queries) vorher entfernt bzw. deployt sein muss. Sonst legt der
|
||||||
|
nächste Prozessstart das Objekt sofort wieder an.
|
||||||
|
3. **Datenverlust explizit benennen** — was ist danach unwiederbringlich weg,
|
||||||
|
was ist regenerierbar (z.B. abgeleitete Indizes wie `ocr_words`).
|
||||||
|
4. **WORM/GoBD-Hinweis**: klarstellen, dass kein archiviertes File unter
|
||||||
|
`store/` und kein `retain_until` berührt wird. Down-SQL darf niemals
|
||||||
|
Dokument-Nutzdaten oder Aufbewahrungssperren löschen.
|
||||||
|
5. Idempotent formulieren (`DROP ... IF EXISTS`) und in `BEGIN; ... COMMIT;`
|
||||||
|
klammern.
|
||||||
|
|
||||||
|
Ausführung ist immer **manuell und bewusst** (`psql -f
|
||||||
|
internal/storage/migrations/NNN_name.down.sql`), nie automatisch beim Start.
|
||||||
|
|
||||||
|
Beispiele (rückwirkend ergänzt, dienen als Vorlage):
|
||||||
|
`024_ocr_words.down.sql`, `025_document_date_score.down.sql`,
|
||||||
|
`026_accounting_api_keys.down.sql`. Ältere Migrationen (001–023) haben keine
|
||||||
|
Down-Datei — sie beschreiben das etablierte Kernschema, dessen Rückbau kein
|
||||||
|
realistisches Szenario ist.
|
||||||
|
|
||||||
## Vorhandene Migrationen
|
## Vorhandene Migrationen
|
||||||
|
|
||||||
|
|||||||
@@ -340,6 +340,36 @@ func (s *Store) ReapStaleJobs(ctx context.Context, timeout time.Duration, maxRet
|
|||||||
return len(stale), nil
|
return len(stale), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CountProcessingJobsByStatus liefert die Queue-Länge je Status über ALLE
|
||||||
|
// Mandanten hinweg. Bewusst ohne tenant_id-Filter: einziger Aufrufer ist der
|
||||||
|
// betriebsinterne Prometheus-Endpunkt GET /metrics (FDN-08), der nur
|
||||||
|
// aggregierte Zahlen ohne Mandantenbezug ausgibt — es verlassen keine
|
||||||
|
// mandantenbezogenen Daten das System. Für mandantenbezogene Auswertungen
|
||||||
|
// niemals diese Funktion nutzen.
|
||||||
|
func (s *Store) CountProcessingJobsByStatus(ctx context.Context) (map[string]int64, error) {
|
||||||
|
rows, err := s.db.Query(ctx, `
|
||||||
|
SELECT status, count(*) FROM processing_jobs GROUP BY status
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("storage: count processing jobs by status: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
out := map[string]int64{}
|
||||||
|
for rows.Next() {
|
||||||
|
var status string
|
||||||
|
var n int64
|
||||||
|
if err := rows.Scan(&status, &n); err != nil {
|
||||||
|
return nil, fmt.Errorf("storage: scan processing job count: %w", err)
|
||||||
|
}
|
||||||
|
out[status] = n
|
||||||
|
}
|
||||||
|
if err := rows.Err(); err != nil {
|
||||||
|
return nil, fmt.Errorf("storage: iterate processing job counts: %w", err)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
// RequeueJob stellt einen Job (typischerweise einen dauerhaft 'failed'
|
// RequeueJob stellt einen Job (typischerweise einen dauerhaft 'failed'
|
||||||
// gelaufenen) wieder in die Queue und setzt retry_count zurück. Wird vom
|
// gelaufenen) wieder in die Queue und setzt retry_count zurück. Wird vom
|
||||||
// späteren manuellen Retry-Endpunkt (Phase 3) genutzt; tenant-scoped.
|
// späteren manuellen Retry-Endpunkt (Phase 3) genutzt; tenant-scoped.
|
||||||
|
|||||||
Reference in New Issue
Block a user