Gesamtprojekt: origin/integration/dms-gesamtstand nachgezogen (INT-20, DOC-22, SRC-14, FDN-22, FDN-23, QA-14)
CI / Web-Barrierefreiheitspruefung (axe, Tastatur, Reflow, Textabstaende) (push) Canceled after 0s
CI / Core-Modul (Build/Vet/Test) (push) Canceled after 0s
CI / DMS Security-Test-Gate (AC5, Merge-Gate) (push) Canceled after 0s
CI / DMS-Modul Supply-Chain-Scan (govulncheck) (push) Canceled after 0s
CI / Web (dms-browser) (push) Canceled after 0s
CI / Web (pdf-viewer) (push) Canceled after 0s
CI / Web (shl) (push) Canceled after 0s
CI / DMS-Modul (Lint/Build/Test/Coverage/Artefakt) (push) Canceled after 0s
CI / Web-Barrierefreiheitspruefung (axe, Tastatur, Reflow, Textabstaende) (push) Canceled after 0s
CI / Core-Modul (Build/Vet/Test) (push) Canceled after 0s
CI / DMS Security-Test-Gate (AC5, Merge-Gate) (push) Canceled after 0s
CI / DMS-Modul Supply-Chain-Scan (govulncheck) (push) Canceled after 0s
CI / Web (dms-browser) (push) Canceled after 0s
CI / Web (pdf-viewer) (push) Canceled after 0s
CI / Web (shl) (push) Canceled after 0s
CI / DMS-Modul (Lint/Build/Test/Coverage/Artefakt) (push) Canceled after 0s
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
commit
536d02556f
74 files changed
+7412
-130
No files matched your search
@@ -13,3 +13,4 @@ ai 8900 8999
|
||||
connect 9500 9599
|
||||
# dms-Port 8610: KoSIT-Validator (MET-07, nur 127.0.0.1), siehe deploy/kosit/README.md
|
||||
# dms-Port 8611: Core-Stellvertreter des FDN-19-Smoke-Tests (cmd/smoke-tool, nur 127.0.0.1, nur Testhost)
|
||||
# dms-Port 8612: veraPDF-Sidecar (SRC-14, nur 127.0.0.1), siehe deploy/verapdf/README.md
|
||||
|
@@ -0,0 +1,79 @@
|
||||
# veraPDF-Sidecar (SRC-14)
|
||||
|
||||
Prueft die PDF/A-Derivate aus SRC-06 gegen veraPDF, den Referenz-Validator fuer
|
||||
PDF/A. Zwei Teile: die offizielle veraPDF-CLI unter `/opt/nexarch-verapdf`
|
||||
(Java, unveraendert vom Hersteller) und ein duenner, selbst geschriebener
|
||||
HTTP-Wrapper `verapdf-sidecar` (Go, `dms/cmd/verapdf-sidecar`), der je Anfrage
|
||||
einen eigenen veraPDF-Lauf startet. Nur auf `127.0.0.1:8612`
|
||||
(DMS-Portbereich 8600-8699, `deploy/ports.tsv`; 8610 ist der KoSIT-Sidecar aus
|
||||
MET-07). Der DMS-Worker spricht ihn ueber `NEXARCH_DMS_WORKER_VERAPDF_ADDR`
|
||||
an. Ohne diese Variable ist das Ergebnis immer `not_checked`.
|
||||
|
||||
veraPDF hat keinen dokumentierten Daemon-Modus; jede Pruefung startet einen
|
||||
eigenen kurzlebigen Java-Prozess (ueblicherweise < 1 s). Das ist das vom
|
||||
Hersteller unterstuetzte Betriebsmodell, nicht ein Workaround.
|
||||
|
||||
## Bestandteile (feste Versionen, Pruefsummen)
|
||||
|
||||
| Datei | Version | Pruefsumme |
|
||||
|---|---|---|
|
||||
| Temurin JRE (Linux x64), wiederverwendet aus `/opt/nexarch-kosit/jre` | 21.0.12.1 | SHA-256 `2413149700df0f7d440500a84a8f764c535f21e5a5e87d38328b64eec2c5b500` (siehe `deploy/kosit/README.md`) |
|
||||
| `verapdf-installer.zip` | veraPDF Greenfield 1.30.2, gebaut 2026-06-03 | SHA-256 `6cc6341cb1af644044054b81f00a6590a7918abb18f762243de115258bcad838`, zusaetzlich per GPG-Signatur (`.asc`) gegen den Schluessel `13DD102B4DD69354D12DE5A83184863278B17FE7` (Carl Wilson, veraPDF-Techlead, `keyserver.ubuntu.com`) geprueft |
|
||||
|
||||
Quellen: `github.com/adoptium/temurin21-binaries`, `software.verapdf.org/releases/`.
|
||||
Der Installer selbst ist ein IzPack-Assistent ohne separat verzeichnete
|
||||
SHA-256-Summe; der Hersteller sichert die Downloads per GPG-Signatur.
|
||||
|
||||
## Installation
|
||||
|
||||
Der Installer laesst sich nicht ganz unbeaufsichtigt starten (kein
|
||||
`auto-install.xml` im Zip); die Konsolen-Antworten sind aber fest und
|
||||
reproduzierbar:
|
||||
|
||||
```
|
||||
# JRE, Installer und Signatur laden, Signatur pruefen (siehe oben)
|
||||
useradd --system --no-create-home --home-dir /opt/nexarch-verapdf --shell /usr/sbin/nologin nexarch-verapdf
|
||||
|
||||
# Nur die CLI installieren (keine GUI, keine Doku, keine Beispiel-Plugins):
|
||||
export JAVA_HOME=/opt/nexarch-kosit/jre PATH=/opt/nexarch-kosit/jre/bin:$PATH
|
||||
printf '1\n/opt/nexarch-verapdf\nO\n1\nN\nY\nN\nN\n1\n1\n1\n1\n1\n' \
|
||||
| java -jar verapdf-izpack-installer-1.30.2.jar -console
|
||||
|
||||
# Sidecar bauen und ablegen (aus diesem Branch):
|
||||
cd dms && go build -o /opt/nexarch-verapdf/sidecar/verapdf-sidecar ./cmd/verapdf-sidecar
|
||||
|
||||
chown -R root:root /opt/nexarch-verapdf && chmod -R go-w /opt/nexarch-verapdf
|
||||
cp deploy/verapdf/nexarch-dms-verapdf.service /etc/systemd/system/
|
||||
systemctl daemon-reload && systemctl enable --now nexarch-dms-verapdf
|
||||
```
|
||||
|
||||
Die Eingabefolge des Installers (Konsolenmodus, Sprache Deutsch): 1 = Hinweis
|
||||
bestaetigen, Zielpfad `/opt/nexarch-verapdf`, O = Verzeichnis anlegen,
|
||||
1 = weiter, dann je Paket N/Y/N/N (GUI aus, CLI an, Dokumentation aus,
|
||||
Beispiel-Plugins aus), danach 1 mehrfach fuer Installation, Abschluss und die
|
||||
Frage nach einem automatischen Installationsskript.
|
||||
|
||||
## Betrieb
|
||||
|
||||
- Worker: `NEXARCH_DMS_WORKER_VERAPDF_ADDR=http://127.0.0.1:8612`, optional
|
||||
`NEXARCH_DMS_WORKER_VERAPDF_FLAVOUR` (Standard `2b`) und
|
||||
`NEXARCH_DMS_WORKER_VERAPDF_TIMEOUT_SECONDS`.
|
||||
- Der Sidecar selbst braucht `NEXARCH_VERAPDF_SIDECAR_BIN` (Pfad zum
|
||||
`verapdf`-Startskript, Standard `/opt/nexarch-verapdf/verapdf`) und
|
||||
`JAVA_HOME`/`PATH` fuer die Java-Laufzeit (Environment-Zeilen in der Unit).
|
||||
- Der Dienst hat keine Zugriffskontrolle: nie an eine andere Adresse als
|
||||
Loopback binden.
|
||||
- Der Sidecar meldet einen fehlgeschlagenen oder zeitueberschrittenen Lauf als
|
||||
HTTP 502/504; der DMS-Client wertet das als `not_checked`, nie `compliant`.
|
||||
|
||||
## Rueckbau
|
||||
|
||||
```
|
||||
systemctl disable --now nexarch-dms-verapdf
|
||||
rm /etc/systemd/system/nexarch-dms-verapdf.service && systemctl daemon-reload
|
||||
rm -rf /opt/nexarch-verapdf
|
||||
userdel nexarch-verapdf
|
||||
```
|
||||
|
||||
Bereits abgelegte Ergebnisse (`revision_pdfa_conformance`) bleiben erhalten;
|
||||
ohne Sidecar entstehen neue Ergebnisse mit `not_checked`.
|
||||
@@ -0,0 +1,28 @@
|
||||
[Unit]
|
||||
Description=NEXARCH DMS veraPDF Sidecar (PDF/A-Konformitaet, nur Loopback)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=nexarch-verapdf
|
||||
Group=nexarch-verapdf
|
||||
WorkingDirectory=/opt/nexarch-verapdf
|
||||
Environment=JAVA_HOME=/opt/nexarch-kosit/jre
|
||||
Environment=PATH=/opt/nexarch-kosit/jre/bin:/usr/bin:/bin
|
||||
Environment=NEXARCH_VERAPDF_SIDECAR_ADDR=127.0.0.1:8612
|
||||
Environment=NEXARCH_VERAPDF_SIDECAR_BIN=/opt/nexarch-verapdf/verapdf
|
||||
Environment=NEXARCH_VERAPDF_SIDECAR_TIMEOUT_SECONDS=30
|
||||
ExecStart=/opt/nexarch-verapdf/sidecar/verapdf-sidecar
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
MemoryMax=768M
|
||||
TasksMax=64
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadOnlyPaths=/opt/nexarch-verapdf
|
||||
RestrictAddressFamilies=AF_INET AF_UNIX
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
+38
-15
@@ -27,6 +27,7 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/searchsync"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/shared"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trigger"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/upload"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/workerruntime"
|
||||
)
|
||||
@@ -89,6 +90,8 @@ func newUploadStack(pool *pgxpool.Pool, stagingDir string) uploadStack {
|
||||
if !hasScan {
|
||||
log.Print("dms-app: Virenscan wird beim Import nicht eingereiht (NEXARCH_DMS_APP_ENQUEUE_JOB_TYPES ohne malwarescan.scan): importierte Dokumente werden ungescannt angezeigt")
|
||||
}
|
||||
// INT-20: Trigger feuern synchron nach Complete, keine Job-Queue noetig.
|
||||
svc.SetTriggerFirer(&trigger.Adapter{Pool: pool, Vault: trigger.CryptoVault{Crypto: cryptoSvc}, TenantSlug: tenantSlug})
|
||||
return uploadStack{svc: svc, storage: storageSvc, crypto: cryptoSvc}
|
||||
}
|
||||
|
||||
@@ -99,6 +102,21 @@ func envSeconds(name string, def time.Duration) time.Duration {
|
||||
return def
|
||||
}
|
||||
|
||||
func envInt(name string, def int) int {
|
||||
if v, err := strconv.Atoi(os.Getenv(name)); err == nil && v > 0 {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func envBool(name string, def bool) bool {
|
||||
v := strings.TrimSpace(strings.ToLower(os.Getenv(name)))
|
||||
if v == "" {
|
||||
return def
|
||||
}
|
||||
return v == "true" || v == "1" || v == "yes"
|
||||
}
|
||||
|
||||
func splitIDs(raw string) []string {
|
||||
var out []string
|
||||
for _, id := range strings.Split(raw, ",") {
|
||||
@@ -125,21 +143,26 @@ func setupAPI(api *http.ServeMux, dsn, prefix string) error {
|
||||
stack := newUploadStack(pool, stagingDir)
|
||||
tenantSlug := os.Getenv("NEXARCH_DMS_APP_TENANT_SLUG")
|
||||
return apiwiring.Mount(api, apiwiring.Deps{
|
||||
ObjectACL: objectACLClientFor(tenantSlug),
|
||||
Pool: pool,
|
||||
Storage: stack.storage,
|
||||
Crypto: stack.crypto,
|
||||
TenantSlug: tenantSlug,
|
||||
Upload: stack.svc,
|
||||
DownloadSecret: []byte(os.Getenv("NEXARCH_DMS_APP_DOWNLOAD_SIGNING_SECRET")),
|
||||
DownloadTTL: envSeconds("NEXARCH_DMS_APP_DOWNLOAD_URL_TTL_SECONDS", 5*time.Minute),
|
||||
TrashRetention: envSeconds("NEXARCH_DMS_APP_TRASH_RETENTION_SECONDS", 30*24*time.Hour),
|
||||
Auditors: splitIDs(os.Getenv("NEXARCH_DMS_APP_AUDITOR_USER_IDS")),
|
||||
SearchEnqueuer: searchEnqueuerFor(pool),
|
||||
LegalHolds: legalHoldCheckerFor(os.Getenv("NEXARCH_DMS_APP_RETENTION_BASE_URL"), os.Getenv("NEXARCH_DMS_APP_TENANT_SLUG"), os.Getenv("NEXARCH_DMS_APP_RETENTION_SERVICE_TOKEN")),
|
||||
PathPrefix: prefix,
|
||||
MailStore: &mailimport.Store{Pool: pool, Crypto: stack.crypto, TenantSlug: tenantSlug},
|
||||
EinvoiceEnqueuer: &einvoice.Enqueuer{Pool: pool, Queue: jobqueue.NewQueue(pool, 5*time.Minute)},
|
||||
ObjectACL: objectACLClientFor(tenantSlug),
|
||||
Pool: pool,
|
||||
Storage: stack.storage,
|
||||
Crypto: stack.crypto,
|
||||
TenantSlug: tenantSlug,
|
||||
Upload: stack.svc,
|
||||
DownloadSecret: []byte(os.Getenv("NEXARCH_DMS_APP_DOWNLOAD_SIGNING_SECRET")),
|
||||
DownloadTTL: envSeconds("NEXARCH_DMS_APP_DOWNLOAD_URL_TTL_SECONDS", 5*time.Minute),
|
||||
TrashRetention: envSeconds("NEXARCH_DMS_APP_TRASH_RETENTION_SECONDS", 30*24*time.Hour),
|
||||
Auditors: splitIDs(os.Getenv("NEXARCH_DMS_APP_AUDITOR_USER_IDS")),
|
||||
SearchEnqueuer: searchEnqueuerFor(pool),
|
||||
LegalHolds: legalHoldCheckerFor(os.Getenv("NEXARCH_DMS_APP_RETENTION_BASE_URL"), os.Getenv("NEXARCH_DMS_APP_TENANT_SLUG"), os.Getenv("NEXARCH_DMS_APP_RETENTION_SERVICE_TOKEN")),
|
||||
PathPrefix: prefix,
|
||||
MailStore: &mailimport.Store{Pool: pool, Crypto: stack.crypto, TenantSlug: tenantSlug},
|
||||
EinvoiceEnqueuer: &einvoice.Enqueuer{Pool: pool, Queue: jobqueue.NewQueue(pool, 5*time.Minute)},
|
||||
ShareLinksEnabled: envBool("NEXARCH_DMS_APP_SHARE_LINKS_ENABLED", true),
|
||||
ShareLinkDefaultTTL: envSeconds("NEXARCH_DMS_APP_SHARE_LINK_DEFAULT_TTL_SECONDS", 7*24*time.Hour),
|
||||
ShareLinkMaxTTL: envSeconds("NEXARCH_DMS_APP_SHARE_LINK_MAX_TTL_SECONDS", 14*24*time.Hour),
|
||||
ShareLinkMaxAttempts: envInt("NEXARCH_DMS_APP_SHARE_LINK_MAX_ATTEMPTS", 5),
|
||||
ShareLinkLockDuration: envSeconds("NEXARCH_DMS_APP_SHARE_LINK_LOCKOUT_SECONDS", 15*time.Minute),
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
// textcrypt-migrate verschluesselt vorhandene Klartext-Textkopien (FDN-22) oder stellt sie
|
||||
// fuer das Zurueckrollen der Migration 0031 wieder her. Ohne Optionen wird verschluesselt.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/textcrypt"
|
||||
)
|
||||
|
||||
func main() {
|
||||
decrypt := flag.Bool("decrypt", false, "Klartext wiederherstellen (vor dem Zurueckrollen von 0031)")
|
||||
countOnly := flag.Bool("count", false, "nur zaehlen, nichts aendern")
|
||||
flag.Parse()
|
||||
|
||||
dsn := os.Getenv("NEXARCH_DMS_WORKER_TENANT_DSN")
|
||||
tenant := os.Getenv("NEXARCH_DMS_WORKER_TENANT_SLUG")
|
||||
kekURL := os.Getenv("NEXARCH_DMS_WORKER_KEK_ENDPOINT_URL")
|
||||
if dsn == "" || tenant == "" || (kekURL == "" && !*countOnly) {
|
||||
log.Fatal("NEXARCH_DMS_WORKER_TENANT_DSN, _TENANT_SLUG und _KEK_ENDPOINT_URL muessen gesetzt sein")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
log.Fatalf("db: %v", err)
|
||||
}
|
||||
defer pool.Close()
|
||||
|
||||
report := func(label string, s textcrypt.Stats) {
|
||||
fmt.Printf("%s: ocr-seiten=%d klartext-extrakt=%d e-rechnung=%d\n", label, s.OCRPages, s.TextIndex, s.EInvoice)
|
||||
}
|
||||
if *countOnly {
|
||||
legacy, err := textcrypt.CountLegacy(ctx, pool)
|
||||
if err != nil {
|
||||
log.Fatalf("zaehlen: %v", err)
|
||||
}
|
||||
enc, err := textcrypt.CountEncrypted(ctx, pool)
|
||||
if err != nil {
|
||||
log.Fatalf("zaehlen: %v", err)
|
||||
}
|
||||
report("klartext", legacy)
|
||||
report("verschluesselt", enc)
|
||||
return
|
||||
}
|
||||
v := textcrypt.Vault{
|
||||
Crypto: dmscrypto.NewService(dmscrypto.NewHTTPKEKProvider(kekURL, os.Getenv("NEXARCH_DMS_WORKER_KEK_CLIENT_ID"), os.Getenv("NEXARCH_DMS_WORKER_KEK_CLIENT_SECRET"), nil)),
|
||||
TenantSlug: tenant,
|
||||
}
|
||||
var done textcrypt.Stats
|
||||
if *decrypt {
|
||||
done, err = textcrypt.DecryptAll(ctx, pool, v)
|
||||
report("entschluesselt", done)
|
||||
} else {
|
||||
done, err = textcrypt.MigrateLegacy(ctx, pool, v)
|
||||
report("verschluesselt", done)
|
||||
}
|
||||
if err != nil {
|
||||
log.Fatalf("abgebrochen: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
// verapdf-sidecar ist ein duenner, lokal gebundener HTTP-Wrapper um die
|
||||
// veraPDF-Kommandozeile (SRC-14). Er haelt keinen eigenen Java-Prozess am
|
||||
// Leben; jede Anfrage startet einen eigenen, kurzlebigen veraPDF-Lauf. Das
|
||||
// entspricht dem unterstuetzten Betriebsmodell des Werkzeugs (kein
|
||||
// dokumentierter Daemon-Modus) und haelt den Sidecar selbst sehr klein.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"time"
|
||||
)
|
||||
|
||||
const maxPDFBytes = 64 << 20
|
||||
|
||||
var reFlavour = regexp.MustCompile(`^[a-z0-9]{1,8}$`)
|
||||
|
||||
func main() {
|
||||
addr := envOr("NEXARCH_VERAPDF_SIDECAR_ADDR", "127.0.0.1:8611")
|
||||
bin := envOr("NEXARCH_VERAPDF_SIDECAR_BIN", "/opt/nexarch-verapdf/verapdf")
|
||||
timeout := 30 * time.Second
|
||||
if v := os.Getenv("NEXARCH_VERAPDF_SIDECAR_TIMEOUT_SECONDS"); v != "" {
|
||||
var secs int
|
||||
if _, err := fmt.Sscanf(v, "%d", &secs); err == nil && secs > 0 {
|
||||
timeout = time.Duration(secs) * time.Second
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(bin); err != nil {
|
||||
log.Fatalf("verapdf-sidecar: werkzeug %q nicht gefunden: %v", bin, err)
|
||||
}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) })
|
||||
mux.HandleFunc("POST /validate", handleValidate(bin, timeout))
|
||||
|
||||
ln, err := net.Listen("tcp", addr)
|
||||
if err != nil {
|
||||
log.Fatalf("verapdf-sidecar: %v", err)
|
||||
}
|
||||
srv := &http.Server{
|
||||
Handler: mux,
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
ReadTimeout: timeout + 10*time.Second,
|
||||
WriteTimeout: timeout + 10*time.Second,
|
||||
IdleTimeout: 60 * time.Second,
|
||||
}
|
||||
log.Printf("verapdf-sidecar: hoert auf %s, werkzeug %s, zeitlimit %s", addr, bin, timeout)
|
||||
log.Fatal(srv.Serve(ln))
|
||||
}
|
||||
|
||||
func envOr(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func handleValidate(bin string, timeout time.Duration) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
flavour := r.URL.Query().Get("flavour")
|
||||
if flavour == "" {
|
||||
flavour = "2b"
|
||||
}
|
||||
if !reFlavour.MatchString(flavour) {
|
||||
writeError(w, http.StatusBadRequest, "ungueltiges profil")
|
||||
return
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, maxPDFBytes+1))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "anfrage nicht lesbar")
|
||||
return
|
||||
}
|
||||
if len(body) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "leerer inhalt")
|
||||
return
|
||||
}
|
||||
if len(body) > maxPDFBytes {
|
||||
writeError(w, http.StatusRequestEntityTooLarge, "derivat groesser als das limit")
|
||||
return
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "verapdf-sidecar-*")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "temporaeres verzeichnis nicht anlegbar")
|
||||
return
|
||||
}
|
||||
defer func() { _ = os.RemoveAll(dir) }()
|
||||
path := dir + "/derivat.pdf"
|
||||
if err := os.WriteFile(path, body, 0o600); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "derivat nicht schreibbar")
|
||||
return
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(r.Context(), timeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, bin, "--format", "json", "--flavour", flavour, path)
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = &stdout, &stderr
|
||||
runErr := cmd.Run()
|
||||
if ctx.Err() != nil {
|
||||
writeError(w, http.StatusGatewayTimeout, "zeituberschreitung beim pruefen")
|
||||
return
|
||||
}
|
||||
if runErr != nil {
|
||||
var exitErr *exec.ExitError
|
||||
if !errors.As(runErr, &exitErr) || stdout.Len() == 0 {
|
||||
log.Printf("verapdf-sidecar: lauf fehlgeschlagen: %v; stderr=%q", runErr, truncate(stderr.String(), 500))
|
||||
writeError(w, http.StatusBadGateway, "veraPDF-lauf fehlgeschlagen")
|
||||
return
|
||||
}
|
||||
// veraPDF kann einen Exit-Code ungleich 0 liefern, obwohl ein
|
||||
// auswertbarer Bericht auf stdout steht (z. B. bei Dateien, die
|
||||
// nicht geparst werden konnten); der Bericht selbst entscheidet.
|
||||
}
|
||||
if !json.Valid(stdout.Bytes()) {
|
||||
log.Printf("verapdf-sidecar: bericht kein gueltiges json; stderr=%q", truncate(stderr.String(), 500))
|
||||
writeError(w, http.StatusBadGateway, "bericht kein gueltiges json")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(stdout.Bytes())
|
||||
}
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
@@ -21,11 +21,13 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/einvoice"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/errortracking"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/httpharden"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/integrity"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/logging"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/mailimport"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/malwarescan"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/metrics"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/pdfaconform"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/retentionclient"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/retentiondestroy"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/searchindex"
|
||||
@@ -224,6 +226,27 @@ func startJobWorkers(ctx context.Context, pool *pgxpool.Pool, queue *jobqueue.Qu
|
||||
} else {
|
||||
log.Print("dms-worker: kein e-rechnungs-validator (NEXARCH_DMS_WORKER_KOSIT_ADDR) konfiguriert, Konformitaet bleibt not_checked")
|
||||
}
|
||||
if addr := os.Getenv("NEXARCH_DMS_WORKER_VERAPDF_ADDR"); addr != "" {
|
||||
timeout := 30 * time.Second
|
||||
if v, err := strconv.Atoi(os.Getenv("NEXARCH_DMS_WORKER_VERAPDF_TIMEOUT_SECONDS")); err == nil && v > 0 {
|
||||
timeout = time.Duration(v) * time.Second
|
||||
}
|
||||
val, err := pdfaconform.NewValidator(addr, os.Getenv("NEXARCH_DMS_WORKER_VERAPDF_FLAVOUR"), timeout)
|
||||
if err != nil {
|
||||
log.Printf("dms-worker: pdf/a-validator ungueltig konfiguriert, Ergebnis bleibt not_checked: %v", err)
|
||||
} else {
|
||||
deps.PDFAValidator = val
|
||||
}
|
||||
} else {
|
||||
log.Print("dms-worker: kein pdf/a-validator (NEXARCH_DMS_WORKER_VERAPDF_ADDR) konfiguriert, Konformitaet bleibt not_checked")
|
||||
}
|
||||
integrityCfg, integritySched, integrityOn, err := integrity.ConfigFromEnv(os.Getenv, "NEXARCH_DMS_WORKER_INTEGRITY_")
|
||||
if err != nil {
|
||||
log.Fatalf("dms-worker: integritaetspruefung: %v", err)
|
||||
}
|
||||
if integrityOn {
|
||||
deps.Integrity = &integrityCfg
|
||||
}
|
||||
rt := workerruntime.New(deps)
|
||||
rt.Middleware = func(next jobqueue.Handler) jobqueue.Handler {
|
||||
return func(ctx context.Context, job *jobqueue.Job) error {
|
||||
@@ -247,6 +270,9 @@ func startJobWorkers(ctx context.Context, pool *pgxpool.Pool, queue *jobqueue.Qu
|
||||
go mailimport.RunScheduler(ctx, pool, queue, 15*time.Second)
|
||||
}
|
||||
go rt.Run(ctx, concurrency, workerruntime.HostName(), time.Second)
|
||||
if integrityOn {
|
||||
go integrity.RunScheduler(ctx, pool, queue, integritySched, 30*time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func signedURLMaxTTL(raw string) time.Duration {
|
||||
|
||||
@@ -130,6 +130,14 @@ Schnittstellenbeschreibung (ohne Anmeldung):
|
||||
GET /openapi.json -> 200 anonymous
|
||||
```
|
||||
|
||||
## Freigabelinks (DOC-22)
|
||||
|
||||
Ein Dokument lässt sich ohne DMS-Konto teilen: `POST /documents/{id}/share-links` (nur der Eigentümer, Rechte `dms.sharelink.write`) legt einen Link mit Pflichtablauf, optionalem Passwort und optionalem Download-Limit an; die Antwort enthält den Token nur dieses eine Mal in der URL. `GET /documents/{id}/share-links` (Rechte `dms.sharelink.read`) listet bestehende Links ohne Token/Passwort, `DELETE /documents/{id}/share-links/{linkID}` widerruft sofort.
|
||||
|
||||
Der Abruf läuft ohne Anmeldung über `GET /share/{token}` bzw. `POST /share/{token}` (Passwort im JSON-Body, nie in der URL oder im Query-String) und ist in der OpenAPI-Beschreibung als `"security": []` markiert. Jeder nicht nutzbare Ausgang (unbekannt, abgelaufen, widerrufen, Limit erreicht) liefert einheitlich `404`; ein gesetztes Passwort liefert ohne Angabe `401 password_required`, nach zu vielen Fehlversuchen `423 locked`.
|
||||
|
||||
Konfiguration per Umgebungsvariable (siehe `BETRIEB-UMGEBUNGSVARIABLEN.md`): `NEXARCH_DMS_APP_SHARE_LINKS_ENABLED` (Mandanten-Schalter), `NEXARCH_DMS_APP_SHARE_LINK_DEFAULT_TTL_SECONDS`, `NEXARCH_DMS_APP_SHARE_LINK_MAX_TTL_SECONDS`, `NEXARCH_DMS_APP_SHARE_LINK_MAX_ATTEMPTS`, `NEXARCH_DMS_APP_SHARE_LINK_LOCKOUT_SECONDS`.
|
||||
|
||||
## Optionale CMIS-Schnittstelle
|
||||
|
||||
Neben der REST-API gibt es eine abschaltbare CMIS-1.1-Schnittstelle (Browser Binding) unter `/api/dms/v1/cmis/browser`. Sie nutzt intern diese REST-API und damit dieselben Rechte- und Mandantenregeln; Einzelheiten in [CMIS.md](CMIS.md).
|
||||
@@ -26,6 +26,11 @@ Aus dem Code abgeleitet (`cmd/app`, `cmd/worker`, `cmd/migrate`, `internal/route
|
||||
| `NEXARCH_DMS_APP_DOWNLOAD_SIGNING_SECRET` | Pflicht bei DSN | Schlüssel für Download-URLs (Secret) |
|
||||
| `NEXARCH_DMS_APP_DOWNLOAD_URL_TTL_SECONDS` | 300 | Gültigkeit signierter Download-URLs |
|
||||
| `NEXARCH_DMS_APP_TRASH_RETENTION_SECONDS` | 2592000 (30 Tage) | Papierkorb-Frist |
|
||||
| `NEXARCH_DMS_APP_SHARE_LINKS_ENABLED` | `true` | Mandanten-Schalter für externe Freigabelinks (DOC-22) |
|
||||
| `NEXARCH_DMS_APP_SHARE_LINK_DEFAULT_TTL_SECONDS` | 604800 (7 Tage) | Standard-Gültigkeit neuer Freigabelinks |
|
||||
| `NEXARCH_DMS_APP_SHARE_LINK_MAX_TTL_SECONDS` | 1209600 (14 Tage) | Obergrenze der Gültigkeit, nicht pro Aufruf aushebelbar |
|
||||
| `NEXARCH_DMS_APP_SHARE_LINK_MAX_ATTEMPTS` | 5 | Fehlversuche bis zur Sperre |
|
||||
| `NEXARCH_DMS_APP_SHARE_LINK_LOCKOUT_SECONDS` | 900 (15 Minuten) | Sperrdauer nach zu vielen Fehlversuchen |
|
||||
| `NEXARCH_DMS_APP_AUDITOR_USER_IDS` | leer | Nutzer-IDs mit Zugriff auf `/audit-log`. Bewusst getrennt von Admin (Aufgabentrennung): auch ein `tenant_admin` mit dem Recht `dms.audit.read` braucht hier einen Eintrag |
|
||||
| `NEXARCH_DMS_APP_STORAGE_DIR` | Pflicht | Lokales Storage-Verzeichnis |
|
||||
| `NEXARCH_DMS_APP_STORAGE_SIGNING_SECRET` | Pflicht | Schlüssel für signierte URLs (Secret) |
|
||||
@@ -77,6 +82,14 @@ Aus dem Code abgeleitet (`cmd/app`, `cmd/worker`, `cmd/migrate`, `internal/route
|
||||
| `NEXARCH_DMS_WORKER_KOSIT_TIMEOUT_SECONDS` | 30 | Timeout je Prüfung |
|
||||
| `NEXARCH_DMS_WORKER_CONCURRENCY` | 2 | Parallele Worker |
|
||||
| `NEXARCH_DMS_WORKER_JOB_LEASE_SECONDS` | 300 | Sperrfrist der Jobwarteschlange in ganzen Sekunden (5 bis 3600, sonst Startabbruch). Nach hartem Abbruch laufen gehaltene Jobs nach dieser Frist weiter; laufende Jobs verlängern ihre Sperre alle Frist/3 (FDN-20) |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_INTERVAL_SECONDS` | leer (aus) | Abstand der periodischen Objektprüfung (FDN-23) in Sekunden (60 bis 2592000). Leer schaltet die Prüfung ab; ungültige Werte verhindern den Start |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_MODE` | `sample` | `sample`: je Lauf höchstens `BATCH` Revisionen, zuletzt am längsten ungeprüfte zuerst; `full`: alle Revisionen bis zum Zeitlimit |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_BATCH` | 200 | Stichprobengröße je Lauf (1 bis 100000); im Modus `full` Stapelgröße der Abfrage |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_MAX_DURATION_SECONDS` | 3600 | Zeitlimit je Lauf (10 bis 86400); danach Status `partial`, der nächste Lauf setzt fort |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_PAUSE_MS` | 50 | Pause zwischen zwei Revisionen zur Lastbegrenzung (0 bis 10000) |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_REVISION_TIMEOUT_SECONDS` | 120 | Zeitlimit je Revision (1 bis 3600) |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_MAX_OBJECT_MB` | 128 | Objekte darüber werden nicht gelesen und als `error` (übersprungen) geführt, nicht als Verletzung; das Objekt wird zum Prüfen vollständig in den Speicher gelesen (`internal/crypto`); die FDN-11-Speichergrenze des Jobtyps ist 3 × Wert + 128 MB, die Laufzeitgrenze `MAX_DURATION_SECONDS` + 60 s |
|
||||
| `NEXARCH_DMS_WORKER_INTEGRITY_WINDOW` | leer (immer) | Zeitfenster `HH:MM-HH:MM` in Serverzeit, in dem Läufe eingereiht werden; über Mitternacht erlaubt (z. B. `22:00-06:00`) |
|
||||
|
||||
## dms-migrate
|
||||
|
||||
@@ -119,3 +132,26 @@ Gilt für `dms-app` (nur Header- und Leerlauf-Timeout, keine globalen Lese-/Schr
|
||||
| `NEXARCH_HTTP_MAX_BODY_BYTES` | 1048576 | 1024 bis 1073741824 (nicht für `dms-app`) |
|
||||
|
||||
Ungültige Werte beenden den Start mit einer Fehlermeldung.
|
||||
|
||||
## Verschluesselung der Textkopien (FDN-22)
|
||||
|
||||
OCR-Text, Klartext-Extrakt (`revision_text_index`) und E-Rechnungs-Suchtext (`revision_einvoice`) liegen nur noch
|
||||
verschluesselt in der Datenbank (AES-256-GCM, DEK je Revision mit dem Mandanten-KEK verpackt, Bindung an Revision
|
||||
und Seite). Es gibt keine neue Umgebungsvariable: Der Worker nutzt denselben Schluesselbezug wie fuer die Dateien
|
||||
(`NEXARCH_DMS_WORKER_KEK_ENDPOINT_URL`, `_KEK_CLIENT_ID`, `_KEK_CLIENT_SECRET`, `_TENANT_SLUG`). Ohne Schluesseldienst
|
||||
wird kein Text abgelegt oder gelesen (Fehler statt Klartext).
|
||||
|
||||
**Ausrollen (Reihenfolge):**
|
||||
1. `migrate` anwenden (Migration `0031_text_encryption`, fuegt nur Spalten hinzu).
|
||||
2. App und Worker mit dem neuen Stand starten; neue Texte werden verschluesselt abgelegt, Altzeilen bleiben lesbar.
|
||||
3. Bestand nachverschluesseln: `textcrypt-migrate` mit den Worker-Variablen und `NEXARCH_DMS_WORKER_TENANT_DSN`.
|
||||
`textcrypt-migrate -count` zeigt Klartext- und verschluesselte Zeilen, ein zweiter Lauf aendert nichts.
|
||||
4. Erst danach ist die Datenbank frei von Klartext-Textkopien; `-count` muss `klartext: ... 0` melden.
|
||||
|
||||
**Zurueckrollen:** `textcrypt-migrate -decrypt`, danach die Down-Migration `0031` (sie verweigert sich, solange noch
|
||||
verschluesselte Texte vorhanden sind).
|
||||
|
||||
**Restrisiko Suchindex:** Der Manticore-Index enthaelt die durchsuchbaren Begriffe (und den gespeicherten Text fuer
|
||||
Ausschnitte) und ist nicht verschluesselt. Betriebsvorgabe: Manticore nur auf Loopback/internem Netz betreiben, Datentraeger
|
||||
verschluesseln (LUKS o. ae.), Zugriff auf das Datenverzeichnis auf den Dienstbenutzer beschraenken, Sicherungen des Index
|
||||
wie die Datenbank behandeln. Endgueltiges Loeschen reiht den Suchindex-Abgleich ein (nur wenn die Suchindex-Anbindung konfiguriert ist), der den Indexeintrag entfernt; der Ablauf der Papierkorbfrist (`PurgeExpired`) ist noch nicht als Dienst eingeplant.
|
||||
@@ -0,0 +1,51 @@
|
||||
# DOC-22 – Prüfprotokoll: Externe Freigabelinks mit Passwort und Ablauf
|
||||
|
||||
Basis: `origin/integration/dms-gesamtstand`. Getestet auf Entwicklungshost 204 (Wegwerf-DB `dms_dev`, seriell über `/tmp/claude-1000/runhost.sh`).
|
||||
|
||||
## Umsetzung
|
||||
|
||||
- Neues Paket `internal/sharelink` (Geschäftslogik: Token-Erzeugung/-Hash, Passwort-Hash, Ablauf/Widerruf/Limit/Sperre) und `internal/sharelinkapi` (HTTP-Bindung).
|
||||
- Migration `0033_share_links` (Tabelle `share_links`, `token_hash` und `password_hash` statt Klartext).
|
||||
- Verwaltende Endpunkte unter `/documents/{id}/share-links` (`POST`, `GET`, `DELETE .../{linkID}`) laufen wie alle dokumentbezogenen Routen durch das bestehende, bereits getestete `authz.Guard` (nur Eigentümer). Der Abrufpfad `GET`/`POST /share/{token}` ist unauthentifiziert und in `openapi.json` als `"security": []` markiert — dadurch automatisch von `moduleauth.Authenticate` und `policyclient.Enforce` ausgenommen (bestehender Mechanismus, kein neuer Code dafür nötig).
|
||||
- Mandanten-Schalter `NEXARCH_DMS_APP_SHARE_LINKS_ENABLED` (Standard an), Obergrenze `NEXARCH_DMS_APP_SHARE_LINK_MAX_TTL_SECONDS` (Standard 14 Tage), Standard-Laufzeit, Sperrschwelle und Sperrdauer ebenfalls per Env.
|
||||
- `docs/API.md` und `docs/BETRIEB-UMGEBUNGSVARIABLEN.md` ergänzt.
|
||||
|
||||
## Akzeptanzkriterien
|
||||
|
||||
1. **Erzeugen/Widerruf, hochentropischer Token nur als Hash:** `TestCreate_TokenIsHighEntropyAndOnlyHashIsStored` (256-Bit-Token, Klartext in keiner Spalte), `TestCreate_RejectsTTLAboveMax`, `TestRevoke_UnknownOrForeignLink`. Bestanden.
|
||||
2. **Unauthentifizierter Abrufpfad, keine Auslieferung fremder/gesperrter/gelöschter Dokumente, keine Interna:** `TestAccess_ServesOriginalContent` (Byte-für-Byte-Vergleich), `TestAccess_TrashedDocument_NeverServed`, `TestAccess_RevokedLink`, `TestAccess_UnknownToken_Generic404`. Bestanden.
|
||||
3. **Erzeugung/Abruf/Fehlversuch/Widerruf im Audit-Log ohne Token/Passwort, Mandant aus Instanz:** `TestList_NeverExposesTokenOrPassword`, Audit-Aufrufe in allen vier Handlern, `TestAccess_TokenFromOtherTenantIsUnknown` (zwei getrennte `testenv`-Datenbanken, strukturelle Trennung per Modell C). Bestanden.
|
||||
|
||||
## Prüfungen vor der Abnahme
|
||||
|
||||
1. **Sicherheitsfälle:** falsches Passwort (`TestAccess_PasswordFlow`, `TestResolve_PasswordProtected`), abgelaufen/widerrufen/Limit erreicht (`TestResolve_ExpiredAndRevokedAndLimitReached`), manipulierter/erratener Token (`TestAccess_UnknownToken_Generic404`, `TestResolve_WrongToken_GenericNotFound` — einheitlicher generischer 404, keine Unterscheidung), fremder Mandant (`TestAccess_TokenFromOtherTenantIsUnknown`), Dokument im Papierkorb (`TestAccess_TrashedDocument_NeverServed`). Alle bestanden.
|
||||
2. **Ratenbegrenzung/Sperre unter Last, zeitkonstanter Vergleich:** `TestResolve_LocksAfterMaxAttempts`, `TestAccess_LocksAfterFailedAttempts` (Sperre greift auch bei danach korrektem Passwort), `TestCountDownload_NeverExceedsLimitUnderConcurrency` (20 parallele Anfragen gegen Limit 5, atomare `UPDATE ... WHERE download_count < limit`, exakt 5 erfolgreich). Passwortvergleich über `bcrypt.CompareHashAndPassword` (konstantzeitig per Konstruktion). Bestanden.
|
||||
3. **Audit-Kette bleibt gültig, kein Token/Passwort im Log:** Alle vier Handler rufen `audittrail.Record` mit Metadaten, die nie Token oder Passwort enthalten (nur `share_link_id`, `expires_at`, `has_password`, `reason`). `TestList_NeverExposesTokenOrPassword` bestätigt das für die Leseseite. Kettenintegrität selbst wird vom bestehenden `audittrail`-Paket sichergestellt (unverändert). Bestanden.
|
||||
|
||||
## Gesamtlauf auf 204
|
||||
|
||||
- `go build ./...`, `go vet ./...`: sauber.
|
||||
- `go test -count=1 ./...` (kompletter Baum): alle Pakete `ok`, keine FAIL.
|
||||
- `gofmt -l`: nach Korrektur leer.
|
||||
- `scripts/check-port-defaults.sh`, `scripts/check-route-prefixes.sh`: ok (keine neuen Top-Level-Präfixe/Ports, alles unter dem bestehenden `/api/dms/v1`).
|
||||
- Bestehende Vertragstests (`internal/openapi`, `internal/apiwiring`, `internal/cmis`) bleiben grün — insbesondere `TestEveryDocumentEndpointEnforcesOwnership` und `TestEveryEndpointErrorUsesCoreFormat` (generische, bereits vorhandene Tests, die automatisch alle `openapi.Operations()` inkl. der neuen durchlaufen) und `TestPublicPathsAreExactlySignatureAndServiceRoutes` (angepasst: `/share/{token}` als vierter öffentlicher Pfad).
|
||||
|
||||
## QA und Entscheidungen
|
||||
|
||||
- **10-Iterationen-Budget:** 2 genutzt — (1) Testfixtur-Kollision durch identischen Dateiinhalt zweier Uploads (Dublettenerkennung), behoben durch eindeutigen Inhalt je Testdokument; (2) `gofmt`-Korrektur nach erster Implementierung.
|
||||
- **Eigentümerprüfung nicht neu implementiert:** Die Standardvorschlag-Formulierung des Tickets nennt „nur Eigentümer und Mandantenadministration". Das bestehende `authz.Guard` prüft für `documents/*`-Pfade ausschließlich Eigentümerschaft (kein Admin-Bypass); eine Erweiterung dieses gemeinsamen, von vielen anderen Endpunkten genutzten Mechanismus wäre ein Eingriff außerhalb des Ticket-Scopes und hätte Nebenwirkungen auf alle anderen Dokument-Routen. Entscheidung: Owner-only, konsistent mit dem Rest der API; Admin-Zugriff auf fremde Dokumente ist die bereits bekannte, separat verfolgte Lücke RBAC-11 (Objektrechte mit Vererbung).
|
||||
- **Keine In-Memory-Rate-Limiter zusätzlich zur DB-Sperre:** Die Sperre liegt in der `share_links`-Zeile selbst (`failed_attempts`, `locked_until`) und ist damit prozess- und neustartsicher. Ein zusätzlicher In-Memory-Limiter wäre für eine einzelne `dms-app`-Instanz redundant und wurde als unnötige Komplexität für die „kleinste Lösung" weggelassen.
|
||||
- **Kein Aufräum-Job für abgelaufene/widerrufene Zeilen:** Abgelaufene/widerrufene Links werden bei jedem Zugriffsversuch korrekt ausgeschlossen (funktional vollständig); ein periodischer Löschjob ist reine Speicherhygiene und keine der drei Akzeptanzkriterien verlangt ihn. Als mögliche Folgearbeit vermerkt, nicht umgesetzt (Scope-Disziplin).
|
||||
- **Dateiname im Content-Disposition:** Nutzt den Dokumenttitel (wie `downloadapi`), nicht die Revisions-ID.
|
||||
|
||||
## Bekannte Lücken / Nicht Bestandteil (laut Ticket)
|
||||
|
||||
- Keine Oberfläche zum Erzeugen/Verwalten von Links (eigenes UX-Ticket, siehe UX-11).
|
||||
- Kein Versand der Links per E-Mail.
|
||||
- Keine Freigabe von Ordnern oder mehreren Dokumenten gleichzeitig.
|
||||
- Keine anonymen Upload-URLs (INT-17).
|
||||
|
||||
## Offen (außerhalb des DMS-Moduls, cross-repo)
|
||||
|
||||
- **`dms.sharelink.write` / `dms.sharelink.read`** sind neue Rechte im `dms.*`-Namensraum (wie von `internal/openapi`/RBAC-06-Vertrag gefordert). Ist eine zentrale Core-Rechteentscheidung (RBAC-06/RBAC-09-Muster) aktiv, müssen diese zwei Rechte dort im `policy_rules`-Katalog ergänzt werden (analog zum RBAC-09-Vorgehen für die übrigen `dms.*`-Rechte), sonst werden sie dort per Default-Deny abgelehnt. Diese PR basiert auf `integration/dms-gesamtstand` (DMS-Modul allein) und kann diese Core-Migration nicht mitliefern — als Folgeticket im Core-Board vorzusehen, sobald DOC-22 in `integration/gesamtprojekt` gemergt wird.
|
||||
- Ohne aktive zentrale Rechteprüfung (`NEXARCH_DMS_APP_POLICY_URL` nicht gesetzt) gilt wie bei allen anderen Endpunkten nur die Eigentümerprüfung des Guards — das ist der heutige Standardbetrieb und keine Regression.
|
||||
@@ -0,0 +1,56 @@
|
||||
# FDN-22: Prüfprotokoll – Verschlüsselung der Textkopien (DSGVO)
|
||||
|
||||
Stand: 2026-09-22. Branch `feature/fdn-22-verschluesselung-textkopien-dms` (Basis `integration/dms-gesamtstand`, Head `0d01dfa`). Getestet auf dem Entwicklungshost 204 (Wegwerf-Datenbank `dms_dev`, je Test eigenes Schema, seriell über die gemeinsame Sperre); nichts auf 131.
|
||||
|
||||
## Umsetzung
|
||||
|
||||
- **Schlüssel:** DEK je Revision (`crypto.Service.NewDataKey`), mit dem Mandanten-KEK verpackt und mit der Revision abgelegt: `revision_ocr.text_wrapped_dek`, `revision_text_index.wrapped_dek`, `revision_einvoice.wrapped_dek`. Der KEK wird wie bei den Dateien frisch bezogen, nie zwischengespeichert.
|
||||
- **Verfahren:** AES-256-GCM, Zufalls-Nonce, Format `Version(0x01) || Nonce || Chiffretext`. Additional Data bindet an Textart, Revision und Seite (`ocr-page:<rev>:<seite>`, `text-index:<rev>`, `einvoice-text:<rev>`). Ein Chiffretext lässt sich weder auf eine andere Seite noch Revision noch Textart verschieben.
|
||||
- **Speicher:** Neues Paket `internal/textcrypt`. Neue Spalten `text_enc` (OCR-Seiten, Klartext-Extrakt) bzw. `search_text_enc` (E-Rechnung). Ein `CHECK` erzwingt, dass neben Chiffretext keine Klartextspalte gefüllt ist.
|
||||
- **Lesepfade:** `ocr.Pages`, `ocr.TextStore`, Sprachbestimmung (`ensureLanguage`, vorher SQL-`string_agg` im Klartext), `plaintext.TextStore`, `einvoice.TextStore` entschlüsseln im Speicher. Die Wiederverwendung eines OCR-Ergebnisses bei identischer Prüfsumme (`reuse`) entschlüsselt und verschlüsselt mit dem Schlüssel der neuen Revision neu (vorher `INSERT ... SELECT` im Klartext).
|
||||
- **Fail closed:** Ohne Schlüsseldienst (`ErrNoCrypto`), mit falschem oder fremdem KEK und bei manipulierten Daten (`ErrDecryptText`) gibt es einen Fehler, nie Klartext. Ein Fehler beim Lesen bricht Suchindex-Abgleich und Regelauswertung ab (Job wird wiederholt) und indexiert nicht stillschweigend ohne Text.
|
||||
- **Bestandsdaten:** Migration `0031_text_encryption` (nur Spalten). `cmd/textcrypt-migrate` verschlüsselt Klartextzeilen je Revision in einer Transaktion (idempotent), `-count` zeigt den Stand, `-decrypt` stellt den Klartext für das Zurückrollen wieder her. Die Down-Migration verweigert sich, solange Chiffretext vorhanden ist.
|
||||
- **Betrieb:** `dms/docs/BETRIEB-UMGEBUNGSVARIABLEN.md` (Reihenfolge Ausrollen/Zurückrollen, Restrisiko Suchindex).
|
||||
|
||||
## Prüfungen
|
||||
|
||||
| # | Prüfung | Zielwert | Ergebnis |
|
||||
|---|---|---|---|
|
||||
| 1 | Rohabfrage der drei Tabellen nach einem echten Lauf (OCR eines Scans, Klartext-Upload, E-Rechnung): Textspalte leer, Wort weder in Text noch im Chiffretext | 0 Treffer | bestanden (`TestRecognisedTextIsStoredEncrypted`, `TestTextIsStoredEncrypted`, `TestSearchTextIsStoredEncrypted`) |
|
||||
| 2a | Falscher KEK, fremder KEK je Textspeicher | Fehler, kein Klartext | bestanden (`TestEncryptedTextFailsClosed`, `TestRecognisedTextFailsClosedWithoutKey`) |
|
||||
| 2b | Fehlender Schlüsseldienst | Fehler, keine Zeile abgelegt | bestanden (`TestTextStoreWithoutCryptoFailsClosed`, `TestVaultFailsClosedWithoutCryptoOrWithWrongKEK`) |
|
||||
| 2c | Chiffretext einer anderen Seite/Revision/Textart, Manipulation, unbekannte Version | abgelehnt | bestanden (`TestOpenRejectsWrongKeyAADTamperAndVersion`, `TestEncryptedPageCannotBeMovedToAnotherPageOrRevision`) |
|
||||
| 3a | Bestandsdaten: Klartext vorher, verschlüsselt nachher, gelesene Ergebnisse gleich, Rohabfrage ohne Klartext | 0 Verlust | bestanden (`TestMigrateLegacy_EncryptsExistingRowsWithoutLosingData`) |
|
||||
| 3b | Zweiter Lauf | 0 Änderungen | bestanden (im selben Test) |
|
||||
| 4 | Suche, Sprachbestimmung, Regeltext, E-Rechnungs-Indexierung unverändert | dieselben Ergebnisse | bestanden: `ocr` (inkl. bestehender Sprachtests auf verschlüsselten Seiten), `qa04`, `qa05`, `workerruntime`, `einvoice` |
|
||||
| 5 | Endgültiges Löschen entfernt Textkopien und Textschlüssel | 0 Zeilen | bestanden (`TestPurgeRemovesEncryptedTextCopies`) |
|
||||
| 6 | Zurückrollen: Down-Migration verweigert bei Chiffretext, nach `DecryptAll` erlaubt, Text unverändert | wie beschrieben | bestanden (`TestDecryptAllThenDownMigrationAllowed`) |
|
||||
| 7 | `CHECK` verhindert Klartext neben Chiffretext | abgelehnt | bestanden (`TestConstraintForbidsPlaintextNextToCipher`) |
|
||||
| 8 | Wiederverwendung bei Duplikat: eigener Schlüssel, anderer Chiffretext, gleicher Text | wie beschrieben | bestanden (`TestReusedTextIsReEncryptedForNewRevision`) |
|
||||
| 9 | Binary `textcrypt-migrate` auf frischem Schema (Migration 0031, `-count`, Verschlüsseln, `-decrypt`, fehlende Umgebung) | Exit 0/1 wie erwartet | bestanden, Schema danach entfernt |
|
||||
| 10 | Volltest `./...` auf 204 | 0 FAIL | 75 Pakete `ok`, 0 FAIL; `go vet` und `gofmt` sauber |
|
||||
|
||||
## QA und Entscheidungen
|
||||
|
||||
Iterationen: 2 von bis zu 10. Beide betrafen Testannahmen bzw. einen Nebenbefund, keine Entwurfsänderung.
|
||||
|
||||
1. **Nebenbefund (behoben):** `plaintext.TextStore.TextOf` stürzte ohne Schlüsseldienst beim Lesen der Datei mit einem Nil-Zeiger ab. Jetzt `ErrNoCrypto`.
|
||||
2. **Testannahme (behoben):** Meine Prüfung der Sprachbestimmung nutzte `referencePages[0]`, bei der keine Sprache bestimmt wurde. Ob das vor FDN-22 ebenso war, habe ich nicht geprüft (der Bild-Pfad liest keinen gespeicherten Text und ist unverändert). Der Test verwendet jetzt die englische Referenzseite (Sprache `en`); die bestehenden Sprachtests aus SRC-07, darunter die Bestimmung aus dem Gesamttext eines durchsuchbaren PDFs (`ensureLanguage` auf verschlüsselten Seiten), laufen unverändert grün.
|
||||
|
||||
**Entscheidungen (mit Alternativen):**
|
||||
|
||||
- **DEK je Revision statt je Seite oder je Mandant.** Je Seite bräuchte je Seite einen KEK-Abruf (Verzögerung, Last auf dem Schlüsseldienst); ein Mandantenschlüssel erschwert Löschen und Trennung. Je Revision: ein Abruf je Vorgang, Löschen der Revision entfernt Text und Schlüssel (Kaskade).
|
||||
- **Eigener Textschlüssel statt Wiederverwendung des Dateischlüssels** (`file_revisions.wrapped_dek`). Ältere Revisionen haben teils keinen Dateischlüssel (Spalte nullable), und getrennte Schlüssel vermeiden Kopplung zwischen Datei und Textkopie.
|
||||
- **Neue Spalten und `CHECK` statt Umwandlung der Klartextspalten.** Erlaubt Ausrollen ohne Stillstand und Nachverschlüsseln im laufenden Betrieb; der `CHECK` verhindert Klartext neben Chiffretext.
|
||||
- **Suchtext der E-Rechnung einbezogen** (nicht nur die zwei im Auftrag genannten Tabellen): gleiche Schutzklasse, gleicher Fehler.
|
||||
- **Suchindex (Manticore) unverändert.** Alternativen: (a) Body nicht speichern und Ausschnitte zur Anfragezeit aus dem entschlüsselten Datenbanktext erzeugen (`CALL SNIPPETS`), (b) unverändert mit Betriebsvorgabe. Gewählt (b): (a) verlangt, dass `searchapi` Datenbank und Schlüsseldienst erreicht (heute nur Index und Rechteprüfer) und ändert Ranking/Ausschnitte; der Index enthält notwendig durchsuchbare Begriffe und lässt sich nicht verschlüsseln. Folgeticket möglich (siehe unten).
|
||||
- **Kein Klartext-Rückfall bei Fehlern**, auch nicht für Altzeilen mit Chiffretext.
|
||||
|
||||
## Restrisiken und offene Punkte
|
||||
|
||||
- **Suchindex enthält Klartext** (durchsuchbare Begriffe und gespeicherter Text für Ausschnitte). Betriebsvorgabe (Loopback/internes Netz, Datenträgerverschlüsselung, Zugriff auf den Dienstbenutzer) in `BETRIEB-UMGEBUNGSVARIABLEN.md`. Ticketvorschlag: Body im Index nicht speichern, Ausschnitte aus dem Datenbanktext.
|
||||
- **Bis zum Lauf von `textcrypt-migrate` liegen Bestandstexte weiter im Klartext.** `textcrypt-migrate -count` muss vor Abschluss `klartext: 0` melden.
|
||||
- **Während des Ausrollens** schreiben alte Worker noch Klartext (das Schema erlaubt es); trifft ein alter Worker eine bereits verschlüsselte Seite, schlägt sein Upsert am `CHECK` fehl und der Job wird wiederholt. Alle Worker auf den neuen Stand bringen, danach `textcrypt-migrate`.
|
||||
- **Nicht in diesem Ticket:** Metadatenfelder (u. a. `einvoice_*`), Fehlerliste `findings`, Schlüsselrotation der Textschlüssel (KEK-Wechsel über Umverpacken wie bei FDN-09), Verschlüsselung des Suchindex.
|
||||
- **Mit dem Gesamtprojekt zusammenführen:** Der Branch liegt auf `integration/dms-gesamtstand`. Trockenlauf-Merge gegen `integration/gesamtprojekt`: konfliktfrei (nur `BETRIEB-UMGEBUNGSVARIABLEN.md` automatisch zusammengeführt), DMS-Migration `0031` ist dort frei.
|
||||
- **`PurgeExpired` ist noch nicht als Dienst eingeplant** (Bibliothek ohne Aufrufer in `cmd/`); Textkopien verschwinden erst mit dem endgültigen Löschen.
|
||||
@@ -0,0 +1,59 @@
|
||||
# FDN-23 – Prüfprotokoll: Periodischer Integritätsjob für Objektprüfsummen
|
||||
|
||||
Basis: `integration/dms-gesamtstand` (Head `0d01dfa`). Branch `feature/fdn-23-integritaetsjob-objektpruefsummen-dms`, Migration `0032_integrity_checks`. Geprüft auf Entwicklungshost 204 (Wegwerf-Datenbank `dms_dev`, testenv-Schemas), seriell über `runhost.sh`. Hinweis zur Migrationsnummer: FDN-22 (Branch `feature/fdn-22-verschluesselung-textkopien-dms`) belegt `0031`, dieses Ticket daher `0032`; beim Zusammenführen auf Namenskollisionen der Reihenfolge achten.
|
||||
|
||||
## Umsetzung
|
||||
|
||||
- `internal/integrity`: `Verifier.Run` prüft je Revision Objekt vorhanden, entschlüsselbar, Größe und SHA-256 des Klartexts gegen `file_revisions`. Zustände `ok`, `mismatch`, `missing`, `unreadable`, `error`.
|
||||
- Tabellen `integrity_runs` (Lauf, Zähler, Status `running/completed/partial/failed`) und `integrity_results` (jüngstes Ergebnis je Revision).
|
||||
- Audit-Ereignis `integrity_violation` (Akteur `system`) bei `mismatch`, `missing`, `unreadable`, je Statuswechsel einmal.
|
||||
- Jobtyp `integrity.verify`, Planer `integrity.RunScheduler` im Worker, nur bei gesetztem `NEXARCH_DMS_WORKER_INTEGRITY_INTERVAL_SECONDS`. Umgebungsvariablen: `docs/BETRIEB-UMGEBUNGSVARIABLEN.md`.
|
||||
- `crypto.ErrKEKUnavailable` (Sentinel) trennt Schlüsseldienst-Ausfall von einer Entschlüsselungsverletzung; Fehlertext unverändert.
|
||||
|
||||
## Akzeptanzkriterien
|
||||
|
||||
| AK | Nachweis | Ergebnis |
|
||||
|---|---|---|
|
||||
| 1 Zustände je Revision und Lauf | `TestRun_DetectsAllStatusesSeparately` (sechs Revisionen: ok, falsche Prüfsumme, getauschter Inhalt, falsche Größe, fehlendes Objekt, beschädigter Chiffretext), `TestRun_WrongKeyIsUnreadable…`, `TestRun_OversizedObjectIsErrorNotViolation` | bestanden |
|
||||
| 2 Audit je Statuswechsel, Kette gültig, `error` keine Verletzung | `TestRun_AuditOncePerStatusChangeAndChainStaysValid` (drei Läufe = 1 Ereignis, Wechsel = 2, `VerifyChain` gültig), KEK-Ausfall ohne Audit | bestanden |
|
||||
| 3 Steuerbar, begrenzt, fortsetzbar, Papierkorb/gelöscht ignoriert | `TestConfigFromEnv_*`, `TestWindow`, `TestEnqueueIfDue_*`, `TestRun_SampleCoversEverything…`, `TestRun_ResumesAfterPartialRun…`, `TestRun_FullStartsNewCycle…`, `TestRun_SkipsTrashedAndDeletedDocuments`, `TestRun_SkipsRevisionsStillSettling`, `TestJobLimits_FollowConfiguration` | bestanden |
|
||||
|
||||
## Prüfungen
|
||||
|
||||
| Nr. | Prüfung | Zielwert | Messwert |
|
||||
|---|---|---|---|
|
||||
| 1 | Beschädigt/fehlend/falscher Schlüssel | mismatch bzw. unreadable, missing, unreadable; unversehrt bleibt ok | erfüllt (6 Fälle, Zusammenfassung 1 ok / 3 mismatch / 1 missing / 1 unreadable, Lauf `completed`) |
|
||||
| 2 | Abbruch und Neustart | keine Revision ausgelassen oder doppelt | erster Lauf `partial` mit Teilmenge, zweiter Lauf prüft genau die restlichen Revisionen (Zyklusbegriff), Sample-Modus deckt in drei Läufen alle sieben Revisionen ab |
|
||||
| 3 | Mandantentrennung | Ergebnisse und Audit nur im eigenen Mandanten, fremder Schlüssel nie ok | erfüllt (`TestRun_TenantSeparation`) |
|
||||
|
||||
Weitere Läufe: `go test ./...` auf 204 ohne FAIL (Volllauf nach der letzten Codeänderung an `integrity` gesondert für `integrity`, `workerruntime`, `audittrail` wiederholt), `integrity` mit `-count=2 -shuffle=on`, `go vet`, `gofmt -l` leer, `check-port-defaults.sh`, `check-route-prefixes.sh` ok. Die Endpunkt-Ende-zu-Ende-Prüfung des Jobs über den Dispatcher: `TestIntegrityJobRunsThroughDispatcherToFinalState`.
|
||||
|
||||
## QA und Entscheidungen
|
||||
|
||||
Iterationen: 6 (davon 4 inhaltlich, keine Blockade).
|
||||
|
||||
Fehler und Korrekturen im QA-Durchgang:
|
||||
1. Testerwartung falsch: Der Sample-Modus füllt nach vollständiger Abdeckung die Stichprobe mit den am längsten ungeprüften Revisionen auf. Test angepasst.
|
||||
2. Entwurfsfehler: Im Modus `full` prüfte ein Folgelauf nach Teilabbruch bereits geprüfte Revisionen erneut. Lösung: Zyklusbegriff (Beginn = erster Lauf seit dem letzten abgeschlossenen Vollauf); Revisionen, die im laufenden Zyklus schon geprüft sind, entfallen.
|
||||
3. FDN-11-Limits: Der Worker bricht jeden Job nach fünf Minuten ab und begrenzt den Heap-Zuwachs auf 512 MB; ein Lauf von einer Stunde wäre damit als Dead-Letter geendet. Neue Jobtyp-Limits (`Runtime.Limits`, `Config.JobLimits`): Laufzeit = Höchstdauer + 60 s, Speicher = 3 × größtes lesbares Objekt + 128 MB. Standard für das größte Objekt: 128 MB, damit die Speichergrenze der bisherigen Standardgrenze entspricht.
|
||||
4. Advisory-Lock: Er gilt pro Datenbank; testenv-Schemas derselben Datenbank sperrten sich (`es laeuft bereits eine pruefung`). Der Schlüssel enthält jetzt das aktuelle Schema; produktiv (eine Datenbank je Mandant) unverändert eine Prüfung je Mandant.
|
||||
5. Audit-Reihenfolge: Ein Fehler beim Audit nach dem Speichern des Ergebnisses hätte das Ereignis dauerhaft verloren. Das Audit-Ereignis wird jetzt vor dem Ergebnis geschrieben (mindestens einmal statt nie).
|
||||
6. `gofmt` (Ausrichtung einer Struktur).
|
||||
|
||||
Getroffene Entscheidungen (Alternative, Grund):
|
||||
- Auswahl „am längsten ungeprüft zuerst“ statt Zufallsstichprobe oder Cursor: deckt über mehrere Läufe alles ab, überlebt Abbrüche ohne eigenen Cursor, Abfrage ist über `checked_at` indexiert.
|
||||
- Ergebnis je Revision als letzter Stand plus Lauf-Zusammenfassung in `integrity_runs`, keine Historientabelle je Revision und Lauf: Die Historie der Verletzungen steht im Audit-Log (Hash-Kette), die Datenmenge bleibt begrenzt.
|
||||
- Revisionen von Dokumenten im Papierkorb werden nicht geprüft (nicht als Fehler gemeldet, nach Wiederherstellung geprüft); endgültig gelöschte ebenso. Revisionen jünger als zwei Minuten werden übersprungen (laufender Upload).
|
||||
- Metrik: keine neuen Prometheus-Zähler, sondern Zähler je Lauf in `integrity_runs` und eine Logzeile je Lauf ohne Dokumentnamen oder Inhalte. Das bestehende Metrikregister kennt nur Anfragen und Queue-Länge; ein Ausbau wäre eigener Umfang.
|
||||
- Job mit `MaxAttempts` 1: Ein fehlgeschlagener Lauf wird zum nächsten Intervall neu eingereiht; keine Wiederholungsstürme gegen einen gestörten Speicher.
|
||||
- Objekte über dem Limit: Status `error` mit Grund (übersprungen), keine Aussage über das Objekt.
|
||||
|
||||
## Grenzen und offene Punkte
|
||||
|
||||
- Ein Objekt wird zum Prüfen vollständig in den Speicher gelesen (Eigenschaft von `internal/crypto`); das begrenzt `MAX_OBJECT_MB`.
|
||||
- Keine API oder Oberfläche für die Ergebnisse (bewusst nicht Umfang); Lesezugriff nur per SQL.
|
||||
- Kein Wiederherstellungsablauf bei Verletzung (Archive-Backup).
|
||||
- Derivate (Vorschau, PDF/A, OCR-Text) und Suchindex werden nicht geprüft.
|
||||
- Die Zeitfenster-Angabe gilt in Serverzeit.
|
||||
- S3-Treiber: Der Fehlerpfad für fehlende Objekte ist nur über den lokalen Treiber getestet.
|
||||
- Mit FDN-22 (Textkopien verschlüsseln) berührt sich nur `internal/crypto/service.go` (Fehlerumhüllung); beim Merge auf Konflikte achten.
|
||||
@@ -0,0 +1,48 @@
|
||||
# INT-20 – Prüfprotokoll: Trigger-/Automatisierungssystem nach Upload
|
||||
|
||||
Basis: `integration/dms-gesamtstand`. Getestet auf Entwicklungshost 204 (Wegwerf-DB `dms_dev`, seriell über die gemeinsame Sperre), `archivdms` unberührt. Nichts installiert.
|
||||
|
||||
## Umsetzung
|
||||
|
||||
- **Neu:** `dms/internal/trigger` (Fachlogik: `Create`, `Get`, `List`, `SetEnabled`, `Delete`, `Fire`), `dms/internal/triggerapi` (HTTP-Verwaltung unter `/triggers`), Migration `0035_triggers` (Tabellen `triggers`, `trigger_runs`).
|
||||
- **Ereignisse (v1):** `document.uploaded` (Hook in `internal/upload.Service.Complete`, analog `AutoRuleApplier`), `document.ocr_completed` (Hook `ocr.Processor.OnText`), `document.einvoice_checked` (neuer Hook `einvoice.Processor.OnConformanceChecked`, feuert nach jeder Konformitätsprüfung inkl. Neuprüfung).
|
||||
- **Bedingung:** optionales Dateinamensmuster (Regex wie MET-05) und/oder Dokumententyp.
|
||||
- **Aktionen:** `webhook` (HTTP-POST, HMAC-SHA256-Signatur im Header `X-Trigger-Signature`, 10 s Zeitlimit, kein Redirect-Folgen), `set_tag` (ruft `tag.Assign`), `set_type` (ruft `documenttype.ChangeDocumentType`).
|
||||
- **Webhook-Geheimnis:** Envelope-verschlüsselt (Muster aus FDN-09/FDN-22), `internal/trigger.CryptoVault` nutzt den vorhandenen `dmscrypto.Service`; kein Klartext-Fallback, nie in einer API-Antwort.
|
||||
- **Verdrahtung:** `cmd/app/api.go` setzt `svc.SetTriggerFirer` (App-Prozess, synchron nach Upload), `internal/workerruntime` setzt die OCR-/E-Rechnung-Hooks (Worker-Prozess). `internal/apiwiring` hängt `triggerapi` ein, `internal/authz` bekommt den Fall `"triggers"` (GET erlaubt für jede Identität, sonst `IsAdmin`, analog `auto-rules`). OpenAPI-Tabelle um 6 Operationen mit `x-permission: dms.trigger.read`/`dms.trigger.write` ergänzt.
|
||||
|
||||
## Akzeptanzkriterien
|
||||
|
||||
1. **Trigger feuert bei passendem Ereignis/Bedingung, nicht bei unpassenden** — bestanden (`TestFire_FilenamePatternMatchAndNoMatch`, `TestFire_SetTypeAction`, `TestFire_DisabledTriggerDoesNotFire`).
|
||||
2. **Fehler in einer Aktion blockiert die Verarbeitung nicht** — bestanden (`TestFire_WebhookFailureDoesNotBlockFireOrOtherTriggers`: ein fehlschlagender Webhook liefert von `Fire` keinen Fehler, der zweite Trigger für dasselbe Ereignis feuert trotzdem, der Fehlschlag steht in `trigger_runs`).
|
||||
3. **Trigger-Verwaltung nur für Berechtigte, Geheimnis nie im Klartext** — bestanden (`TestAPI_ManageTriggersOverHTTP`: ohne Identität 400; `TestAPI_WebhookSecretNeverReturned`: Geheimnis weder in der API-Antwort noch unverschlüsselt in der Datenbank).
|
||||
|
||||
## Prüfungen vor der Abnahme
|
||||
|
||||
| Prüfung | Ergebnis |
|
||||
|---|---|
|
||||
| Webhook-SSRF-Negativfälle (Loopback, private Netze, Link-Local, ungültiges Schema) | bestanden — `TestValidateWebhookURL_RejectsPrivateAndLoopbackTargets` (7 Fälle), Gegenprobe `TestValidateWebhookURL_AcceptsPublicHTTPS` und `TestValidateWebhookURL_AllowPrivateOverride` |
|
||||
| Fehlgeschlagene Aktion blockiert weder Upload noch übrige Trigger | bestanden — `TestFire_WebhookFailureDoesNotBlockFireOrOtherTriggers` |
|
||||
| Unberechtigter Zugriff auf die Trigger-Verwaltung wird abgelehnt | bestanden auf Anwendungsebene (`handleCreate`/`handleDelete`/`handleSetEnabled` verlangen eine Identität, 400 ohne). Die feingranulare Admin-Pflicht (`authz`-Fall `"triggers"`, GET offen/Schreiben nur Admin) ist strukturell identisch zum bereits produktiven `auto-rules`-Fall, aber nicht durch einen neuen End-zu-Ende-Test über den vollen `authz.Guard` abgedeckt — kein Test dafür existierte zuvor auch für `auto-rules` (siehe „Offen“). |
|
||||
|
||||
## Gesamtlauf auf 204
|
||||
|
||||
- `go vet ./...`: sauber.
|
||||
- `go test -count=1 ./...` (gesamtes DMS-Modul): alle Pakete `ok`, kein FAIL, in zwei unabhängigen Läufen bestätigt (ein früherer Lauf zeigte ein abgeschnittenes „FAIL“ am Endes eines `tail -80`-Auszugs ohne erkennbare Ursache; beide Wiederholungen mit `grep -E FAIL` über den vollständigen Log waren leer — als Host-Lastartefakt gewertet, nicht reproduzierbar).
|
||||
- `gofmt -l .`: zunächst 2 unformatierte Dateien (`internal/trigger/trigger.go`, `internal/triggerapi/handler.go`), remote korrigiert und übernommen; danach leer.
|
||||
- `scripts/check-port-defaults.sh`, `scripts/check-route-prefixes.sh`: beide ok (keine neuen Ports/Routen außerhalb des bestehenden Präfixes).
|
||||
|
||||
## QA und Entscheidungen
|
||||
|
||||
- **Ereignisumfang v1:** Nur die drei Ereignisse, die bereits als Hook-Punkte im System existieren (Upload, OCR-Text, E-Rechnung-Konformität). „Tag hinzugefügt“ und „Typ geändert“ sind eigene Schreibpfade in `tagapi`/`documentapi` ohne Hook-Infrastruktur — laut Ticket „Nicht Bestandteil“, absichtlich nicht gebaut.
|
||||
- **`created_by` ohne Fremdschlüssel:** Erste Fassung band `created_by` als `UUID REFERENCES users(id)`; das scheiterte an System-/Testaktoren wie `"tester"` oder `audittrail.SystemActor`. Umgestellt auf `TEXT` ohne Fremdschlüssel, konsistent mit dem Freitext-Akteur-Modell von `audittrail`.
|
||||
- **SSRF-Prüfung nur zur Anlagezeit robust:** `ValidateWebhookURL` löst Hostnamen zur Anlagezeit auf; ein DNS-Rebinding zwischen Anlage und Versand wird nicht verhindert (gleiche dokumentierte Grenze wie bei F-08/F-22 im Kern-Modul). Für v1 als hinreichend bewertet, weil Trigger-Anlage bereits `IsAdmin` voraussetzt.
|
||||
- **Iterationen:** 3 Fix-Runden bis zum grünen Zustand — (1) `created_by`-Fremdschlüssel entfernt, (2) Testkonstellation für den Webhook-Ausfalltest angepasst (Ziel muss die SSRF-Prüfung zur Anlagezeit passieren, der Verbindungsfehler soll erst beim Senden entstehen), (3) doppelter Dateiinhalt in einem Testfall löste die vorhandene Dublettenerkennung (DOC-02) aus, mit unterschiedlichem Inhalt behoben. Kein Zielkonflikt, keine Rückfrage nötig.
|
||||
|
||||
## Offen / Grenzen
|
||||
|
||||
- E-Mail als Aktion: im DMS existiert kein Versandweg, laut Ticket ausdrücklich nicht Teil dieser Kachel.
|
||||
- Ordner verschieben als Aktion, Templating der Webhook-Nutzlast: nicht gebaut (Ticket-Scope).
|
||||
- Oberfläche zur Trigger-Verwaltung: nicht Teil dieser Kachel (UX-11-Familie).
|
||||
- Die `dms.trigger.read`/`dms.trigger.write`-Rechte sind in der DMS-eigenen OpenAPI-Tabelle beschrieben; ein Katalogeintrag in den zentralen Core-`policy_rules` (wie bei RBAC-09/UX-11) fehlt noch — Folgeticket im Core-Board, analog zum bereits dokumentierten Muster bei DOC-22 (`dms.sharelink.*`).
|
||||
- Kein neuer End-zu-Ende-Test über `authz.Guard` für den Admin-Fall (siehe Tabelle oben); die Feststellung ist strukturell begründet, nicht per Test bewiesen.
|
||||
@@ -254,3 +254,59 @@ Pruefungen auf 204 (Head nach Merge `6141fc9`, frischer Klon, seriell ueber die
|
||||
Skips (unveraendert gegenueber dem Lauf zu PR #97): `TestAppendOnly_AppRoleHasNoUpdateOrDeletePrivilege` (`dms_dev` ohne `CREATEROLE`), `TestKoSIT_LiveOutcomeMatchesExpectation` (`TEST_KOSIT_EXPECT` fehlt), 2x clamd, 2x soffice, `TestSign_UsesConfiguredTSA` (keine TSA-Env), 3x S3 (`TEST_S3_*` fehlt), die zwei Latenztests ohne `TEST_PERF=1` im Paketlauf (separat bestanden).
|
||||
|
||||
Hinweis zum Wrapper: `RUNHOST_*`-Argumente mit `|` im `-run`-Regex werden von `runhost.sh` durch die Shell zerlegt; Tests einzeln starten.
|
||||
|
||||
## Nachtrag: #111 (FDN-22, Verschluesselung der Textkopien) und #112 (FDN-23, Integritaetsjob)
|
||||
|
||||
Eingemergt in der Reihenfolge #111, dann #112. Beide Merges liefen automatisch durch (`git merge --no-ff`, "ort"-Strategie), kein manueller Konflikteingriff noetig. Der erwartete Konflikt in `dms/internal/crypto/service.go` (beide PRs aendern die Fehlerumhuellung um `ErrKEKUnavailable`) wurde vom Merge automatisch korrekt zusammengefuehrt: beide `fmt.Errorf("%w: %w", ErrKEKUnavailable, err)`-Stellen sind erhalten.
|
||||
|
||||
- Migrationen: 0031 (`text_encryption`, FDN-22) und 0032 (`integrity_checks`, FDN-23) sind eindeutig, direkt aufeinanderfolgend nach 0030.
|
||||
- Head nach Merge: `090617f`.
|
||||
|
||||
Pruefungen auf 204 (frischer Klon, seriell ueber die Sperre in `runhost.sh`):
|
||||
|
||||
| Pruefung | Ergebnis |
|
||||
|---|---|
|
||||
| `go vet ./...` | sauber |
|
||||
| `go test -count=1 ./...` (1 Lauf) | alle Pakete `ok`, 0 FAIL (u.a. `internal/textcrypt`, `internal/integrity`, `internal/ocr` mit den neuen Verschluesselungstests) |
|
||||
| `gofmt -l` (geaenderte Verzeichnisse: textcrypt, integrity, crypto, workerruntime, ocr, plaintext, cmd/textcrypt-migrate, cmd/worker) | leer |
|
||||
| `check-port-defaults.sh`, `check-route-prefixes.sh` | ok |
|
||||
|
||||
Kein Aufraeumen auf 204 noetig (Wegwerf-DB `dms_dev`, keine eigenen Schemas angelegt). Ein von einem anderen, parallelen Lauf verwaistes Klonverzeichnis (`/root/nexarch-code-iso-*`) wurde bei diesem Merge nicht angefasst.
|
||||
|
||||
## Nachtrag: #119 (DOC-22, externe Freigabelinks) und #120 (SRC-14, PDF/A-Validierung mit veraPDF)
|
||||
|
||||
Eingemergt in der Reihenfolge #119, dann #120. Beide Merges liefen automatisch durch (`git merge --no-ff`, "ort"-Strategie), keine Dateikonflikte.
|
||||
|
||||
- Migrationskollision: Beide PRs vergaben unabhaengig voneinander `0033` (DOC-22: `0033_share_links`, SRC-14: `0033_pdfa_conformance`) — kein Dateikonflikt, weil unterschiedliche Dateinamen, aber doppelte Nummer. Geloest durch Umbenennen der SRC-14-Migration auf `0034_pdfa_conformance` (up/down), da diese Migration noch nicht in einem produktiven Stand angewendet wurde und das Migrationssystem ueber den Dateinamen trackt (kein Code verweist auf den Dateinamen, `grep` ohne Treffer). `0033_share_links` (DOC-22) blieb unveraendert.
|
||||
- Head nach Merge: `040e355`.
|
||||
|
||||
Pruefungen auf 204 (frischer Klon des Zwischenbranches, seriell ueber die Sperre in `runhost.sh`):
|
||||
|
||||
| Pruefung | Ergebnis |
|
||||
|---|---|
|
||||
| `go vet ./...` | sauber |
|
||||
| `go test -count=1 ./...` (2 Laeufe) | alle Pakete `ok`, 0 FAIL, 0 panic (u.a. `internal/sharelink`, `internal/sharelinkapi`, `internal/pdfaconform` neu) |
|
||||
| `gofmt -l .` | leer |
|
||||
| `check-port-defaults.sh`, `check-route-prefixes.sh` | ok |
|
||||
| Migrationen 0001-0034 frisch | implizit bei jedem Testlauf ueber `testenv`/`testdb` (u.a. in `sharelinkapi`, `pdfaconform`, `migrate`) fehlerfrei durchlaufen; kein eigenstaendiges `migrations-fresh-check.sh` im DMS-Teilrepo vorhanden |
|
||||
|
||||
QA-Durchgang: 0 Iterationen noetig, beide Merges und die Umbenennung liefen im ersten Anlauf sauber durch. Kein Aufraeumen auf 204 noetig (nur der Wegwerf-DB `dms_dev` genutzt, keine eigenen Schemas/Ressourcen angelegt).
|
||||
|
||||
## Nachtrag: #127 (INT-20, Trigger-/Automatisierungssystem nach Upload)
|
||||
|
||||
Eingemergt via `git merge --no-ff`, keine Dateikonflikte. Migration `0035_triggers` ist eindeutig (kein Konflikt mit den vorhandenen Nummern 0001-0034; nur die seit langem bekannte Dreifachvergabe von `0016` bleibt unveraendert bestehen).
|
||||
|
||||
Neue Pakete `internal/trigger` und `internal/triggerapi` (Ereignisse `document.uploaded`, `document.ocr_completed`, `document.einvoice_checked`; Aktionen Webhook mit SSRF-Schutz/HMAC-Signatur, Tag setzen, Dokumenttyp setzen).
|
||||
|
||||
Head nach Merge: `8fb4aef`.
|
||||
|
||||
Pruefungen auf 204 (frischer Klon des Zwischenbranches, seriell ueber die Sperre in `runhost.sh`):
|
||||
|
||||
| Pruefung | Ergebnis |
|
||||
|---|---|
|
||||
| `go vet ./...` | sauber |
|
||||
| `go test -count=1 ./...` | alle Pakete `ok`, 0 FAIL (u.a. `internal/trigger`, `internal/triggerapi` neu, 5,2 s bzw. 2,1 s) |
|
||||
| `gofmt -l dms/` | leer |
|
||||
| `check-port-defaults.sh`, `check-route-prefixes.sh` | ok |
|
||||
|
||||
QA-Durchgang: 0 Iterationen noetig, Merge und Tests liefen im ersten Anlauf sauber durch.
|
||||
@@ -0,0 +1,170 @@
|
||||
# SRC-14 – Prüfprotokoll: PDF/A-Validierung mit veraPDF
|
||||
|
||||
Geprüft auf Entwicklungshost 192.168.1.204 gegen den echten veraPDF-Sidecar
|
||||
(`nexarch-dms-verapdf.service`, `127.0.0.1:8612`) und eine Wegwerf-Datenbank
|
||||
(`dms_dev`). `archivdms` (Dienst, Daten, Postgres-Cluster auf 5432,
|
||||
Manticore-Tabellen `documents_tenant_*`) blieb unberührt.
|
||||
|
||||
## Installation (Entscheidung erforderlich, vom Betreiber freigegeben)
|
||||
|
||||
Freigabe: veraPDF auf Host 204 als Tarball/Installation nach dem Muster des
|
||||
KoSIT-Sidecars aus MET-07 einrichten, Lizenz und Version im Protokoll belegen.
|
||||
|
||||
- **veraPDF Greenfield 1.30.2** (gebaut 2026-06-03 10:55 UTC), CLI-Paket
|
||||
(ohne GUI, ohne Dokumentation, ohne Beispiel-Plugins), installiert unter
|
||||
`/opt/nexarch-verapdf`.
|
||||
- Herkunft: `https://software.verapdf.org/releases/verapdf-installer.zip`.
|
||||
- Prüfsumme SHA-256 des Installers:
|
||||
`6cc6341cb1af644044054b81f00a6590a7918abb18f762243de115258bcad838`.
|
||||
- Zusätzlich per GPG-Signatur (`.asc`) gegen den auf `keyserver.ubuntu.com`
|
||||
veröffentlichten Schlüssel `13DD102B4DD69354D12DE5A83184863278B17FE7`
|
||||
(Carl Wilson, veraPDF-Techlead) geprüft: `Good signature`.
|
||||
- Lizenz: GNU GPLv3+ oder Mozilla Public License v2+ (Dual-Lizenz laut
|
||||
Installer-Begrüßungstext), zur Wahl des Nutzers.
|
||||
- **Temurin JRE 21.0.12.1**, wiederverwendet aus `/opt/nexarch-kosit/jre`
|
||||
(identische, bereits in MET-07 geprüfte Quelle und Prüfsumme; nur lesend
|
||||
genutzt, keine Schreibrechte für den neuen Dienstbenutzer).
|
||||
- Eigener unprivilegierter Systembenutzer `nexarch-verapdf`, eigener Dienst
|
||||
`nexarch-dms-verapdf` nur auf `127.0.0.1:8612` (DMS-Portbereich, Eintrag in
|
||||
`deploy/ports.tsv`; 8610 = KoSIT, 8611 = FDN-19-Smoke-Stellvertreter, beide
|
||||
belegt).
|
||||
- Rückbau dokumentiert und reproduzierbar in `deploy/verapdf/README.md`.
|
||||
|
||||
veraPDF hat keinen dokumentierten Daemon-Modus. Der Sidecar
|
||||
(`dms/cmd/verapdf-sidecar`, selbst geschrieben, nur Go-Standardbibliothek)
|
||||
ist deshalb ein dünner HTTP-Wrapper, der je Anfrage einen eigenen,
|
||||
kurzlebigen `verapdf`-Prozess startet (< 2 s in den Prüfungen unten). Das
|
||||
ist das vom Hersteller unterstützte Betriebsmodell.
|
||||
|
||||
## Umsetzung
|
||||
|
||||
- `dms/internal/pdfaconform`: Client (`Validator.Validate`) und Prozessor
|
||||
(`Processor.Validate`, Jobtyp `pdfa.validate`), Muster identisch zu
|
||||
`internal/einvoice` (KoSIT-Anbindung aus MET-07): Ergebnis ist eine
|
||||
Verlaufszeile je Prüfung (`revision_pdfa_conformance`, Migration `0033`),
|
||||
unveränderte Wiederholungen hängen nichts an, jede echte Änderung erzeugt
|
||||
einen Audit-Eintrag `pdfa_validated`.
|
||||
- `internal/ocr.Processor.OnPDFACreated`: neuer Hook, der nach erfolgreichem
|
||||
Ablegen eines PDF/A-Derivats (SRC-06) die Prüfung einreiht
|
||||
(`pdfaconform.Enqueue`, idempotent über `IdempotencyKey`).
|
||||
`internal/workerruntime` setzt den Hook automatisch, sobald PDF/A-Jobs
|
||||
registriert sind, und registriert den `pdfa.validate`-Handler immer
|
||||
(ohne konfigurierten Validator liefert er `not_checked`).
|
||||
- Env: `NEXARCH_DMS_WORKER_VERAPDF_ADDR` (leer = aus), optional
|
||||
`..._FLAVOUR` (Standard `2b`) und `..._TIMEOUT_SECONDS`.
|
||||
|
||||
## Akzeptanzkriterien
|
||||
|
||||
**AK1 – Ergebnis je Derivat mit Status, Profil, Version, Zeitpunkt, Hash,
|
||||
Fehlerliste; nicht konformes Derivat bleibt erhalten.**
|
||||
`TestValidate_ResultCarriesVersionHashAndAudit_NonCompliantStays`: ein
|
||||
absichtlich nicht konformes Ergebnis wird mit Profil (`PDF/A-2b validation
|
||||
profile`), Werkzeugversion (`1.30.2`), Berichts-Hash und einer Fehlerliste
|
||||
(Klausel `6.1.3`) gespeichert, ein Audit-Eintrag entsteht, und
|
||||
`revision_pdfa` (das Derivat selbst) bleibt im Zustand `ready` unverändert –
|
||||
bestanden.
|
||||
`TestValidate_UnchangedResultDoesNotDuplicate`: zwei gleiche Läufe hängen nur
|
||||
eine Zeile und einen Audit-Eintrag an – bestanden.
|
||||
|
||||
**AK2 – Werkzeug nicht erreichbar, Zeitüberschreitung, Größenlimit, HTML-
|
||||
Antwort oder unbekanntes Antwortformat ergeben `not_checked` mit Grund; die
|
||||
Erzeugung des Derivats bleibt unbeeinflusst.**
|
||||
`TestValidate_StatusMapping` (Unit, gestellte Antworten): nicht erreichbar,
|
||||
502 (Sidecar-Fehler), 504 (Zeitüberschreitung), 413 (zu groß), kaputtes JSON,
|
||||
leerer Körper, leere Job-Liste, nicht parsbares PDF (`taskException`) – alle
|
||||
neun Fälle ergeben `not_checked`, kein Fall `compliant`.
|
||||
`TestValidate_WithoutValidatorIsNotChecked` und
|
||||
`TestValidate_UnreachableSidecarIsNotChecked` (mit echter DB):
|
||||
`revision_pdfa.status` bleibt `ready`, unverändert – bestanden.
|
||||
`TestQA_ServiceFailureIsNeverCompliant` (echter Sidecar): falsche Adresse und
|
||||
1-Nanosekunden-Zeitlimit ergeben `not_checked`; ein anschließender Lauf ohne
|
||||
Neustart der Anwendung liefert wieder `compliant` – bestanden (Prüfung 2).
|
||||
|
||||
**AK3 – Anbindung optional per Umgebungsvariable, Dienst nur Loopback/
|
||||
internes Netz, Einrichtung unter `deploy/` beschrieben und rückbaubar.**
|
||||
`TestNewValidator_EmptyAddrDisables`: leere Adresse ⇒ `nil`, keine Prüfung
|
||||
angestoßen. Betrieb: `ss -ltn` zeigt `127.0.0.1:8612` (kein `0.0.0.0`); ein
|
||||
Zugriff über die LAN-Adresse des Hosts (`192.168.1.204:8612`) schlägt fehl
|
||||
(Verbindungsaufbau ohne Antwort) – bestanden (Prüfung 3, siehe unten).
|
||||
`deploy/verapdf/README.md` beschreibt Installation und Rückbau; der Rückbau
|
||||
selbst wurde für diesen Prüflauf nicht ausgeführt, weil der Dienst für
|
||||
Folgeprüfungen (QA-15 u. a.) auf 204 bestehen bleiben soll (analog zum
|
||||
KoSIT-Sidecar).
|
||||
|
||||
## Prüfungen vor der Abnahme
|
||||
|
||||
**1. Referenzsatz aus synthetischen gültigen PDF/A-2b und mutierten
|
||||
ungültigen PDFs liefert je den erwarteten Status.**
|
||||
`TestQA_ReferenceSetGetsExpectedStatus` gegen den echten Sidecar:
|
||||
- Das von `internal/pdfa.Finalize` selbst erzeugte Derivat ist `compliant`
|
||||
(144 von 144 Regeln bestanden laut veraPDF-Bericht).
|
||||
- Ein PDF ohne PDF/A-Merkmale (kein OutputIntent, keine XMP-Metadaten) ist
|
||||
`non_compliant` mit mindestens einer Regelverletzung.
|
||||
- Ein mutiertes Derivat mit zerstörter xref-Tabelle ist `not_checked`
|
||||
(veraPDF kann die Datei nicht parsen).
|
||||
Ergebnis: bestanden, alle drei Fälle.
|
||||
|
||||
**2. Ausfall und Blockade des Dienstes: nie `compliant`, Wiederanlauf ohne
|
||||
Neustart der Anwendung.**
|
||||
`TestQA_ServiceFailureIsNeverCompliant`: falsche Adresse und
|
||||
Zeitüberschreitung ergeben `not_checked`; derselbe, unveränderte
|
||||
`Validator` liefert im nächsten Aufruf ohne Neustart wieder `compliant`.
|
||||
Ergebnis: bestanden.
|
||||
|
||||
**3. Der Dienst ist von außen nicht erreichbar; die Version des Werkzeugs
|
||||
steht im Ergebnis.**
|
||||
`ss -ltn` auf 204: `LISTEN 127.0.0.1:8612`, kein `0.0.0.0`-Bind. Ein Abruf
|
||||
über die LAN-Adresse (`192.168.1.204:8612/healthz`) liefert keine Antwort
|
||||
(Zeitüberschreitung). Die Werkzeugversion (`1.30.2`) steht in jedem
|
||||
`Outcome.ValidatorVersion` und wird in `revision_pdfa_conformance`
|
||||
gespeichert (siehe AK1-Nachweis).
|
||||
Ergebnis: bestanden.
|
||||
|
||||
## Gesamtlauf
|
||||
|
||||
- `go build`, `go vet` und `gofmt -l` (nur die vier geänderten Nicht-Test-
|
||||
Dateien betroffen; nach Korrektur der Importreihenfolge und
|
||||
Struct-Feld-Ausrichtung sauber) auf 204: sauber.
|
||||
- `go test ./internal/pdfaconform/... ./internal/ocr/...` (inkl. echtem
|
||||
Sidecar über `TEST_VERAPDF_ADDR`): alle Pakete `ok`.
|
||||
- Diff gegen `integration/dms-gesamtstand`: 12 Dateien, 965 Zeilen
|
||||
hinzugefügt, 1 Zeile geändert (kein bestehender Code umgebaut).
|
||||
|
||||
## QA und Entscheidungen
|
||||
|
||||
- **Betriebsform (Daemon vs. Einzelaufruf):** Einzelaufruf je Anfrage hinter
|
||||
einem selbst geschriebenen, dauerhaft laufenden HTTP-Wrapper. Begründung:
|
||||
veraPDF bietet keinen offiziellen Daemon-Modus (anders als der
|
||||
KoSIT-Validator); ein Prozessstart pro Prüfung ist der vom Hersteller
|
||||
vorgesehene Weg und in den Prüfungen deutlich unter 2 s geblieben.
|
||||
- **JRE-Wiederverwendung:** Die bereits für MET-07 geprüfte Temurin-JRE wird
|
||||
nur lesend wiederverwendet statt ein zweites Mal heruntergeladen –
|
||||
identische Prüfsumme, kein zusätzliches Risiko, weniger Plattenverbrauch.
|
||||
- **Nur CLI-Paket installiert:** GUI, Dokumentation und Beispiel-Plugins des
|
||||
Installers wurden abgewählt (kleinere Angriffsfläche, keine Kopien einer
|
||||
grafischen Oberfläche auf einem Server ohne Display).
|
||||
- **Port 8611 vermieden:** War bereits für den FDN-19-Smoke-Stellvertreter
|
||||
reserviert (`deploy/ports.tsv`); der Sidecar läuft stattdessen auf 8612.
|
||||
- **Fehlgeschlagene veraPDF-Läufe mit auswertbarem Bericht:** Der Sidecar
|
||||
akzeptiert einen Exit-Code ungleich 0, wenn `stdout` dennoch gültiges JSON
|
||||
enthält (beobachtetes Verhalten bei nicht parsbaren PDFs); der Bericht
|
||||
selbst entscheidet über den Status, nicht der Exit-Code.
|
||||
- 0 Iterationen bei Sicherheitsfilter-Blockaden oder Freigaben-
|
||||
Überschreitungen; alle Installationsschritte blieben innerhalb der
|
||||
erteilten Freigabe (nur Host 204, nur `/opt/nexarch-verapdf`).
|
||||
|
||||
## Nicht Bestandteil dieser Kachel (laut Ticket)
|
||||
|
||||
- Reparatur nicht konformer PDFs.
|
||||
- Prüfung von PDF/A-3-Anhängen und E-Rechnungen (bleibt bei MET-07).
|
||||
- Änderung der Erzeugung des Derivats (SRC-06 unverändert, nur der
|
||||
`OnPDFACreated`-Hook kam hinzu).
|
||||
|
||||
## Offen
|
||||
|
||||
- Kein API-Endpunkt für das Ergebnis (folgt mit INT-01/weiteren API-Tickets,
|
||||
analog zu den E-Rechnungs-Ergebnissen).
|
||||
- Die Installation auf Host 131 (Vorstufe Produktion) wurde nicht
|
||||
durchgeführt; das ist ein separater Freigabeschritt.
|
||||
- Kein Test mit einem PDF/A-3-Derivat oder anderen Flavours als `2b`, da
|
||||
SRC-06 ausschließlich PDF/A-2b erzeugt.
|
||||
@@ -27,28 +27,36 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/moduleauth"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/openapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/policyclient"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/sharelinkapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/tagapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trash"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trashapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trigger"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/triggerapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/upload"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/uploadapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/versioningapi"
|
||||
)
|
||||
|
||||
type Deps struct {
|
||||
Pool *pgxpool.Pool
|
||||
Storage *storage.Service
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
Upload *upload.Service
|
||||
DownloadSecret []byte
|
||||
DownloadTTL time.Duration
|
||||
TrashRetention time.Duration
|
||||
Auditors []string
|
||||
SearchEnqueuer trash.SearchIndexEnqueuer
|
||||
LegalHolds legalhold.Checker
|
||||
PathPrefix string
|
||||
Pool *pgxpool.Pool
|
||||
Storage *storage.Service
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
Upload *upload.Service
|
||||
DownloadSecret []byte
|
||||
DownloadTTL time.Duration
|
||||
ShareLinksEnabled bool
|
||||
ShareLinkDefaultTTL time.Duration
|
||||
ShareLinkMaxTTL time.Duration
|
||||
ShareLinkMaxAttempts int
|
||||
ShareLinkLockDuration time.Duration
|
||||
TrashRetention time.Duration
|
||||
Auditors []string
|
||||
SearchEnqueuer trash.SearchIndexEnqueuer
|
||||
LegalHolds legalhold.Checker
|
||||
PathPrefix string
|
||||
// MailStore wird gesetzt, wenn Postfaecher/Regeln/Klaerungskorb (INT-03/
|
||||
// INT-04, UX-11) ueber diese Instanz verwaltbar sein sollen.
|
||||
MailStore *mailimport.Store
|
||||
@@ -84,6 +92,11 @@ func Mount(api *http.ServeMux, d Deps) error {
|
||||
download.SetPathPrefix(d.PathPrefix)
|
||||
download.RegisterRoutes(api)
|
||||
|
||||
share := sharelinkapi.New(d.Pool, d.Storage, d.Crypto, d.TenantSlug, d.ShareLinksEnabled,
|
||||
d.ShareLinkDefaultTTL, d.ShareLinkMaxTTL, d.ShareLinkMaxAttempts, d.ShareLinkLockDuration)
|
||||
share.SetPathPrefix(d.PathPrefix)
|
||||
share.RegisterRoutes(api)
|
||||
|
||||
trashSrv := trashapi.NewServer(d.Pool, d.Storage, d.TrashRetention)
|
||||
if d.SearchEnqueuer != nil {
|
||||
trashSrv.SetSearchIndexEnqueuer(d.SearchEnqueuer)
|
||||
@@ -97,6 +110,7 @@ func Mount(api *http.ServeMux, d Deps) error {
|
||||
mailimportapi.NewServer(d.MailStore).RegisterRoutes(api)
|
||||
}
|
||||
einvoiceapi.NewServer(d.Pool, d.EinvoiceEnqueuer, &downloadapi.DocumentOwnerChecker{Pool: d.Pool}).RegisterRoutes(api)
|
||||
triggerapi.NewServer(d.Pool, trigger.CryptoVault{Crypto: d.Crypto}, d.TenantSlug).RegisterRoutes(api)
|
||||
|
||||
api.HandleFunc("GET /openapi.json", openapi.Handler().ServeHTTP)
|
||||
return nil
|
||||
|
||||
@@ -156,6 +156,8 @@ func needsGuard(method string, seg []string) bool {
|
||||
return len(seg) == 2 || (len(seg) == 1 && method == http.MethodPost)
|
||||
case "auto-rules":
|
||||
return true
|
||||
case "triggers":
|
||||
return true
|
||||
case "ocr", "search", "saved-searches":
|
||||
return true
|
||||
case "cmis":
|
||||
@@ -186,6 +188,11 @@ func evaluate(r *http.Request, c Checker, caller, method string, seg []string, w
|
||||
return c.IsAdmin(ctx, caller)
|
||||
case "auto-rules":
|
||||
return c.IsAdmin(ctx, caller)
|
||||
case "triggers":
|
||||
if method == http.MethodGet {
|
||||
return true, nil
|
||||
}
|
||||
return c.IsAdmin(ctx, caller)
|
||||
case "bulk-edit":
|
||||
var body struct {
|
||||
Actor string `json:"actor"`
|
||||
|
||||
@@ -2,10 +2,13 @@ package crypto
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
)
|
||||
|
||||
var ErrKEKUnavailable = errors.New("crypto: tenant-kek beziehen")
|
||||
|
||||
// Envelope ist das Ergebnis einer Seal-Operation: der Chiffretext-Stream
|
||||
// plus der mit dem Tenant-KEK verpackte DEK, der zusammen mit dem Objekt
|
||||
// persistiert werden muss (z.B. in file_revisions, FDN-02 — die konkrete
|
||||
@@ -41,7 +44,7 @@ func (s *Service) Seal(ctx context.Context, tenantSlug string, plaintext io.Read
|
||||
}
|
||||
kek, err := s.kek.TenantKEK(ctx, tenantSlug)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("crypto: tenant-kek beziehen: %w", err)
|
||||
return nil, fmt.Errorf("%w: %w", ErrKEKUnavailable, err)
|
||||
}
|
||||
wrappedDEK, err := WrapDEK(kek, dek)
|
||||
if err != nil {
|
||||
@@ -55,7 +58,7 @@ func (s *Service) Seal(ctx context.Context, tenantSlug string, plaintext io.Read
|
||||
func (s *Service) Open(ctx context.Context, tenantSlug string, wrappedDEK []byte, ciphertext io.Reader) (io.Reader, error) {
|
||||
kek, err := s.kek.TenantKEK(ctx, tenantSlug)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("crypto: tenant-kek beziehen: %w", err)
|
||||
return nil, fmt.Errorf("%w: %w", ErrKEKUnavailable, err)
|
||||
}
|
||||
dek, err := UnwrapDEK(kek, wrappedDEK)
|
||||
if err != nil {
|
||||
@@ -63,3 +66,31 @@ func (s *Service) Open(ctx context.Context, tenantSlug string, wrappedDEK []byte
|
||||
}
|
||||
return DecryptStream(dek, ciphertext)
|
||||
}
|
||||
|
||||
// NewDataKey erzeugt einen DEK und verpackt ihn mit dem aktuellen Tenant-KEK.
|
||||
// Der Klartext-DEK dient dem Aufrufer zum Verschluesseln im Speicher, der
|
||||
// verpackte DEK wird persistiert.
|
||||
func (s *Service) NewDataKey(ctx context.Context, tenantSlug string) (dek, wrapped []byte, err error) {
|
||||
dek, err = GenerateDEK()
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
kek, err := s.kek.TenantKEK(ctx, tenantSlug)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("crypto: tenant-kek beziehen: %w", err)
|
||||
}
|
||||
wrapped, err = WrapDEK(kek, dek)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return dek, wrapped, nil
|
||||
}
|
||||
|
||||
// UnwrapDataKey entpackt einen mit NewDataKey verpackten DEK.
|
||||
func (s *Service) UnwrapDataKey(ctx context.Context, tenantSlug string, wrapped []byte) ([]byte, error) {
|
||||
kek, err := s.kek.TenantKEK(ctx, tenantSlug)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("crypto: tenant-kek beziehen: %w", err)
|
||||
}
|
||||
return UnwrapDEK(kek, wrapped)
|
||||
}
|
||||
@@ -44,7 +44,15 @@ func (p *Processor) conform(ctx context.Context, documentID, revisionID string,
|
||||
default:
|
||||
out = p.Validator.Validate(ctx, xmlData)
|
||||
}
|
||||
return p.storeConformance(ctx, documentID, revisionID, out)
|
||||
if err := p.storeConformance(ctx, documentID, revisionID, out); err != nil {
|
||||
return err
|
||||
}
|
||||
if p.OnConformanceChecked != nil {
|
||||
if err := p.OnConformanceChecked(ctx, documentID, revisionID); err != nil {
|
||||
return fmt.Errorf("einvoice: nachbearbeitung nach konformitaetspruefung: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Processor) storeConformance(ctx context.Context, documentID, revisionID string, out Outcome) error {
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/metadatafield"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/textcrypt"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -90,6 +91,9 @@ type Processor struct {
|
||||
TenantSlug string
|
||||
// OnRead wird nach dem Ablegen der Werte gerufen (z. B. Suchindex aktualisieren).
|
||||
OnRead func(ctx context.Context, documentID, revisionID string) error
|
||||
// OnConformanceChecked wird nach jeder KoSIT-Konformitaetspruefung gerufen
|
||||
// (Erstpruefung und Neupruefung, INT-20-Trigger document.einvoice_checked).
|
||||
OnConformanceChecked func(ctx context.Context, documentID, revisionID string) error
|
||||
}
|
||||
|
||||
func (p *Processor) Handler(ctx context.Context, job *jobqueue.Job) error {
|
||||
@@ -149,18 +153,34 @@ func (p *Processor) Process(ctx context.Context, documentID, revisionID string)
|
||||
|
||||
func (p *Processor) recordUnreadable(ctx context.Context, documentID, revisionID, container string, cause error) error {
|
||||
findings, _ := json.Marshal([]Finding{{Code: "E000", Message: cause.Error()}})
|
||||
if _, err := p.Pool.Exec(ctx, upsertSQL, revisionID, string(FormatUnknown), container, "unreadable", findings, ""); err != nil {
|
||||
if _, err := p.Pool.Exec(ctx, upsertSQL, revisionID, string(FormatUnknown), container, "unreadable", findings, nil, nil); err != nil {
|
||||
return fmt.Errorf("einvoice: ergebnis speichern: %w", err)
|
||||
}
|
||||
return p.audit(ctx, documentID, revisionID, string(FormatUnknown), "unreadable", 1)
|
||||
}
|
||||
|
||||
const upsertSQL = `
|
||||
INSERT INTO revision_einvoice (revision_id, format, container, status, findings, search_text)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
INSERT INTO revision_einvoice (revision_id, format, container, status, findings, search_text, search_text_enc, wrapped_dek)
|
||||
VALUES ($1, $2, $3, $4, $5, '', $6, $7)
|
||||
ON CONFLICT (revision_id) DO UPDATE
|
||||
SET format = EXCLUDED.format, container = EXCLUDED.container, status = EXCLUDED.status,
|
||||
findings = EXCLUDED.findings, search_text = EXCLUDED.search_text, updated_at = now()`
|
||||
findings = EXCLUDED.findings, search_text = '', search_text_enc = EXCLUDED.search_text_enc,
|
||||
wrapped_dek = EXCLUDED.wrapped_dek, updated_at = now()`
|
||||
|
||||
// sealSearchText verschluesselt den Suchtext der Revision (leer bleibt leer).
|
||||
func (p *Processor) sealSearchText(ctx context.Context, revisionID, text string) (sealed, wrapped []byte, err error) {
|
||||
if text == "" {
|
||||
return nil, nil, nil
|
||||
}
|
||||
dek, wrapped, err := textcrypt.Vault{Crypto: p.Crypto, TenantSlug: p.TenantSlug}.NewKey(ctx)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("einvoice: %w", err)
|
||||
}
|
||||
if sealed, err = textcrypt.Seal(dek, textcrypt.AADInvoice(revisionID), text); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return sealed, wrapped, nil
|
||||
}
|
||||
|
||||
func (p *Processor) store(ctx context.Context, documentID, revisionID, container string, r *Result) error {
|
||||
findings := r.Findings
|
||||
@@ -186,7 +206,11 @@ func (p *Processor) store(ctx context.Context, documentID, revisionID, container
|
||||
if len(findings) > 0 {
|
||||
status = "invalid"
|
||||
}
|
||||
if _, err := p.Pool.Exec(ctx, upsertSQL, revisionID, string(r.Invoice.Format), container, status, encoded, r.SearchText()); err != nil {
|
||||
sealed, wrapped, err := p.sealSearchText(ctx, revisionID, r.SearchText())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := p.Pool.Exec(ctx, upsertSQL, revisionID, string(r.Invoice.Format), container, status, encoded, sealed, wrapped); err != nil {
|
||||
return fmt.Errorf("einvoice: ergebnis speichern: %w", err)
|
||||
}
|
||||
if err := p.audit(ctx, documentID, revisionID, string(r.Invoice.Format), status, len(findings)); err != nil {
|
||||
@@ -279,16 +303,32 @@ func ensureFields(ctx context.Context, pool *pgxpool.Pool) (map[string]metadataf
|
||||
}
|
||||
|
||||
// TextStore liefert den Volltext der aktuellen Revision einer E-Rechnung.
|
||||
type TextStore struct{ Pool *pgxpool.Pool }
|
||||
type TextStore struct {
|
||||
Pool *pgxpool.Pool
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
}
|
||||
|
||||
func (t TextStore) TextOf(ctx context.Context, documentID string) (string, error) {
|
||||
var text string
|
||||
var revisionID, text string
|
||||
var enc, wrapped []byte
|
||||
err := t.Pool.QueryRow(ctx, `
|
||||
SELECT e.search_text FROM documents d JOIN revision_einvoice e ON e.revision_id = d.current_revision_id
|
||||
SELECT e.revision_id::text, e.search_text, e.search_text_enc, e.wrapped_dek FROM documents d JOIN revision_einvoice e ON e.revision_id = d.current_revision_id
|
||||
WHERE d.id = $1
|
||||
`, documentID).Scan(&text)
|
||||
`, documentID).Scan(&revisionID, &text, &enc, &wrapped)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return "", nil
|
||||
}
|
||||
return text, err
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if enc == nil {
|
||||
return text, nil
|
||||
}
|
||||
v := textcrypt.Vault{Crypto: t.Crypto, TenantSlug: t.TenantSlug}
|
||||
dek, err := v.Key(ctx, wrapped)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return textcrypt.Open(dek, textcrypt.AADInvoice(revisionID), enc)
|
||||
}
|
||||
@@ -219,3 +219,28 @@ func TestEnqueuer_OnlyPDFAndXMLBecomeJobs(t *testing.T) {
|
||||
t.Errorf("wiederholtes einreihen ist nicht idempotent: %d jobs", jobs())
|
||||
}
|
||||
}
|
||||
|
||||
// FDN-22: der Suchtext der E-Rechnung liegt nur verschluesselt in der Datenbank und wird beim Lesen entschluesselt.
|
||||
func TestSearchTextIsStoredEncrypted(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
docID, revID := uploadAny(t, e, "rechnung.xml", "text/xml", testcorpus.GenerateUBLInvoice(testcorpus.DefaultInvoice()), nil)
|
||||
if err := processor(e).Process(ctx, docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var plain string
|
||||
var enc, wrapped []byte
|
||||
if err := e.Pool.QueryRow(ctx, `SELECT search_text, search_text_enc, wrapped_dek FROM revision_einvoice WHERE revision_id = $1`, revID).Scan(&plain, &enc, &wrapped); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plain != "" || len(enc) == 0 || len(wrapped) == 0 || bytes.Contains(enc, []byte("RE-2026-0042")) || bytes.Contains(enc, []byte("Musterbau")) {
|
||||
t.Fatalf("suchtext nicht verschluesselt: plain=%q enc=%d wrapped=%d", plain, len(enc), len(wrapped))
|
||||
}
|
||||
text, err := (TextStore{Pool: e.Pool, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug}).TextOf(ctx, docID)
|
||||
if err != nil || !bytes.Contains([]byte(text), []byte("RE-2026-0042")) {
|
||||
t.Fatalf("textstore: %q %v", text, err)
|
||||
}
|
||||
if blind, err := (TextStore{Pool: e.Pool}).TextOf(ctx, docID); err == nil || blind != "" {
|
||||
t.Fatalf("ohne schluesseldienst: %q %v", blind, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
package integrity
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
)
|
||||
|
||||
const (
|
||||
StatusOK = "ok"
|
||||
StatusMismatch = "mismatch"
|
||||
StatusMissing = "missing"
|
||||
StatusUnreadable = "unreadable"
|
||||
StatusError = "error"
|
||||
|
||||
ModeSample = "sample"
|
||||
ModeFull = "full"
|
||||
|
||||
JobType = "integrity.verify"
|
||||
|
||||
AuditAction = "integrity_violation"
|
||||
auditActor = "system"
|
||||
|
||||
// Frisch angelegte Revisionen werden erst nach dieser Frist geprueft,
|
||||
// damit ein noch nicht abgeschlossener Upload kein "missing" ergibt.
|
||||
settleAfter = 2 * time.Minute
|
||||
|
||||
// Je Schema, damit Testschemas derselben Datenbank sich nicht sperren; produktiv gilt eine Pruefung je Mandanten-Datenbank.
|
||||
lockExpr = `hashtextextended('nexarch.integrity:' || current_schema(), 0)`
|
||||
)
|
||||
|
||||
var ErrBusy = errors.New("integrity: es laeuft bereits eine pruefung")
|
||||
|
||||
type Config struct {
|
||||
Mode string
|
||||
Batch int
|
||||
MaxDuration time.Duration
|
||||
Pause time.Duration
|
||||
RevisionTimeout time.Duration
|
||||
MaxObjectBytes int64
|
||||
}
|
||||
|
||||
type Summary struct {
|
||||
RunID string
|
||||
Status string
|
||||
Checked int
|
||||
OK int
|
||||
Mismatch int
|
||||
Missing int
|
||||
Unreadable int
|
||||
Error int
|
||||
}
|
||||
|
||||
func (s Summary) Violations() int { return s.Mismatch + s.Missing + s.Unreadable }
|
||||
|
||||
type Verifier struct {
|
||||
Pool *pgxpool.Pool
|
||||
Storage *storage.Service
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
Config Config
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
type revision struct {
|
||||
ID string
|
||||
StorageKey string
|
||||
Checksum string
|
||||
SizeBytes int64
|
||||
WrappedDEK []byte
|
||||
PrevStatus *string
|
||||
}
|
||||
|
||||
func (v *Verifier) now() time.Time {
|
||||
if v.Now != nil {
|
||||
return v.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (v *Verifier) Run(ctx context.Context) (Summary, error) {
|
||||
conn, err := v.Pool.Acquire(ctx)
|
||||
if err != nil {
|
||||
return Summary{}, fmt.Errorf("integrity: verbindung: %w", err)
|
||||
}
|
||||
defer conn.Release()
|
||||
var locked bool
|
||||
if err := conn.QueryRow(ctx, `SELECT pg_try_advisory_lock(`+lockExpr+`)`).Scan(&locked); err != nil {
|
||||
return Summary{}, fmt.Errorf("integrity: sperre: %w", err)
|
||||
}
|
||||
if !locked {
|
||||
return Summary{}, ErrBusy
|
||||
}
|
||||
defer func() { _, _ = conn.Exec(context.Background(), `SELECT pg_advisory_unlock(`+lockExpr+`)`) }()
|
||||
|
||||
mode := v.Config.Mode
|
||||
if mode != ModeFull {
|
||||
mode = ModeSample
|
||||
}
|
||||
sum := Summary{}
|
||||
if err := v.Pool.QueryRow(ctx, `INSERT INTO integrity_runs (mode) VALUES ($1) RETURNING id`, mode).Scan(&sum.RunID); err != nil {
|
||||
return sum, fmt.Errorf("integrity: lauf anlegen: %w", err)
|
||||
}
|
||||
|
||||
var since *time.Time
|
||||
if mode == ModeFull {
|
||||
var c time.Time
|
||||
if err := v.Pool.QueryRow(ctx, `
|
||||
SELECT COALESCE(min(started_at), now()) FROM integrity_runs
|
||||
WHERE mode = 'full' AND started_at > COALESCE(
|
||||
(SELECT max(started_at) FROM integrity_runs WHERE mode = 'full' AND status = 'completed' AND id <> $1::uuid),
|
||||
'-infinity')
|
||||
`, sum.RunID).Scan(&c); err != nil {
|
||||
v.finish(context.Background(), sum, "failed", err.Error())
|
||||
return sum, fmt.Errorf("integrity: zyklusbeginn: %w", err)
|
||||
}
|
||||
since = &c
|
||||
}
|
||||
|
||||
started := v.now()
|
||||
deadline := time.Time{}
|
||||
if v.Config.MaxDuration > 0 {
|
||||
deadline = started.Add(v.Config.MaxDuration)
|
||||
}
|
||||
limit := v.Config.Batch
|
||||
if limit <= 0 {
|
||||
limit = 200
|
||||
}
|
||||
finalStatus := "completed"
|
||||
note := ""
|
||||
|
||||
loop:
|
||||
for {
|
||||
want := limit
|
||||
if mode == ModeSample {
|
||||
want = limit - sum.Checked
|
||||
if want <= 0 {
|
||||
break
|
||||
}
|
||||
} else if want > 500 {
|
||||
want = 500
|
||||
}
|
||||
batch, err := v.candidates(ctx, sum.RunID, want, since)
|
||||
if err != nil {
|
||||
finalStatus, note = "failed", err.Error()
|
||||
v.finish(context.Background(), sum, finalStatus, note)
|
||||
return sum, err
|
||||
}
|
||||
if len(batch) == 0 {
|
||||
break
|
||||
}
|
||||
for _, r := range batch {
|
||||
if ctx.Err() != nil {
|
||||
finalStatus, note = "partial", "abgebrochen"
|
||||
break loop
|
||||
}
|
||||
if !deadline.IsZero() && v.now().After(deadline) {
|
||||
finalStatus, note = "partial", "zeitlimit erreicht"
|
||||
break loop
|
||||
}
|
||||
status, detail := v.check(ctx, r)
|
||||
if err := v.record(ctx, sum.RunID, r, status, detail); err != nil {
|
||||
finalStatus, note = "failed", err.Error()
|
||||
v.finish(context.Background(), sum, finalStatus, note)
|
||||
return sum, err
|
||||
}
|
||||
sum.Checked++
|
||||
switch status {
|
||||
case StatusOK:
|
||||
sum.OK++
|
||||
case StatusMismatch:
|
||||
sum.Mismatch++
|
||||
case StatusMissing:
|
||||
sum.Missing++
|
||||
case StatusUnreadable:
|
||||
sum.Unreadable++
|
||||
default:
|
||||
sum.Error++
|
||||
}
|
||||
if v.Config.Pause > 0 {
|
||||
select {
|
||||
case <-time.After(v.Config.Pause):
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sum.Status = finalStatus
|
||||
v.finish(context.Background(), sum, finalStatus, note)
|
||||
log.Printf("integrity: lauf %s (%s): %d geprueft, %d ok, %d abweichungen, %d fehler", sum.RunID, finalStatus, sum.Checked, sum.OK, sum.Violations(), sum.Error)
|
||||
return sum, nil
|
||||
}
|
||||
|
||||
func (v *Verifier) candidates(ctx context.Context, runID string, n int, since *time.Time) ([]revision, error) {
|
||||
rows, err := v.Pool.Query(ctx, `
|
||||
SELECT r.id, r.storage_key, r.checksum_sha256, r.size_bytes, r.wrapped_dek, ir.status
|
||||
FROM file_revisions r
|
||||
JOIN documents d ON d.id = r.document_id
|
||||
LEFT JOIN integrity_results ir ON ir.revision_id = r.id
|
||||
WHERE d.deleted_at IS NULL AND d.trashed_at IS NULL
|
||||
AND r.created_at < $3
|
||||
AND (ir.revision_id IS NULL OR CASE WHEN $4::timestamptz IS NULL THEN ir.run_id IS DISTINCT FROM $1::uuid ELSE ir.checked_at < $4::timestamptz END)
|
||||
ORDER BY ir.checked_at NULLS FIRST, r.id
|
||||
LIMIT $2
|
||||
`, runID, n, v.now().Add(-settleAfter), since)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("integrity: kandidaten: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []revision
|
||||
for rows.Next() {
|
||||
var r revision
|
||||
if err := rows.Scan(&r.ID, &r.StorageKey, &r.Checksum, &r.SizeBytes, &r.WrappedDEK, &r.PrevStatus); err != nil {
|
||||
return nil, fmt.Errorf("integrity: kandidaten: %w", err)
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (v *Verifier) check(ctx context.Context, r revision) (string, string) {
|
||||
if v.Config.RevisionTimeout > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, v.Config.RevisionTimeout)
|
||||
defer cancel()
|
||||
}
|
||||
if max := v.Config.MaxObjectBytes; max > 0 && r.SizeBytes > max {
|
||||
return StatusError, fmt.Sprintf("uebersprungen: objekt %d bytes groesser als limit %d", r.SizeBytes, max)
|
||||
}
|
||||
rc, err := v.Storage.Get(ctx, r.StorageKey)
|
||||
if errors.Is(err, storage.ErrNotFound) {
|
||||
return StatusMissing, "objekt nicht im speicher vorhanden"
|
||||
}
|
||||
if err != nil {
|
||||
return StatusError, "speicher: " + shorten(err)
|
||||
}
|
||||
defer func() { _ = rc.Close() }()
|
||||
if len(r.WrappedDEK) == 0 {
|
||||
return StatusUnreadable, "kein verpackter schluessel zur revision gespeichert"
|
||||
}
|
||||
plain, err := v.Crypto.Open(ctx, v.TenantSlug, r.WrappedDEK, rc)
|
||||
switch {
|
||||
case errors.Is(err, dmscrypto.ErrKEKUnavailable):
|
||||
return StatusError, "schluesseldienst: " + shorten(err)
|
||||
case errors.Is(err, dmscrypto.ErrDecryptFailed):
|
||||
return StatusUnreadable, "entschluesselung fehlgeschlagen (falscher schluessel oder beschaedigtes objekt)"
|
||||
case err != nil:
|
||||
return StatusError, "lesen: " + shorten(err)
|
||||
}
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, plain)
|
||||
if err != nil {
|
||||
return StatusError, "lesen: " + shorten(err)
|
||||
}
|
||||
got := hex.EncodeToString(h.Sum(nil))
|
||||
var problems []string
|
||||
if n != r.SizeBytes {
|
||||
problems = append(problems, fmt.Sprintf("groesse %d statt %d", n, r.SizeBytes))
|
||||
}
|
||||
if !strings.EqualFold(got, r.Checksum) {
|
||||
problems = append(problems, "sha256 weicht ab")
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
return StatusMismatch, strings.Join(problems, ", ")
|
||||
}
|
||||
return StatusOK, ""
|
||||
}
|
||||
|
||||
func (v *Verifier) record(ctx context.Context, runID string, r revision, status, detail string) error {
|
||||
violation := status == StatusMismatch || status == StatusMissing || status == StatusUnreadable
|
||||
// Audit zuerst: scheitert es, bleibt der Statuswechsel ungespeichert und der naechste Lauf meldet ihn erneut
|
||||
// (mindestens einmal statt nie).
|
||||
if violation && (r.PrevStatus == nil || *r.PrevStatus != status) && v.stillExists(ctx, r.ID) {
|
||||
if err := audittrail.Record(ctx, v.Pool, auditActor, AuditAction, "file_revision", r.ID, map[string]any{"status": status, "run_id": runID}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
var lastOK any
|
||||
if status == StatusOK {
|
||||
lastOK = v.now()
|
||||
}
|
||||
if _, err := v.Pool.Exec(ctx, `
|
||||
INSERT INTO integrity_results (revision_id, run_id, status, detail, checked_at, last_ok_at)
|
||||
SELECT $1, $2, $3, $4, now(), $5::timestamptz
|
||||
WHERE EXISTS (SELECT 1 FROM file_revisions WHERE id = $1)
|
||||
ON CONFLICT (revision_id) DO UPDATE SET
|
||||
run_id = EXCLUDED.run_id, status = EXCLUDED.status, detail = EXCLUDED.detail, checked_at = now(),
|
||||
last_ok_at = COALESCE(EXCLUDED.last_ok_at, integrity_results.last_ok_at)
|
||||
`, r.ID, runID, status, detail, lastOK); err != nil {
|
||||
return fmt.Errorf("integrity: ergebnis speichern: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (v *Verifier) stillExists(ctx context.Context, revisionID string) bool {
|
||||
var live bool
|
||||
err := v.Pool.QueryRow(ctx, `
|
||||
SELECT true FROM file_revisions r JOIN documents d ON d.id = r.document_id
|
||||
WHERE r.id = $1 AND d.deleted_at IS NULL AND d.trashed_at IS NULL
|
||||
`, revisionID).Scan(&live)
|
||||
return err == nil && live
|
||||
}
|
||||
|
||||
func (v *Verifier) finish(ctx context.Context, s Summary, status, note string) {
|
||||
_, err := v.Pool.Exec(ctx, `
|
||||
UPDATE integrity_runs SET finished_at = now(), status = $2, checked = $3, ok = $4, mismatch = $5,
|
||||
missing = $6, unreadable = $7, error = $8, note = $9 WHERE id = $1
|
||||
`, s.RunID, status, s.Checked, s.OK, s.Mismatch, s.Missing, s.Unreadable, s.Error, note)
|
||||
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
|
||||
log.Printf("integrity: lauf %s nicht abgeschlossen: %v", s.RunID, err)
|
||||
}
|
||||
}
|
||||
|
||||
func shorten(err error) string {
|
||||
msg := err.Error()
|
||||
if len(msg) > 200 {
|
||||
msg = msg[:200]
|
||||
}
|
||||
return msg
|
||||
}
|
||||
@@ -0,0 +1,417 @@
|
||||
package integrity
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
type kek struct {
|
||||
key []byte
|
||||
err error
|
||||
}
|
||||
|
||||
func (k kek) TenantKEK(context.Context, string) ([]byte, error) { return k.key, k.err }
|
||||
|
||||
func newVerifier(e *testenv.Env, cfg Config) *Verifier {
|
||||
if cfg.Mode == "" {
|
||||
cfg.Mode = ModeFull
|
||||
}
|
||||
return &Verifier{
|
||||
Pool: e.Pool, Storage: e.Storage, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug, Config: cfg,
|
||||
Now: func() time.Time { return time.Now().Add(time.Hour) },
|
||||
}
|
||||
}
|
||||
|
||||
func doc(t *testing.T, e *testenv.Env, name string) (docID, revID string) {
|
||||
t.Helper()
|
||||
return e.UploadDoc(t, name, "text/plain", []byte("inhalt "+name))
|
||||
}
|
||||
|
||||
func statusOf(t *testing.T, e *testenv.Env, revID string) string {
|
||||
t.Helper()
|
||||
var s string
|
||||
if err := e.Pool.QueryRow(context.Background(), `SELECT status FROM integrity_results WHERE revision_id = $1`, revID).Scan(&s); err != nil {
|
||||
return ""
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func storageKey(t *testing.T, e *testenv.Env, revID string) (string, int64) {
|
||||
t.Helper()
|
||||
var key string
|
||||
var size int64
|
||||
if err := e.Pool.QueryRow(context.Background(), `SELECT storage_key, size_bytes FROM file_revisions WHERE id = $1`, revID).Scan(&key, &size); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return key, size
|
||||
}
|
||||
|
||||
func auditCount(t *testing.T, e *testenv.Env, revID string) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
if err := e.Pool.QueryRow(context.Background(), `SELECT count(*) FROM audit_log WHERE action = $1 AND object_reference = $2`, AuditAction, revID).Scan(&n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// AK1 / Pruefung 1: alle fuenf Zustaende werden getrennt erkannt.
|
||||
func TestRun_DetectsAllStatusesSeparately(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
_, okRev := doc(t, e, "ok.txt")
|
||||
_, sumRev := doc(t, e, "summe.txt")
|
||||
_, swapRev := doc(t, e, "getauscht.txt")
|
||||
_, cutRev := doc(t, e, "verkuerzt.txt")
|
||||
_, goneRev := doc(t, e, "fehlt.txt")
|
||||
_, rotRev := doc(t, e, "beschaedigt.txt")
|
||||
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE file_revisions SET checksum_sha256 = $2 WHERE id = $1`, sumRev, strings.Repeat("0", 64)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
swapKey, _ := storageKey(t, e, swapRev)
|
||||
env, err := e.Crypto.Seal(ctx, testenv.TenantSlug, bytes.NewReader([]byte("ganz anderer inhalt, gleiche laenge?")))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.Storage.Put(ctx, swapKey, env.Ciphertext, -1, "application/octet-stream"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE file_revisions SET wrapped_dek = $2 WHERE id = $1`, swapRev, env.WrappedDEK); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE file_revisions SET size_bytes = size_bytes + 5 WHERE id = $1`, cutRev); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
goneKey, goneSize := storageKey(t, e, goneRev)
|
||||
if err := e.Storage.Delete(ctx, goneKey, goneSize); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
rotKey, _ := storageKey(t, e, rotRev)
|
||||
rc, err := e.Storage.Get(ctx, rotKey)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := io.ReadAll(rc)
|
||||
_ = rc.Close()
|
||||
raw[len(raw)-1] ^= 0xff
|
||||
if _, err := e.Storage.Put(ctx, rotKey, bytes.NewReader(raw), int64(len(raw)), "application/octet-stream"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
sum, err := newVerifier(e, Config{}).Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]string{
|
||||
okRev: StatusOK, sumRev: StatusMismatch, swapRev: StatusMismatch,
|
||||
cutRev: StatusMismatch, goneRev: StatusMissing, rotRev: StatusUnreadable,
|
||||
}
|
||||
for rev, st := range want {
|
||||
if got := statusOf(t, e, rev); got != st {
|
||||
t.Errorf("revision %s: status %q, erwartet %q", rev, got, st)
|
||||
}
|
||||
}
|
||||
if sum.Checked != 6 || sum.OK != 1 || sum.Mismatch != 3 || sum.Missing != 1 || sum.Unreadable != 1 || sum.Status != "completed" {
|
||||
t.Errorf("zusammenfassung: %+v", sum)
|
||||
}
|
||||
var stored int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT checked FROM integrity_runs WHERE id = $1 AND status = 'completed' AND finished_at IS NOT NULL`, sum.RunID).Scan(&stored)
|
||||
if stored != 6 {
|
||||
t.Errorf("lauf nicht abgeschlossen gespeichert: %d", stored)
|
||||
}
|
||||
}
|
||||
|
||||
// Falscher Schluessel und Schluesseldienst-Ausfall (AK1/AK2, Pruefung 1 und 3).
|
||||
func TestRun_WrongKeyIsUnreadableAndKEKOutageIsErrorNotViolation(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
_, rev := doc(t, e, "a.txt")
|
||||
|
||||
wrong := newVerifier(e, Config{})
|
||||
wrong.Crypto = dmscrypto.NewService(kek{key: bytes.Repeat([]byte{0x77}, dmscrypto.KEKSize)})
|
||||
if _, err := wrong.Run(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := statusOf(t, e, rev); got != StatusUnreadable {
|
||||
t.Fatalf("falscher schluessel: %q, erwartet unreadable", got)
|
||||
}
|
||||
if n := auditCount(t, e, rev); n != 1 {
|
||||
t.Fatalf("audit-ereignisse: %d, erwartet 1", n)
|
||||
}
|
||||
|
||||
down := newVerifier(e, Config{})
|
||||
down.Crypto = dmscrypto.NewService(kek{err: errors.New("kek-dienst nicht erreichbar")})
|
||||
sum, err := down.Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := statusOf(t, e, rev); got != StatusError || sum.Error != 1 {
|
||||
t.Fatalf("kek-ausfall: status %q, zusammenfassung %+v, erwartet error", got, sum)
|
||||
}
|
||||
if n := auditCount(t, e, rev); n != 1 {
|
||||
t.Fatalf("kek-ausfall darf kein weiteres audit-ereignis erzeugen: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
// AK2: genau ein Audit-Ereignis je Statuswechsel, Kette bleibt gueltig.
|
||||
func TestRun_AuditOncePerStatusChangeAndChainStaysValid(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
_, rev := doc(t, e, "a.txt")
|
||||
key, size := storageKey(t, e, rev)
|
||||
if err := e.Storage.Delete(ctx, key, size); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v := newVerifier(e, Config{})
|
||||
for i := 0; i < 3; i++ {
|
||||
if _, err := v.Run(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if n := auditCount(t, e, rev); n != 1 {
|
||||
t.Fatalf("audit-ereignisse nach drei laeufen: %d, erwartet 1", n)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE file_revisions SET checksum_sha256 = $2 WHERE id = $1`, rev, strings.Repeat("1", 64)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
env, _ := e.Crypto.Seal(ctx, testenv.TenantSlug, strings.NewReader("inhalt a.txt"))
|
||||
if _, err := e.Storage.Put(ctx, key, env.Ciphertext, -1, "application/octet-stream"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE file_revisions SET wrapped_dek = $2 WHERE id = $1`, rev, env.WrappedDEK); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := v.Run(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := statusOf(t, e, rev); got != StatusMismatch {
|
||||
t.Fatalf("status %q, erwartet mismatch", got)
|
||||
}
|
||||
if n := auditCount(t, e, rev); n != 2 {
|
||||
t.Fatalf("audit-ereignisse nach statuswechsel: %d, erwartet 2", n)
|
||||
}
|
||||
res, err := audittrail.VerifyChain(ctx, e.Pool)
|
||||
if err != nil || !res.Valid {
|
||||
t.Fatalf("hash-kette: %+v %v", res, err)
|
||||
}
|
||||
}
|
||||
|
||||
// AK3: geloeschte und Papierkorb-Dokumente werden nicht geprueft und nie gemeldet.
|
||||
func TestRun_SkipsTrashedAndDeletedDocuments(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
trashedDoc, trashedRev := doc(t, e, "papierkorb.txt")
|
||||
deletedDoc, deletedRev := doc(t, e, "geloescht.txt")
|
||||
for _, r := range []string{trashedRev, deletedRev} {
|
||||
key, size := storageKey(t, e, r)
|
||||
if err := e.Storage.Delete(ctx, key, size); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE documents SET trashed_at = now() WHERE id = $1`, trashedDoc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE documents SET deleted_at = now() WHERE id = $1`, deletedDoc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sum, err := newVerifier(e, Config{}).Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sum.Checked != 0 || statusOf(t, e, trashedRev) != "" || statusOf(t, e, deletedRev) != "" {
|
||||
t.Fatalf("papierkorb/geloescht wurde geprueft: %+v", sum)
|
||||
}
|
||||
if auditCount(t, e, trashedRev)+auditCount(t, e, deletedRev) != 0 {
|
||||
t.Fatal("audit-ereignis fuer papierkorb/geloescht")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_SkipsRevisionsStillSettling(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
_, rev := doc(t, e, "neu.txt")
|
||||
v := newVerifier(e, Config{})
|
||||
v.Now = time.Now
|
||||
sum, err := v.Run(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sum.Checked != 0 || statusOf(t, e, rev) != "" {
|
||||
t.Fatalf("frische revision wurde geprueft: %+v", sum)
|
||||
}
|
||||
}
|
||||
|
||||
// AK3 / Pruefung 2: Stichprobe je Lauf begrenzt, aelteste zuerst, nach mehreren Laeufen ist alles geprueft.
|
||||
func TestRun_SampleCoversEverythingAcrossRunsWithoutRepeats(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
var revs []string
|
||||
for i := 0; i < 7; i++ {
|
||||
_, r := doc(t, e, fmt.Sprintf("d%d.txt", i))
|
||||
revs = append(revs, r)
|
||||
}
|
||||
v := newVerifier(e, Config{Mode: ModeSample, Batch: 3})
|
||||
seen := map[string]bool{}
|
||||
for run := 0; run < 3; run++ {
|
||||
before := map[string]bool{}
|
||||
for _, r := range revs {
|
||||
if statusOf(t, e, r) != "" {
|
||||
before[r] = true
|
||||
}
|
||||
}
|
||||
sum, err := v.Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sum.Checked != 3 {
|
||||
t.Fatalf("lauf %d: %d geprueft, erwartet 3", run, sum.Checked)
|
||||
}
|
||||
if run < 2 {
|
||||
for _, r := range revs {
|
||||
if statusOf(t, e, r) != "" && !before[r] {
|
||||
if seen[r] {
|
||||
t.Fatalf("revision %s in lauf %d erneut vor allen anderen geprueft", r, run)
|
||||
}
|
||||
seen[r] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, r := range revs {
|
||||
if statusOf(t, e, r) != StatusOK {
|
||||
t.Errorf("revision %s nicht geprueft", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Pruefung 2: Abbruch mitten im Lauf, der naechste Lauf setzt fort.
|
||||
func TestRun_ResumesAfterPartialRunWithoutSkipping(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
var revs []string
|
||||
for i := 0; i < 8; i++ {
|
||||
_, r := doc(t, e, fmt.Sprintf("d%d.txt", i))
|
||||
revs = append(revs, r)
|
||||
}
|
||||
v := newVerifier(e, Config{Mode: ModeFull, Pause: 40 * time.Millisecond, MaxDuration: 100 * time.Millisecond})
|
||||
clock := time.Now().Add(time.Hour)
|
||||
v.Now = func() time.Time { clock = clock.Add(30 * time.Millisecond); return clock }
|
||||
first, err := v.Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first.Status != "partial" || first.Checked == 0 || first.Checked >= 8 {
|
||||
t.Fatalf("erster lauf: %+v, erwartet partial mit teilmenge", first)
|
||||
}
|
||||
v.Config.MaxDuration = time.Hour
|
||||
v.Config.Pause = 0
|
||||
second, err := v.Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if second.Status != "completed" || second.Checked != 8-first.Checked {
|
||||
t.Fatalf("zweiter lauf: %+v, erwartet nur die restlichen %d", second, 8-first.Checked)
|
||||
}
|
||||
for _, r := range revs {
|
||||
if statusOf(t, e, r) != StatusOK {
|
||||
t.Errorf("revision %s nicht ok/geprueft", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_RefusesParallelRun(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
conn, err := e.Pool.Acquire(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Release()
|
||||
var got bool
|
||||
if err := conn.QueryRow(ctx, `SELECT pg_try_advisory_lock(`+lockExpr+`)`).Scan(&got); err != nil || !got {
|
||||
t.Fatalf("sperre: %v %v", got, err)
|
||||
}
|
||||
defer func() { _, _ = conn.Exec(ctx, `SELECT pg_advisory_unlock(`+lockExpr+`)`) }()
|
||||
if _, err := newVerifier(e, Config{}).Run(ctx); !errors.Is(err, ErrBusy) {
|
||||
t.Fatalf("paralleler lauf: %v, erwartet ErrBusy", err)
|
||||
}
|
||||
if err := newVerifier(e, Config{}).Handler(ctx, &jobqueue.Job{}); err != nil {
|
||||
t.Fatalf("handler bei belegter sperre: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_OversizedObjectIsErrorNotViolation(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
_, rev := doc(t, e, "a.txt")
|
||||
sum, err := newVerifier(e, Config{MaxObjectBytes: 3}).Run(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if statusOf(t, e, rev) != StatusError || sum.Violations() != 0 || auditCount(t, e, rev) != 0 {
|
||||
t.Fatalf("zu grosses objekt: %+v", sum)
|
||||
}
|
||||
}
|
||||
|
||||
// Pruefung 3: Mandantentrennung. Ergebnisse liegen nur in der Datenbank des Mandanten, ein fremder Schluessel ist nie ok.
|
||||
func TestRun_TenantSeparation(t *testing.T) {
|
||||
a := testenv.New(t)
|
||||
b := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
_, revA := doc(t, a, "a.txt")
|
||||
_, revB := doc(t, b, "b.txt")
|
||||
va := newVerifier(a, Config{})
|
||||
va.Crypto = dmscrypto.NewService(kek{key: bytes.Repeat([]byte{0x55}, dmscrypto.KEKSize)})
|
||||
if _, err := va.Run(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := statusOf(t, a, revA); got != StatusUnreadable {
|
||||
t.Fatalf("fremder schluessel: %q, erwartet unreadable", got)
|
||||
}
|
||||
var nB int
|
||||
_ = b.Pool.QueryRow(ctx, `SELECT count(*) FROM integrity_results`).Scan(&nB)
|
||||
if nB != 0 || statusOf(t, b, revB) != "" {
|
||||
t.Fatalf("ergebnisse von mandant a in mandant b sichtbar: %d", nB)
|
||||
}
|
||||
if auditCount(t, b, revA) != 0 {
|
||||
t.Fatal("audit-ereignis von mandant a in mandant b sichtbar")
|
||||
}
|
||||
if _, err := newVerifier(b, Config{}).Run(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if statusOf(t, b, revB) != StatusOK {
|
||||
t.Fatalf("mandant b: %q", statusOf(t, b, revB))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_FullStartsNewCycleAfterCompletedRun(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
for i := 0; i < 3; i++ {
|
||||
doc(t, e, fmt.Sprintf("d%d.txt", i))
|
||||
}
|
||||
v := newVerifier(e, Config{Mode: ModeFull})
|
||||
for run := 0; run < 2; run++ {
|
||||
sum, err := v.Run(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sum.Status != "completed" || sum.Checked != 3 {
|
||||
t.Fatalf("lauf %d: %+v, erwartet vollstaendiger neuer zyklus mit 3 revisionen", run, sum)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,221 @@
|
||||
package integrity
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
)
|
||||
|
||||
const (
|
||||
MinInterval = time.Minute
|
||||
MaxInterval = 30 * 24 * time.Hour
|
||||
DefaultBatch = 200
|
||||
MaxBatch = 100000
|
||||
DefaultMaxDuration = time.Hour
|
||||
DefaultPause = 50 * time.Millisecond
|
||||
MaxPause = 10 * time.Second
|
||||
DefaultRevTimeout = 2 * time.Minute
|
||||
DefaultMaxObject = int64(128 << 20)
|
||||
)
|
||||
|
||||
type Window struct {
|
||||
FromMinute int
|
||||
ToMinute int
|
||||
}
|
||||
|
||||
func (w *Window) Contains(t time.Time) bool {
|
||||
if w == nil {
|
||||
return true
|
||||
}
|
||||
m := t.Hour()*60 + t.Minute()
|
||||
if w.FromMinute <= w.ToMinute {
|
||||
return m >= w.FromMinute && m < w.ToMinute
|
||||
}
|
||||
return m >= w.FromMinute || m < w.ToMinute
|
||||
}
|
||||
|
||||
type Schedule struct {
|
||||
Interval time.Duration
|
||||
Window *Window
|
||||
}
|
||||
|
||||
func parseClock(s string) (int, error) {
|
||||
parts := strings.Split(strings.TrimSpace(s), ":")
|
||||
if len(parts) != 2 {
|
||||
return 0, fmt.Errorf("uhrzeit %q nicht im format HH:MM", s)
|
||||
}
|
||||
h, err1 := strconv.Atoi(parts[0])
|
||||
m, err2 := strconv.Atoi(parts[1])
|
||||
if err1 != nil || err2 != nil || h < 0 || h > 23 || m < 0 || m > 59 {
|
||||
return 0, fmt.Errorf("uhrzeit %q ungueltig", s)
|
||||
}
|
||||
return h*60 + m, nil
|
||||
}
|
||||
|
||||
func ParseWindow(raw string) (*Window, error) {
|
||||
raw = strings.TrimSpace(raw)
|
||||
if raw == "" {
|
||||
return nil, nil
|
||||
}
|
||||
from, to, ok := strings.Cut(raw, "-")
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("zeitfenster %q nicht im format HH:MM-HH:MM", raw)
|
||||
}
|
||||
f, err := parseClock(from)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
t, err := parseClock(to)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if f == t {
|
||||
return nil, fmt.Errorf("zeitfenster %q hat keine dauer", raw)
|
||||
}
|
||||
return &Window{FromMinute: f, ToMinute: t}, nil
|
||||
}
|
||||
|
||||
func intIn(name, raw string, def, min, max int) (int, error) {
|
||||
raw = strings.TrimSpace(raw)
|
||||
if raw == "" {
|
||||
return def, nil
|
||||
}
|
||||
v, err := strconv.Atoi(raw)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%s=%q ist keine ganze zahl", name, raw)
|
||||
}
|
||||
if v < min || v > max {
|
||||
return 0, fmt.Errorf("%s=%d ausserhalb von %d..%d", name, v, min, max)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// ConfigFromEnv liest die Steuerung. Ohne INTERVAL_SECONDS ist die Pruefung aus (enabled=false).
|
||||
func ConfigFromEnv(get func(string) string, prefix string) (cfg Config, sched Schedule, enabled bool, err error) {
|
||||
key := func(n string) string { return prefix + n }
|
||||
raw := strings.TrimSpace(get(key("INTERVAL_SECONDS")))
|
||||
if raw == "" {
|
||||
return cfg, sched, false, nil
|
||||
}
|
||||
secs, err := intIn(key("INTERVAL_SECONDS"), raw, 0, int(MinInterval.Seconds()), int(MaxInterval.Seconds()))
|
||||
if err != nil {
|
||||
return cfg, sched, false, err
|
||||
}
|
||||
sched.Interval = time.Duration(secs) * time.Second
|
||||
|
||||
mode := strings.ToLower(strings.TrimSpace(get(key("MODE"))))
|
||||
switch mode {
|
||||
case "", ModeSample:
|
||||
cfg.Mode = ModeSample
|
||||
case ModeFull:
|
||||
cfg.Mode = ModeFull
|
||||
default:
|
||||
return cfg, sched, false, fmt.Errorf("%s=%q: erlaubt sind sample und full", key("MODE"), mode)
|
||||
}
|
||||
if cfg.Batch, err = intIn(key("BATCH"), get(key("BATCH")), DefaultBatch, 1, MaxBatch); err != nil {
|
||||
return cfg, sched, false, err
|
||||
}
|
||||
maxSecs, err := intIn(key("MAX_DURATION_SECONDS"), get(key("MAX_DURATION_SECONDS")), int(DefaultMaxDuration.Seconds()), 10, 24*3600)
|
||||
if err != nil {
|
||||
return cfg, sched, false, err
|
||||
}
|
||||
cfg.MaxDuration = time.Duration(maxSecs) * time.Second
|
||||
pauseMs, err := intIn(key("PAUSE_MS"), get(key("PAUSE_MS")), int(DefaultPause.Milliseconds()), 0, int(MaxPause.Milliseconds()))
|
||||
if err != nil {
|
||||
return cfg, sched, false, err
|
||||
}
|
||||
cfg.Pause = time.Duration(pauseMs) * time.Millisecond
|
||||
revSecs, err := intIn(key("REVISION_TIMEOUT_SECONDS"), get(key("REVISION_TIMEOUT_SECONDS")), int(DefaultRevTimeout.Seconds()), 1, 3600)
|
||||
if err != nil {
|
||||
return cfg, sched, false, err
|
||||
}
|
||||
cfg.RevisionTimeout = time.Duration(revSecs) * time.Second
|
||||
maxMB, err := intIn(key("MAX_OBJECT_MB"), get(key("MAX_OBJECT_MB")), int(DefaultMaxObject>>20), 1, 1<<20)
|
||||
if err != nil {
|
||||
return cfg, sched, false, err
|
||||
}
|
||||
cfg.MaxObjectBytes = int64(maxMB) << 20
|
||||
if sched.Window, err = ParseWindow(get(key("WINDOW"))); err != nil {
|
||||
return cfg, sched, false, fmt.Errorf("%s: %w", key("WINDOW"), err)
|
||||
}
|
||||
return cfg, sched, true, nil
|
||||
}
|
||||
|
||||
func (v *Verifier) Handler(ctx context.Context, _ *jobqueue.Job) error {
|
||||
_, err := v.Run(ctx)
|
||||
if errors.Is(err, ErrBusy) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func Due(ctx context.Context, pool *pgxpool.Pool, sched Schedule, now time.Time) (bool, error) {
|
||||
if sched.Interval <= 0 || !sched.Window.Contains(now) {
|
||||
return false, nil
|
||||
}
|
||||
var last *time.Time
|
||||
if err := pool.QueryRow(ctx, `SELECT max(started_at) FROM integrity_runs`).Scan(&last); err != nil {
|
||||
return false, fmt.Errorf("integrity: letzten lauf lesen: %w", err)
|
||||
}
|
||||
if last == nil {
|
||||
return true, nil
|
||||
}
|
||||
return now.Sub(*last) >= sched.Interval, nil
|
||||
}
|
||||
|
||||
func EnqueueIfDue(ctx context.Context, pool *pgxpool.Pool, queue *jobqueue.Queue, sched Schedule, now time.Time) (bool, error) {
|
||||
due, err := Due(ctx, pool, sched, now)
|
||||
if err != nil || !due {
|
||||
return false, err
|
||||
}
|
||||
key := fmt.Sprintf("%s:%d", JobType, now.Unix()/int64(sched.Interval.Seconds()))
|
||||
if _, err := queue.Enqueue(ctx, JobType, struct{}{}, jobqueue.EnqueueOptions{IdempotencyKey: key, MaxAttempts: 1}); err != nil {
|
||||
return false, fmt.Errorf("integrity: lauf einreihen: %w", err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func RunScheduler(ctx context.Context, pool *pgxpool.Pool, queue *jobqueue.Queue, sched Schedule, tick time.Duration) {
|
||||
if tick <= 0 {
|
||||
tick = 30 * time.Second
|
||||
}
|
||||
t := time.NewTicker(tick)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if _, err := EnqueueIfDue(ctx, pool, queue, sched, time.Now()); err != nil {
|
||||
log.Printf("integrity: planer: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// JobLimits liefert die FDN-11-Grenzen des Jobtyps: die Laufzeit folgt der
|
||||
// konfigurierten Hoechstdauer (sonst bricht die Standardgrenze von fuenf
|
||||
// Minuten jeden Lauf ab), der Speicher dem groessten lesbaren Objekt
|
||||
// (Chiffretext und Klartext liegen gleichzeitig im Speicher).
|
||||
func (c Config) JobLimits() jobqueue.Limits {
|
||||
maxObj := c.MaxObjectBytes
|
||||
if maxObj <= 0 {
|
||||
maxObj = DefaultMaxObject
|
||||
}
|
||||
d := c.MaxDuration
|
||||
if d <= 0 {
|
||||
d = DefaultMaxDuration
|
||||
}
|
||||
return jobqueue.Limits{
|
||||
MaxDuration: d + time.Minute,
|
||||
MaxMemoryBytes: uint64(3*maxObj) + 128<<20,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
package integrity
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
func envMap(m map[string]string) func(string) string {
|
||||
return func(k string) string { return m[k] }
|
||||
}
|
||||
|
||||
const p = "NEXARCH_DMS_WORKER_INTEGRITY_"
|
||||
|
||||
func TestConfigFromEnv_DisabledWithoutInterval(t *testing.T) {
|
||||
_, _, enabled, err := ConfigFromEnv(envMap(map[string]string{}), p)
|
||||
if err != nil || enabled {
|
||||
t.Fatalf("enabled=%v err=%v, erwartet aus", enabled, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigFromEnv_DefaultsAndBounds(t *testing.T) {
|
||||
cfg, sched, enabled, err := ConfigFromEnv(envMap(map[string]string{p + "INTERVAL_SECONDS": "3600"}), p)
|
||||
if err != nil || !enabled {
|
||||
t.Fatalf("err=%v enabled=%v", err, enabled)
|
||||
}
|
||||
if sched.Interval != time.Hour || cfg.Mode != ModeSample || cfg.Batch != DefaultBatch || cfg.MaxDuration != DefaultMaxDuration ||
|
||||
cfg.Pause != DefaultPause || cfg.RevisionTimeout != DefaultRevTimeout || cfg.MaxObjectBytes != DefaultMaxObject {
|
||||
t.Fatalf("standardwerte: %+v %+v", cfg, sched)
|
||||
}
|
||||
bad := map[string]string{
|
||||
"INTERVAL_SECONDS": "30",
|
||||
"MODE": "alles",
|
||||
"BATCH": "0",
|
||||
"MAX_DURATION_SECONDS": "1",
|
||||
"PAUSE_MS": "-1",
|
||||
"REVISION_TIMEOUT_SECONDS": "0",
|
||||
"MAX_OBJECT_MB": "abc",
|
||||
"WINDOW": "25:00-03:00",
|
||||
}
|
||||
for k, v := range bad {
|
||||
env := map[string]string{p + "INTERVAL_SECONDS": "3600"}
|
||||
env[p+k] = v
|
||||
if _, _, _, err := ConfigFromEnv(envMap(env), p); err == nil {
|
||||
t.Errorf("%s=%q wurde akzeptiert", k, v)
|
||||
}
|
||||
}
|
||||
if _, _, _, err := ConfigFromEnv(envMap(map[string]string{p + "INTERVAL_SECONDS": "3600", p + "WINDOW": "02:00-02:00"}), p); err == nil {
|
||||
t.Error("fenster ohne dauer akzeptiert")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindow(t *testing.T) {
|
||||
w, err := ParseWindow("22:00-06:00")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
at := func(h, m int) time.Time { return time.Date(2026, 9, 22, h, m, 0, 0, time.UTC) }
|
||||
for _, c := range []struct {
|
||||
t time.Time
|
||||
in bool
|
||||
}{{at(23, 0), true}, {at(3, 30), true}, {at(6, 0), false}, {at(12, 0), false}, {at(22, 0), true}} {
|
||||
if got := w.Contains(c.t); got != c.in {
|
||||
t.Errorf("%s: %v, erwartet %v", c.t.Format("15:04"), got, c.in)
|
||||
}
|
||||
}
|
||||
var none *Window
|
||||
if !none.Contains(at(12, 0)) {
|
||||
t.Error("kein fenster muss immer gelten")
|
||||
}
|
||||
if _, err := ParseWindow("nachts"); err == nil || !strings.Contains(err.Error(), "HH:MM") {
|
||||
t.Errorf("format: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnqueueIfDue_IntervalWindowAndIdempotence(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx := context.Background()
|
||||
queue := jobqueue.NewQueue(e.Pool, time.Minute)
|
||||
sched := Schedule{Interval: time.Hour}
|
||||
now := time.Date(2026, 9, 22, 10, 0, 0, 0, time.UTC)
|
||||
|
||||
if ok, err := EnqueueIfDue(ctx, e.Pool, queue, sched, now); err != nil || !ok {
|
||||
t.Fatalf("erster lauf faellig: %v %v", ok, err)
|
||||
}
|
||||
if ok, err := EnqueueIfDue(ctx, e.Pool, queue, sched, now.Add(time.Second)); err != nil || !ok {
|
||||
t.Fatalf("gleicher zeitraum ohne gestarteten lauf: %v %v", ok, err)
|
||||
}
|
||||
var jobs int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM processing_jobs WHERE job_type = $1`, JobType).Scan(&jobs)
|
||||
if jobs != 1 {
|
||||
t.Fatalf("idempotenz: %d jobs, erwartet 1", jobs)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `INSERT INTO integrity_runs (mode, started_at) VALUES ('sample', $1)`, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok, _ := EnqueueIfDue(ctx, e.Pool, queue, sched, now.Add(30*time.Minute)); ok {
|
||||
t.Fatal("vor ablauf des intervalls eingereiht")
|
||||
}
|
||||
if ok, _ := EnqueueIfDue(ctx, e.Pool, queue, sched, now.Add(61*time.Minute)); !ok {
|
||||
t.Fatal("nach ablauf des intervalls nicht eingereiht")
|
||||
}
|
||||
w, _ := ParseWindow("22:00-06:00")
|
||||
if ok, _ := EnqueueIfDue(ctx, e.Pool, queue, Schedule{Interval: time.Hour, Window: w}, now.Add(48*time.Hour)); ok {
|
||||
t.Fatal("ausserhalb des zeitfensters eingereiht")
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobLimits_FollowConfiguration(t *testing.T) {
|
||||
l := Config{MaxDuration: 2 * time.Hour, MaxObjectBytes: 64 << 20}.JobLimits()
|
||||
if l.MaxDuration != 2*time.Hour+time.Minute || l.MaxMemoryBytes != uint64(3*(64<<20))+128<<20 {
|
||||
t.Fatalf("limits: %+v", l)
|
||||
}
|
||||
if def := (Config{}).JobLimits(); def.MaxDuration <= jobqueue.DefaultLimits.MaxDuration {
|
||||
t.Fatalf("standard-laufzeit %s muss ueber der Standardgrenze %s liegen", def.MaxDuration, jobqueue.DefaultLimits.MaxDuration)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package ocr
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/textcrypt"
|
||||
)
|
||||
|
||||
type otherKEK struct{}
|
||||
|
||||
func (otherKEK) TenantKEK(context.Context, string) ([]byte, error) {
|
||||
return bytes.Repeat([]byte{0x5A}, dmscrypto.KEKSize), nil
|
||||
}
|
||||
|
||||
type rawPage struct {
|
||||
page int
|
||||
text string
|
||||
enc []byte
|
||||
}
|
||||
|
||||
func rawPages(t *testing.T, h *harness, revID string) []rawPage {
|
||||
t.Helper()
|
||||
rows, err := h.e.Pool.Query(context.Background(), `SELECT page, text, text_enc FROM revision_ocr_pages WHERE revision_id = $1 ORDER BY page`, revID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []rawPage
|
||||
for rows.Next() {
|
||||
var r rawPage
|
||||
if err := rows.Scan(&r.page, &r.text, &r.enc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func wrappedKey(t *testing.T, h *harness, revID string) []byte {
|
||||
t.Helper()
|
||||
var k []byte
|
||||
if err := h.e.Pool.QueryRow(context.Background(), `SELECT text_wrapped_dek FROM revision_ocr WHERE revision_id = $1`, revID).Scan(&k); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return k
|
||||
}
|
||||
|
||||
// FDN-22 AK1 und Pruefung 1: erkannter Text liegt nur verschluesselt in der Datenbank, die Auslieferung entschluesselt im Speicher.
|
||||
func TestRecognisedTextIsStoredEncrypted(t *testing.T) {
|
||||
h := newHarness(t)
|
||||
ctx := context.Background()
|
||||
eng := DefaultEngine()
|
||||
lines := englishPage
|
||||
img := renderReference(t, eng, lines, 200)
|
||||
docID, revID, err := h.upload(t, "scan.png", "image/png", pngBytes(t, img), false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if st := h.drain(t, revID, h.processor("w1", nil)); st.Status != StatusReady {
|
||||
t.Fatalf("status %s (%v)", st.Status, st.Error)
|
||||
}
|
||||
rows := rawPages(t, h, revID)
|
||||
if len(rows) != 1 || rows[0].text != "" || len(rows[0].enc) == 0 {
|
||||
t.Fatalf("rohzeilen: %+v", rows)
|
||||
}
|
||||
if len(wrappedKey(t, h, revID)) == 0 {
|
||||
t.Fatal("kein verpackter textschluessel abgelegt")
|
||||
}
|
||||
word := "musterhandel"
|
||||
if strings.Contains(strings.ToLower(string(rows[0].enc)), word) {
|
||||
t.Fatalf("klartextwort %q im chiffretext", word)
|
||||
}
|
||||
pages, err := Pages(ctx, h.e.Pool, h.e.Crypto, testenv.TenantSlug, revID)
|
||||
if err != nil || len(pages) != 1 || !strings.Contains(strings.ToLower(pages[0].Text), word) {
|
||||
t.Fatalf("entschluesselte seite: %+v %v", pages, err)
|
||||
}
|
||||
text, err := (TextStore{Pool: h.e.Pool, Crypto: h.e.Crypto, TenantSlug: testenv.TenantSlug}).TextOf(ctx, docID)
|
||||
if err != nil || !strings.Contains(strings.ToLower(text), word) {
|
||||
t.Fatalf("textstore: %q %v", text, err)
|
||||
}
|
||||
if l, ok, err := RevisionLanguage(ctx, h.e.Pool, revID); err != nil || !ok || l.Code != "en" {
|
||||
t.Errorf("sprache aus dem verschluesselten text nicht bestimmt: %+v %v %v", l, ok, err)
|
||||
}
|
||||
}
|
||||
|
||||
// AK2: fremder KEK und fehlender Dienst liefern keinen Klartext.
|
||||
func TestRecognisedTextFailsClosedWithoutKey(t *testing.T) {
|
||||
h := newHarness(t)
|
||||
ctx := context.Background()
|
||||
img := renderReference(t, DefaultEngine(), referencePages[0], 200)
|
||||
docID, revID, err := h.upload(t, "scan.png", "image/png", pngBytes(t, img), false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.drain(t, revID, h.processor("w1", nil))
|
||||
|
||||
foreign := dmscrypto.NewService(otherKEK{})
|
||||
if _, err := Pages(ctx, h.e.Pool, foreign, testenv.TenantSlug, revID); !errors.Is(err, textcrypt.ErrDecryptText) {
|
||||
t.Errorf("fremder kek: %v", err)
|
||||
}
|
||||
if _, err := Pages(ctx, h.e.Pool, nil, testenv.TenantSlug, revID); !errors.Is(err, textcrypt.ErrNoCrypto) {
|
||||
t.Errorf("ohne dienst: %v", err)
|
||||
}
|
||||
if text, err := (TextStore{Pool: h.e.Pool}).TextOf(ctx, docID); err == nil || text != "" {
|
||||
t.Errorf("textstore ohne dienst: %q %v", text, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ein bewusstes Duplikat uebernimmt den Text und verschluesselt ihn neu (eigener Schluessel, Bindung an die neue Revision).
|
||||
func TestReusedTextIsReEncryptedForNewRevision(t *testing.T) {
|
||||
h := newHarness(t)
|
||||
ctx := context.Background()
|
||||
img := renderReference(t, DefaultEngine(), referencePages[0], 200)
|
||||
data := pngBytes(t, img)
|
||||
_, rev1, err := h.upload(t, "a.png", "image/png", data, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.drain(t, rev1, h.processor("w1", nil))
|
||||
_, rev2, err := h.upload(t, "b.png", "image/png", data, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if st := h.drain(t, rev2, h.processor("w1", nil)); st.Status != StatusReady {
|
||||
t.Fatalf("status %s", st.Status)
|
||||
}
|
||||
if bytes.Equal(wrappedKey(t, h, rev1), wrappedKey(t, h, rev2)) {
|
||||
t.Fatal("beide revisionen teilen sich den textschluessel")
|
||||
}
|
||||
a, b := rawPages(t, h, rev1), rawPages(t, h, rev2)
|
||||
if len(a) != 1 || len(b) != 1 || b[0].text != "" || bytes.Equal(a[0].enc, b[0].enc) {
|
||||
t.Fatalf("roh: %+v / %+v", a, b)
|
||||
}
|
||||
p1, _ := Pages(ctx, h.e.Pool, h.e.Crypto, testenv.TenantSlug, rev1)
|
||||
p2, err := Pages(ctx, h.e.Pool, h.e.Crypto, testenv.TenantSlug, rev2)
|
||||
if err != nil || len(p2) != 1 || p2[0].Text != p1[0].Text || p2[0].Source != SourceReused {
|
||||
t.Fatalf("uebernommener text: %+v %v", p2, err)
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
@@ -44,12 +45,18 @@ func (p *Processor) ensureLanguage(ctx context.Context, revisionID string) error
|
||||
if err := p.Pool.QueryRow(ctx, `SELECT language FROM revision_ocr WHERE revision_id = $1`, revisionID).Scan(&have); err != nil || have != nil {
|
||||
return err
|
||||
}
|
||||
var text string
|
||||
if err := p.Pool.QueryRow(ctx, `
|
||||
SELECT COALESCE(left(string_agg(text, E'\n' ORDER BY page), $2), '') FROM revision_ocr_pages WHERE revision_id = $1
|
||||
`, revisionID, languageSampleChars).Scan(&text); err != nil {
|
||||
pages, err := loadPages(ctx, p.Pool, p.vault(), revisionID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
texts := make([]string, 0, len(pages))
|
||||
for _, pg := range pages {
|
||||
texts = append(texts, pg.Text)
|
||||
}
|
||||
text := strings.Join(texts, "\n")
|
||||
if runes := []rune(text); len(runes) > languageSampleChars {
|
||||
text = string(runes[:languageSampleChars])
|
||||
}
|
||||
res := langdetect.Detect(text)
|
||||
if res.Lang == "" {
|
||||
return nil
|
||||
|
||||
+33
-56
@@ -97,6 +97,9 @@ type Processor struct {
|
||||
// (fertig erkannt, aus durchsuchbarem PDF uebernommen oder wiederverwendet),
|
||||
// z. B. fuer die Auto-Regeln (MET-05), deren Textregeln erst dann greifen koennen.
|
||||
OnText func(ctx context.Context, documentID, revisionID string) error
|
||||
// OnPDFACreated wird aufgerufen, sobald ein PDF/A-Derivat abgelegt wurde
|
||||
// (SRC-14: reiht die Konformitaetspruefung mit veraPDF ein).
|
||||
OnPDFACreated func(ctx context.Context, documentID, revisionID string) error
|
||||
// Preprocess schaltet Rauschunterdrueckung/Entzerrung (Standard an, nur fuer Vergleichsmessungen aus).
|
||||
DisablePreprocess bool
|
||||
MaxPDFAPages int
|
||||
@@ -221,15 +224,6 @@ func detect(data []byte, mime string) kind {
|
||||
return kindUnsupported
|
||||
}
|
||||
|
||||
func (p *Processor) storePage(ctx context.Context, revisionID string, page int, text, source string, angle, conf *float64) error {
|
||||
_, err := p.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_ocr_pages (revision_id, page, text, source, angle_deg, mean_confidence, processed_by) VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (revision_id, page) DO UPDATE SET text = EXCLUDED.text, source = EXCLUDED.source, angle_deg = EXCLUDED.angle_deg,
|
||||
mean_confidence = EXCLUDED.mean_confidence, processed_by = EXCLUDED.processed_by, created_at = now()
|
||||
`, revisionID, page, text, source, angle, conf, p.worker())
|
||||
return err
|
||||
}
|
||||
|
||||
// reuse uebernimmt das Ergebnis einer Revision mit identischer Pruefsumme (bewusstes Duplikat).
|
||||
func (p *Processor) reuse(ctx context.Context, documentID, revisionID, checksum string) (bool, error) {
|
||||
var srcID string
|
||||
@@ -252,12 +246,29 @@ func (p *Processor) reuse(ctx context.Context, documentID, revisionID, checksum
|
||||
`, revisionID, srcID); err != nil {
|
||||
return false, fmt.Errorf("ocr: sprache uebernehmen: %w", err)
|
||||
}
|
||||
if _, err := p.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_ocr_pages (revision_id, page, text, source, angle_deg, mean_confidence, processed_by)
|
||||
SELECT $1, page, text, 'reused', angle_deg, mean_confidence, processed_by FROM revision_ocr_pages WHERE revision_id = $2
|
||||
ON CONFLICT (revision_id, page) DO NOTHING
|
||||
`, revisionID, srcID); err != nil {
|
||||
return false, fmt.Errorf("ocr: seiten uebernehmen: %w", err)
|
||||
srcPages, err := loadPages(ctx, p.Pool, p.vault(), srcID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("ocr: seiten der quellrevision lesen: %w", err)
|
||||
}
|
||||
if len(srcPages) > 0 {
|
||||
dek, err := revisionKey(ctx, p.Pool, p.vault(), revisionID)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
for _, pg := range srcPages {
|
||||
var angle, conf *float64
|
||||
if pg.AngleDeg != nil {
|
||||
a := float64(*pg.AngleDeg)
|
||||
angle = &a
|
||||
}
|
||||
if pg.Confidence != nil {
|
||||
c := float64(*pg.Confidence)
|
||||
conf = &c
|
||||
}
|
||||
if err := p.storePageKey(ctx, dek, revisionID, pg.Number, pg.Text, SourceReused, angle, conf, false); err != nil {
|
||||
return false, fmt.Errorf("ocr: seiten uebernehmen: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
reason := "ergebnis der revision " + srcID + " wiederverwendet (identische pruefsumme)"
|
||||
if err := p.setStatus(ctx, revisionID, status, total, &reason); err != nil {
|
||||
@@ -309,13 +320,19 @@ func (p *Processor) plan(ctx context.Context, pl planPayload) error {
|
||||
return err
|
||||
}
|
||||
var pending []int
|
||||
var textKey []byte
|
||||
for page := 1; page <= total; page++ {
|
||||
text, err := p.Engine.PDFText(ctx, pdfPath, page)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len([]rune(strings.TrimSpace(text))) >= minPDFTextChars {
|
||||
if err := p.storePage(ctx, pl.RevisionID, page, text, SourcePDFText, nil, nil); err != nil {
|
||||
if textKey == nil {
|
||||
if textKey, err = revisionKey(ctx, p.Pool, p.vault(), pl.RevisionID); err != nil {
|
||||
return fmt.Errorf("ocr: textschluessel: %w", err)
|
||||
}
|
||||
}
|
||||
if err := p.storePageKey(ctx, textKey, pl.RevisionID, page, text, SourcePDFText, nil, nil, true); err != nil {
|
||||
return fmt.Errorf("ocr: seitentext speichern: %w", err)
|
||||
}
|
||||
continue
|
||||
@@ -514,43 +531,3 @@ type Page struct {
|
||||
Confidence *float32
|
||||
ProcessedBy *string
|
||||
}
|
||||
|
||||
func Pages(ctx context.Context, pool *pgxpool.Pool, revisionID string) ([]Page, error) {
|
||||
rows, err := pool.Query(ctx, `SELECT page, text, source, angle_deg, mean_confidence, processed_by FROM revision_ocr_pages WHERE revision_id = $1 ORDER BY page`, revisionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := []Page{}
|
||||
for rows.Next() {
|
||||
var pg Page
|
||||
if err := rows.Scan(&pg.Number, &pg.Text, &pg.Source, &pg.AngleDeg, &pg.Confidence, &pg.ProcessedBy); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, pg)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// TextStore liefert den Volltext der aktuellen Revision (fuer Regeln, Suche).
|
||||
type TextStore struct{ Pool *pgxpool.Pool }
|
||||
|
||||
func (t TextStore) TextOf(ctx context.Context, documentID string) (string, error) {
|
||||
rows, err := t.Pool.Query(ctx, `
|
||||
SELECT p.text FROM documents d JOIN revision_ocr_pages p ON p.revision_id = d.current_revision_id
|
||||
WHERE d.id = $1 ORDER BY p.page
|
||||
`, documentID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer rows.Close()
|
||||
var parts []string
|
||||
for rows.Next() {
|
||||
var s string
|
||||
if err := rows.Scan(&s); err != nil {
|
||||
return "", err
|
||||
}
|
||||
parts = append(parts, s)
|
||||
}
|
||||
return strings.Join(parts, "\n\n"), rows.Err()
|
||||
}
|
||||
@@ -209,7 +209,7 @@ func TestRecognitionRateOnReferenceScans(t *testing.T) {
|
||||
if st.Status != StatusReady {
|
||||
t.Fatalf("seite %d: status %s (%v)", i, st.Status, st.Error)
|
||||
}
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, revID)
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, h.e.Crypto, testenv.TenantSlug, revID)
|
||||
if len(pages) != 1 || pages[0].Source != SourceOCR || pages[0].Number != 1 {
|
||||
t.Fatalf("seitenweise speicherung: %+v", pages)
|
||||
}
|
||||
@@ -219,7 +219,7 @@ func TestRecognitionRateOnReferenceScans(t *testing.T) {
|
||||
if acc < 0.95 {
|
||||
t.Errorf("scan %d: genauigkeit %.3f unter 0,95 (text: %q)", i+1, acc, pages[0].Text)
|
||||
}
|
||||
if text, _ := (TextStore{Pool: h.e.Pool}).TextOf(context.Background(), docID); !strings.Contains(strings.ToLower(text), strings.ToLower(strings.Fields(lines[0])[0])) {
|
||||
if text, _ := (TextStore{Pool: h.e.Pool, Crypto: h.e.Crypto, TenantSlug: testenv.TenantSlug}).TextOf(context.Background(), docID); !strings.Contains(strings.ToLower(text), strings.ToLower(strings.Fields(lines[0])[0])) {
|
||||
t.Errorf("textstore liefert den text nicht: %q", text)
|
||||
}
|
||||
}
|
||||
@@ -272,7 +272,7 @@ func TestSearchablePDFIsSkipped(t *testing.T) {
|
||||
if st.Status != StatusSkipped || st.SkipReason == nil || !strings.Contains(*st.SkipReason, "durchsuchbar") {
|
||||
t.Fatalf("status %s, grund %v", st.Status, st.SkipReason)
|
||||
}
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, revID)
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, h.e.Crypto, testenv.TenantSlug, revID)
|
||||
if len(pages) != 1 || pages[0].Source != SourcePDFText || accuracy(referencePages[0], pages[0].Text) < 0.95 {
|
||||
t.Fatalf("uebernommener pdf-text: %+v", pages)
|
||||
}
|
||||
@@ -321,7 +321,7 @@ func TestScalesAcrossMultipleWorkers(t *testing.T) {
|
||||
if st.Status != StatusReady {
|
||||
t.Fatalf("status %s (%v)", st.Status, st.Error)
|
||||
}
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, revID)
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, h.e.Crypto, testenv.TenantSlug, revID)
|
||||
if len(pages) != pageCount {
|
||||
t.Fatalf("%d seiten gespeichert, erwartet %d", len(pages), pageCount)
|
||||
}
|
||||
@@ -362,7 +362,7 @@ func TestLargeDocumentContinuesInFollowUpJobsAndPageTimeoutFailsVisibly(t *testi
|
||||
if n := h.jobCount(t, JobPages); n < 4 {
|
||||
t.Errorf("nur %d seitenjobs: das zeitbudget wurde nicht an folgejobs abgegeben", n)
|
||||
}
|
||||
if pages, _ := Pages(context.Background(), h.e.Pool, revID); len(pages) != 8 {
|
||||
if pages, _ := Pages(context.Background(), h.e.Pool, h.e.Crypto, testenv.TenantSlug, revID); len(pages) != 8 {
|
||||
t.Errorf("%d von 8 seiten gespeichert", len(pages))
|
||||
}
|
||||
|
||||
@@ -412,7 +412,7 @@ func TestDuplicatesAreNotRecognisedAgain(t *testing.T) {
|
||||
if st.Status != StatusReady || st.SkipReason == nil || !strings.Contains(*st.SkipReason, "wiederverwendet") {
|
||||
t.Fatalf("bewusstes duplikat: %+v %v", st, st.SkipReason)
|
||||
}
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, rev2)
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, h.e.Crypto, testenv.TenantSlug, rev2)
|
||||
if len(pages) != 1 || pages[0].Source != SourceReused {
|
||||
t.Errorf("seiten des bewussten duplikats: %+v", pages)
|
||||
}
|
||||
|
||||
@@ -236,7 +236,13 @@ func (p *Processor) buildPDFA(ctx context.Context, pl planPayload) error {
|
||||
size_bytes = $4, original_size_bytes = $5, page_count = $6, error = NULL, updated_at = now()
|
||||
WHERE revision_id = $1
|
||||
`, pl.RevisionID, key, env.WrappedDEK, len(out), len(data), pages)
|
||||
return err
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if p.OnPDFACreated != nil {
|
||||
return p.OnPDFACreated(ctx, pl.DocumentID, pl.RevisionID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PDFAState ist der Zustand des PDF/A-Derivats einer Revision.
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/ocr/imgproc"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testcorpus"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -110,7 +111,7 @@ func TestConfidencePerPageIsAvailable(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.drain(t, revID, h.processor("w1", nil))
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, revID)
|
||||
pages, _ := Pages(context.Background(), h.e.Pool, h.e.Crypto, testenv.TenantSlug, revID)
|
||||
if len(pages) != len(referencePages) {
|
||||
t.Fatalf("seiten: %d", len(pages))
|
||||
}
|
||||
@@ -157,7 +158,7 @@ func TestPoorScansLandInReviewListAndThresholdIsConfigurable(t *testing.T) {
|
||||
}
|
||||
below := 0
|
||||
for id, rev := range bad {
|
||||
pages, _ := Pages(ctx, h.e.Pool, rev)
|
||||
pages, _ := Pages(ctx, h.e.Pool, h.e.Crypto, testenv.TenantSlug, rev)
|
||||
conf := float64(*pages[0].Confidence)
|
||||
_, listed := inList[id]
|
||||
t.Logf("degradierter scan: konfidenz %.1f, in pruefliste: %v", conf, listed)
|
||||
@@ -175,7 +176,7 @@ func TestPoorScansLandInReviewListAndThresholdIsConfigurable(t *testing.T) {
|
||||
t.Fatalf("nur %d von %d degradierten scans unter dem schwellwert, testset zu mild", below, len(bad))
|
||||
}
|
||||
for id, rev := range good {
|
||||
pages, _ := Pages(ctx, h.e.Pool, rev)
|
||||
pages, _ := Pages(ctx, h.e.Pool, h.e.Crypto, testenv.TenantSlug, rev)
|
||||
t.Logf("guter scan: konfidenz %.1f", *pages[0].Confidence)
|
||||
if _, ok := inList[id]; ok {
|
||||
t.Errorf("guter scan %s steht faelschlich in der pruefliste", id)
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package ocr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/textcrypt"
|
||||
)
|
||||
|
||||
func (p *Processor) vault() textcrypt.Vault {
|
||||
return textcrypt.Vault{Crypto: p.Crypto, TenantSlug: p.TenantSlug}
|
||||
}
|
||||
|
||||
func revisionKey(ctx context.Context, pool *pgxpool.Pool, v textcrypt.Vault, revisionID string) ([]byte, error) {
|
||||
return textcrypt.OCRKey(ctx, pool, v, revisionID)
|
||||
}
|
||||
|
||||
func (p *Processor) storePage(ctx context.Context, revisionID string, page int, text, source string, angle, conf *float64) error {
|
||||
dek, err := revisionKey(ctx, p.Pool, p.vault(), revisionID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return p.storePageKey(ctx, dek, revisionID, page, text, source, angle, conf, true)
|
||||
}
|
||||
|
||||
func (p *Processor) storePageKey(ctx context.Context, dek []byte, revisionID string, page int, text, source string, angle, conf *float64, overwrite bool) error {
|
||||
sealed, err := textcrypt.Seal(dek, textcrypt.AADPage(revisionID, page), text)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
onConflict := "DO NOTHING"
|
||||
if overwrite {
|
||||
onConflict = `DO UPDATE SET text = '', text_enc = EXCLUDED.text_enc, source = EXCLUDED.source, angle_deg = EXCLUDED.angle_deg,
|
||||
mean_confidence = EXCLUDED.mean_confidence, processed_by = EXCLUDED.processed_by, created_at = now()`
|
||||
}
|
||||
_, err = p.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_ocr_pages (revision_id, page, text, text_enc, source, angle_deg, mean_confidence, processed_by) VALUES ($1, $2, '', $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (revision_id, page) `+onConflict, revisionID, page, sealed, source, angle, conf, p.worker())
|
||||
return err
|
||||
}
|
||||
|
||||
// loadPages liest die Seiten einer Revision und entschluesselt sie im Speicher; unverschluesselte
|
||||
// Altzeilen (text_enc IS NULL) bleiben bis zur Nachverschluesselung lesbar.
|
||||
func loadPages(ctx context.Context, pool *pgxpool.Pool, v textcrypt.Vault, revisionID string) ([]Page, error) {
|
||||
rows, err := pool.Query(ctx, `SELECT page, text, text_enc, source, angle_deg, mean_confidence, processed_by FROM revision_ocr_pages WHERE revision_id = $1 ORDER BY page`, revisionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
type raw struct {
|
||||
pg Page
|
||||
enc []byte
|
||||
}
|
||||
var all []raw
|
||||
needKey := false
|
||||
for rows.Next() {
|
||||
var r raw
|
||||
if err := rows.Scan(&r.pg.Number, &r.pg.Text, &r.enc, &r.pg.Source, &r.pg.AngleDeg, &r.pg.Confidence, &r.pg.ProcessedBy); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
needKey = needKey || r.enc != nil
|
||||
all = append(all, r)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var dek []byte
|
||||
if needKey {
|
||||
var wrapped []byte
|
||||
if err := pool.QueryRow(ctx, `SELECT text_wrapped_dek FROM revision_ocr WHERE revision_id = $1`, revisionID).Scan(&wrapped); err != nil {
|
||||
return nil, fmt.Errorf("ocr: textschluessel lesen: %w", err)
|
||||
}
|
||||
if dek, err = v.Key(ctx, wrapped); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
out := make([]Page, 0, len(all))
|
||||
for _, r := range all {
|
||||
if r.enc != nil {
|
||||
if r.pg.Text, err = textcrypt.Open(dek, textcrypt.AADPage(revisionID, r.pg.Number), r.enc); err != nil {
|
||||
return nil, fmt.Errorf("ocr: seite %d: %w", r.pg.Number, err)
|
||||
}
|
||||
}
|
||||
out = append(out, r.pg)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Pages liefert den entschluesselten Text je Seite.
|
||||
func Pages(ctx context.Context, pool *pgxpool.Pool, crypto *dmscrypto.Service, tenantSlug, revisionID string) ([]Page, error) {
|
||||
return loadPages(ctx, pool, textcrypt.Vault{Crypto: crypto, TenantSlug: tenantSlug}, revisionID)
|
||||
}
|
||||
|
||||
// TextStore liefert den Volltext der aktuellen Revision (fuer Regeln, Suche).
|
||||
type TextStore struct {
|
||||
Pool *pgxpool.Pool
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
}
|
||||
|
||||
func (t TextStore) TextOf(ctx context.Context, documentID string) (string, error) {
|
||||
var revisionID *string
|
||||
if err := t.Pool.QueryRow(ctx, `SELECT current_revision_id::text FROM documents WHERE id = $1`, documentID).Scan(&revisionID); err != nil || revisionID == nil {
|
||||
return "", nil
|
||||
}
|
||||
pages, err := loadPages(ctx, t.Pool, textcrypt.Vault{Crypto: t.Crypto, TenantSlug: t.TenantSlug}, *revisionID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
parts := make([]string, 0, len(pages))
|
||||
for _, pg := range pages {
|
||||
parts = append(parts, pg.Text)
|
||||
}
|
||||
return strings.Join(parts, "\n\n"), nil
|
||||
}
|
||||
@@ -4540,6 +4540,957 @@
|
||||
},
|
||||
"x-permission": "dms.document.read"
|
||||
}
|
||||
},
|
||||
"/documents/{id}/share-links": {
|
||||
"post": {
|
||||
"summary": "Freigabelink für ein Dokument anlegen",
|
||||
"tags": [
|
||||
"Freigabelinks"
|
||||
],
|
||||
"operationId": "post_document_share_links",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Dokument-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"ttl_seconds": {
|
||||
"type": "integer",
|
||||
"description": "Gültigkeitsdauer; Standard und Obergrenze per Umgebungsvariable"
|
||||
},
|
||||
"password": {
|
||||
"type": "string",
|
||||
"description": "Optional; wird nur als Hash gespeichert"
|
||||
},
|
||||
"download_limit": {
|
||||
"type": "integer",
|
||||
"description": "Optionale Obergrenze der Abrufe"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"201": {
|
||||
"description": "Angelegt, enthält den Token einmalig in der URL",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"url": {
|
||||
"type": "string",
|
||||
"description": "Nur in der Antwort auf das Anlegen enthalten"
|
||||
},
|
||||
"expires_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"download_limit": {
|
||||
"type": "integer",
|
||||
"nullable": true
|
||||
},
|
||||
"download_count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"has_password": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"revoked_at": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"nullable": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Ungültige Anfrage",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"503": {
|
||||
"description": "Nicht verfügbar",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.sharelink.write"
|
||||
},
|
||||
"get": {
|
||||
"summary": "Freigabelinks eines Dokuments auflisten (ohne Token/Passwort)",
|
||||
"tags": [
|
||||
"Freigabelinks"
|
||||
],
|
||||
"operationId": "get_document_share_links",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Dokument-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Liste",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"url": {
|
||||
"type": "string",
|
||||
"description": "Nur in der Antwort auf das Anlegen enthalten"
|
||||
},
|
||||
"expires_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"download_limit": {
|
||||
"type": "integer",
|
||||
"nullable": true
|
||||
},
|
||||
"download_count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"has_password": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"revoked_at": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"nullable": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.sharelink.read"
|
||||
}
|
||||
},
|
||||
"/documents/{id}/share-links/{linkID}": {
|
||||
"delete": {
|
||||
"summary": "Freigabelink sofort widerrufen",
|
||||
"tags": [
|
||||
"Freigabelinks"
|
||||
],
|
||||
"operationId": "delete_document_share_link",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Dokument-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "linkID",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Freigabelink-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"204": {
|
||||
"description": "Widerrufen"
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.sharelink.write"
|
||||
}
|
||||
},
|
||||
"/share/{token}": {
|
||||
"get": {
|
||||
"summary": "Freigegebenes Dokument ohne Konto abrufen (ohne Passwort)",
|
||||
"tags": [
|
||||
"Freigabelinks"
|
||||
],
|
||||
"operationId": "get_share",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "token",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Freigabelink-Token",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Dateiinhalt",
|
||||
"content": {
|
||||
"application/octet-stream": {
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"format": "binary"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"423": {
|
||||
"description": "Gesperrt",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "-",
|
||||
"security": []
|
||||
},
|
||||
"post": {
|
||||
"summary": "Freigegebenes Dokument mit Passwort abrufen",
|
||||
"tags": [
|
||||
"Freigabelinks"
|
||||
],
|
||||
"operationId": "post_share",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "token",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Freigabelink-Token",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"password": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Dateiinhalt",
|
||||
"content": {
|
||||
"application/octet-stream": {
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"format": "binary"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"423": {
|
||||
"description": "Gesperrt",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "-",
|
||||
"security": []
|
||||
}
|
||||
},
|
||||
"/triggers": {
|
||||
"post": {
|
||||
"summary": "Trigger anlegen",
|
||||
"tags": [
|
||||
"Trigger"
|
||||
],
|
||||
"operationId": "post_triggers",
|
||||
"requestBody": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"name",
|
||||
"event",
|
||||
"action_type"
|
||||
],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"event": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"document.uploaded",
|
||||
"document.ocr_completed",
|
||||
"document.einvoice_checked"
|
||||
]
|
||||
},
|
||||
"filename_pattern": {
|
||||
"type": "string"
|
||||
},
|
||||
"document_type_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"action_type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"webhook",
|
||||
"set_tag",
|
||||
"set_type"
|
||||
]
|
||||
},
|
||||
"action_tag_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"action_document_type_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"webhook_url": {
|
||||
"type": "string"
|
||||
},
|
||||
"webhook_secret": {
|
||||
"type": "string",
|
||||
"description": "Optional; wird nur verschluesselt gespeichert, nie im Klartext gelesen"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"201": {
|
||||
"description": "Angelegt",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"event": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"document.uploaded",
|
||||
"document.ocr_completed",
|
||||
"document.einvoice_checked"
|
||||
]
|
||||
},
|
||||
"filename_pattern": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"document_type_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"action_type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"webhook",
|
||||
"set_tag",
|
||||
"set_type"
|
||||
]
|
||||
},
|
||||
"action_tag_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"action_document_type_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"webhook_url": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"created_by": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Ungültige Anfrage",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.trigger.write"
|
||||
},
|
||||
"get": {
|
||||
"summary": "Trigger auflisten",
|
||||
"tags": [
|
||||
"Trigger"
|
||||
],
|
||||
"operationId": "get_triggers",
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Liste",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"event": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"document.uploaded",
|
||||
"document.ocr_completed",
|
||||
"document.einvoice_checked"
|
||||
]
|
||||
},
|
||||
"filename_pattern": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"document_type_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"action_type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"webhook",
|
||||
"set_tag",
|
||||
"set_type"
|
||||
]
|
||||
},
|
||||
"action_tag_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"action_document_type_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"webhook_url": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"created_by": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.trigger.read"
|
||||
}
|
||||
},
|
||||
"/triggers/{id}": {
|
||||
"get": {
|
||||
"summary": "Trigger lesen",
|
||||
"tags": [
|
||||
"Trigger"
|
||||
],
|
||||
"operationId": "get_trigger",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Trigger-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Trigger",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"event": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"document.uploaded",
|
||||
"document.ocr_completed",
|
||||
"document.einvoice_checked"
|
||||
]
|
||||
},
|
||||
"filename_pattern": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"document_type_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"action_type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"webhook",
|
||||
"set_tag",
|
||||
"set_type"
|
||||
]
|
||||
},
|
||||
"action_tag_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"action_document_type_id": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"webhook_url": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"created_by": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.trigger.read"
|
||||
},
|
||||
"delete": {
|
||||
"summary": "Trigger loeschen",
|
||||
"tags": [
|
||||
"Trigger"
|
||||
],
|
||||
"operationId": "delete_trigger",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Trigger-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"204": {
|
||||
"description": "Geloescht"
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Ungültige Anfrage",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.trigger.write"
|
||||
}
|
||||
},
|
||||
"/triggers/{id}/enable": {
|
||||
"post": {
|
||||
"summary": "Trigger aktivieren",
|
||||
"tags": [
|
||||
"Trigger"
|
||||
],
|
||||
"operationId": "post_trigger_enable",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Trigger-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"204": {
|
||||
"description": "Ok"
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Ungültige Anfrage",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.trigger.write"
|
||||
}
|
||||
},
|
||||
"/triggers/{id}/disable": {
|
||||
"post": {
|
||||
"summary": "Trigger deaktivieren",
|
||||
"tags": [
|
||||
"Trigger"
|
||||
],
|
||||
"operationId": "post_trigger_disable",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "Trigger-ID",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"204": {
|
||||
"description": "Ok"
|
||||
},
|
||||
"404": {
|
||||
"description": "Nicht gefunden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"400": {
|
||||
"description": "Ungültige Anfrage",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Nicht angemeldet",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Keine Berechtigung",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-permission": "dms.trigger.write"
|
||||
}
|
||||
}
|
||||
},
|
||||
"components": {
|
||||
@@ -4577,4 +5528,4 @@
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -135,12 +135,12 @@ func TestMatchPrefersLiteralSegments(t *testing.T) {
|
||||
|
||||
func TestPublicPathsAreExactlySignatureAndServiceRoutes(t *testing.T) {
|
||||
got := PublicPaths()
|
||||
for _, p := range []string{"/download", "/openapi.json", "/retention/destroy-callback"} {
|
||||
for _, p := range []string{"/download", "/openapi.json", "/retention/destroy-callback", "/share/{token}"} {
|
||||
if !got[p] {
|
||||
t.Errorf("%s fehlt in PublicPaths", p)
|
||||
}
|
||||
}
|
||||
if len(got) != 3 {
|
||||
if len(got) != 4 {
|
||||
t.Errorf("unerwartete oeffentliche pfade: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package pdfaconform
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/ocr"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
)
|
||||
|
||||
const (
|
||||
JobType = "pdfa.validate"
|
||||
SystemActor = "system:pdfaconform"
|
||||
)
|
||||
|
||||
type jobPayload struct {
|
||||
DocumentID string `json:"document_id"`
|
||||
RevisionID string `json:"revision_id"`
|
||||
}
|
||||
|
||||
// Processor prueft PDF/A-Derivate (SRC-06) gegen den optionalen veraPDF-Sidecar.
|
||||
type Processor struct {
|
||||
Pool *pgxpool.Pool
|
||||
Storage *storage.Service
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
Validator *Validator
|
||||
}
|
||||
|
||||
// Enqueue reiht die Konformitaetspruefung eines PDF/A-Derivats idempotent ein.
|
||||
func Enqueue(ctx context.Context, pool *pgxpool.Pool, queue *jobqueue.Queue, documentID, revisionID string) error {
|
||||
_, err := queue.Enqueue(ctx, JobType, jobPayload{DocumentID: documentID, RevisionID: revisionID}, jobqueue.EnqueueOptions{
|
||||
IdempotencyKey: "pdfa:validate:" + revisionID,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (p *Processor) Handler(ctx context.Context, job *jobqueue.Job) error {
|
||||
var pl jobPayload
|
||||
if err := json.Unmarshal(job.Payload, &pl); err != nil {
|
||||
return fmt.Errorf("pdfaconform: auftrag dekodieren: %w", err)
|
||||
}
|
||||
return p.Validate(ctx, pl.DocumentID, pl.RevisionID)
|
||||
}
|
||||
|
||||
// Validate prueft das PDF/A-Derivat einer Revision und haengt das Ergebnis an.
|
||||
// Gibt es kein bereites Derivat mehr (z. B. Ueberspringen nach erneuter
|
||||
// Erzeugung), ist das kein Fehler.
|
||||
func (p *Processor) Validate(ctx context.Context, documentID, revisionID string) error {
|
||||
pdf, err := ocr.OpenPDFA(ctx, p.Pool, p.Storage, p.Crypto, p.TenantSlug, revisionID)
|
||||
if errors.Is(err, ocr.ErrPDFANotFound) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("pdfaconform: derivat lesen: %w", err)
|
||||
}
|
||||
var out Outcome
|
||||
if p.Validator == nil {
|
||||
out = notChecked("kein Validator konfiguriert")
|
||||
} else {
|
||||
out = p.Validator.Validate(ctx, pdf)
|
||||
}
|
||||
return p.storeConformance(ctx, documentID, revisionID, out)
|
||||
}
|
||||
|
||||
func (p *Processor) storeConformance(ctx context.Context, documentID, revisionID string, out Outcome) error {
|
||||
msgs := out.Messages
|
||||
if msgs == nil {
|
||||
msgs = []Message{}
|
||||
}
|
||||
encoded, err := json.Marshal(msgs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pdfaconform: meldungen serialisieren: %w", err)
|
||||
}
|
||||
tag, err := p.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_pdfa_conformance
|
||||
(revision_id, status, profile, validator_version, report_sha256, reason, messages)
|
||||
SELECT $1, $2, $3, $4, $5, $6, $7
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM (
|
||||
SELECT status, profile, validator_version, report_sha256, reason
|
||||
FROM revision_pdfa_conformance WHERE revision_id = $1
|
||||
ORDER BY checked_at DESC, id DESC LIMIT 1
|
||||
) last
|
||||
WHERE last.status = $2 AND last.profile = $3 AND last.validator_version = $4
|
||||
AND last.report_sha256 = $5 AND last.reason = $6
|
||||
)`, revisionID, out.Status, out.Profile, out.ValidatorVersion, out.ReportSHA256, out.Reason, encoded)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pdfaconform: konformitaet speichern: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return nil
|
||||
}
|
||||
if _, err := audittrail.Append(ctx, p.Pool, SystemActor, "pdfa_validated", "document", documentID, map[string]any{
|
||||
"revision_id": revisionID, "status": out.Status, "profile": out.Profile,
|
||||
"validator_version": out.ValidatorVersion, "report_sha256": out.ReportSHA256,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("pdfaconform: audit-eintrag anfuegen: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Conformance ist das juengste gespeicherte Ergebnis.
|
||||
type Conformance struct {
|
||||
Status string `json:"status"`
|
||||
Profile string `json:"profile"`
|
||||
ValidatorVersion string `json:"validator_version"`
|
||||
ReportSHA256 string `json:"report_sha256"`
|
||||
Reason string `json:"reason"`
|
||||
Messages []Message `json:"messages"`
|
||||
}
|
||||
|
||||
// LatestConformance liefert das juengste Ergebnis einer Revision (nil = nie geprueft).
|
||||
func LatestConformance(ctx context.Context, pool *pgxpool.Pool, revisionID string) (*Conformance, error) {
|
||||
var c Conformance
|
||||
var raw []byte
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT status, profile, validator_version, report_sha256, reason, messages
|
||||
FROM revision_pdfa_conformance WHERE revision_id = $1 ORDER BY checked_at DESC, id DESC LIMIT 1`, revisionID).
|
||||
Scan(&c.Status, &c.Profile, &c.ValidatorVersion, &c.ReportSHA256, &c.Reason, &raw)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pdfaconform: konformitaet lesen: %w", err)
|
||||
}
|
||||
if err := json.Unmarshal(raw, &c.Messages); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &c, nil
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
package pdfaconform
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/ocr"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
func processor(e *testenv.Env) *Processor {
|
||||
return &Processor{Pool: e.Pool, Storage: e.Storage, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug}
|
||||
}
|
||||
|
||||
// seedPDFA legt ein bereites PDF/A-Derivat an, ohne die OCR-Pipeline (SRC-06)
|
||||
// laufen zu lassen; das haelt den Test auf den Konformitaetsschritt fokussiert.
|
||||
func seedPDFA(t *testing.T, e *testenv.Env, docID, revID string, pdf []byte) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
env, err := e.Crypto.Seal(ctx, testenv.TenantSlug, bytes.NewReader(pdf))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cipher, err := io.ReadAll(env.Ciphertext)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key := storage.DerivativeKey(docID, revID, "pdfa.pdf")
|
||||
if _, err := e.Storage.Put(ctx, key, bytes.NewReader(cipher), int64(len(cipher)), "application/octet-stream"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_pdfa (revision_id, status, storage_key, wrapped_dek, size_bytes)
|
||||
VALUES ($1, 'ready', $2, $3, $4)`, revID, key, env.WrappedDEK, len(pdf)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func rowsOf(t *testing.T, e *testenv.Env, revID string) []Conformance {
|
||||
t.Helper()
|
||||
rows, err := e.Pool.Query(context.Background(), `
|
||||
SELECT status, profile, validator_version, report_sha256, reason
|
||||
FROM revision_pdfa_conformance WHERE revision_id = $1 ORDER BY checked_at, id`, revID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []Conformance
|
||||
for rows.Next() {
|
||||
var c Conformance
|
||||
if err := rows.Scan(&c.Status, &c.Profile, &c.ValidatorVersion, &c.ReportSHA256, &c.Reason); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out = append(out, c)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func auditCount(t *testing.T, e *testenv.Env, docID string) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
if err := e.Pool.QueryRow(context.Background(), `SELECT count(*) FROM audit_log WHERE action = 'pdfa_validated' AND object_reference = $1`, docID).Scan(&n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// AK1: Ergebnis mit Status, Profil, Werkzeugversion, Berichts-Hash und Audit; nicht konformes Derivat bleibt erhalten.
|
||||
func TestValidate_ResultCarriesVersionHashAndAudit_NonCompliantStays(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, revID := e.UploadDoc(t, "s.pdf", "application/pdf", []byte("dummy"))
|
||||
seedPDFA(t, e, docID, revID, []byte("%PDF-fake"))
|
||||
p := processor(e)
|
||||
p.Validator = stub(t, 200, nonCompliantReport)
|
||||
|
||||
if err := p.Validate(context.Background(), docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rows := rowsOf(t, e, revID)
|
||||
if len(rows) != 1 || rows[0].Status != StatusNonCompliant || rows[0].ValidatorVersion != "1.30.2" || rows[0].ReportSHA256 == "" {
|
||||
t.Fatalf("konformitaet: %+v", rows)
|
||||
}
|
||||
if n := auditCount(t, e, docID); n != 1 {
|
||||
t.Errorf("audit-eintraege = %d, want 1", n)
|
||||
}
|
||||
if st, err := ocr.GetPDFAState(context.Background(), e.Pool, revID); err != nil || st.Status != ocr.StatusReady {
|
||||
t.Errorf("derivat veraendert: status=%v err=%v", st.Status, err)
|
||||
}
|
||||
}
|
||||
|
||||
// AK2: kein Validator konfiguriert -> not_checked, Erzeugung des Derivats bleibt unberuehrt.
|
||||
func TestValidate_WithoutValidatorIsNotChecked(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, revID := e.UploadDoc(t, "s.pdf", "application/pdf", []byte("dummy"))
|
||||
seedPDFA(t, e, docID, revID, []byte("%PDF-fake"))
|
||||
p := processor(e)
|
||||
|
||||
if err := p.Validate(context.Background(), docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rows := rowsOf(t, e, revID)
|
||||
if len(rows) != 1 || rows[0].Status != StatusNotChecked || rows[0].Reason == "" {
|
||||
t.Fatalf("konformitaet: %+v", rows)
|
||||
}
|
||||
if st, err := ocr.GetPDFAState(context.Background(), e.Pool, revID); err != nil || st.Status != ocr.StatusReady {
|
||||
t.Errorf("derivat veraendert: status=%v err=%v", st.Status, err)
|
||||
}
|
||||
}
|
||||
|
||||
// AK2: nicht erreichbarer Sidecar -> not_checked, nie compliant.
|
||||
func TestValidate_UnreachableSidecarIsNotChecked(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, revID := e.UploadDoc(t, "s.pdf", "application/pdf", []byte("dummy"))
|
||||
seedPDFA(t, e, docID, revID, []byte("%PDF-fake"))
|
||||
p := processor(e)
|
||||
v, err := NewValidator("http://127.0.0.1:1", "2b", 200*time.Millisecond)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p.Validator = v
|
||||
|
||||
if err := p.Validate(context.Background(), docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rows := rowsOf(t, e, revID)
|
||||
if len(rows) != 1 || rows[0].Status != StatusNotChecked {
|
||||
t.Fatalf("konformitaet: %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
// AK1: eine unveraenderte Wiederholung haengt keine weitere Zeile an.
|
||||
func TestValidate_UnchangedResultDoesNotDuplicate(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, revID := e.UploadDoc(t, "s.pdf", "application/pdf", []byte("dummy"))
|
||||
seedPDFA(t, e, docID, revID, []byte("%PDF-fake"))
|
||||
p := processor(e)
|
||||
p.Validator = stub(t, 200, compliantReport)
|
||||
|
||||
if err := p.Validate(context.Background(), docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := p.Validate(context.Background(), docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rows := rowsOf(t, e, revID); len(rows) != 1 {
|
||||
t.Fatalf("konformitaet: %d zeilen, want 1", len(rows))
|
||||
}
|
||||
if n := auditCount(t, e, docID); n != 1 {
|
||||
t.Errorf("audit-eintraege = %d, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Ohne bereites Derivat (z. B. abgeraeumt) ist die Pruefung kein Fehler.
|
||||
func TestValidate_NoDerivativeIsNoOp(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, revID := e.UploadDoc(t, "s.pdf", "application/pdf", []byte("dummy"))
|
||||
p := processor(e)
|
||||
p.Validator = stub(t, 200, compliantReport)
|
||||
|
||||
if err := p.Validate(context.Background(), docID, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rows := rowsOf(t, e, revID); len(rows) != 0 {
|
||||
t.Fatalf("konformitaet: %d zeilen, want 0", len(rows))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package pdfaconform
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/pdfa"
|
||||
)
|
||||
|
||||
// realValidator spricht den echten veraPDF-Sidecar an (SRC-14, Pruefungen 1-3).
|
||||
func realValidator(t *testing.T) *Validator {
|
||||
t.Helper()
|
||||
addr := os.Getenv("TEST_VERAPDF_ADDR")
|
||||
if addr == "" {
|
||||
t.Skip("TEST_VERAPDF_ADDR nicht gesetzt, Test gegen den echten veraPDF-Sidecar uebersprungen")
|
||||
}
|
||||
v, err := NewValidator(addr, "2b", 60*time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func basePDF() []byte {
|
||||
var b bytes.Buffer
|
||||
offs := map[int]int{}
|
||||
b.WriteString("%PDF-1.5\n%\xde\xad\xbe\xef\n")
|
||||
add := func(nr int, body string) {
|
||||
offs[nr] = b.Len()
|
||||
fmt.Fprintf(&b, "%d 0 obj\n%s\nendobj\n", nr, body)
|
||||
}
|
||||
add(1, "<<\n /Type /Catalog\n /Pages 2 0 R\n>>")
|
||||
add(2, "<<\n /Type /Pages\n /Kids [ 3 0 R ]\n /Count 1\n>>")
|
||||
add(3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 100 100] >>")
|
||||
x := b.Len()
|
||||
b.WriteString("xref\n0 4\n0000000000 65535 f \n")
|
||||
for i := 1; i <= 3; i++ {
|
||||
fmt.Fprintf(&b, "%010d 00000 n \n", offs[i])
|
||||
}
|
||||
fmt.Fprintf(&b, "trailer\n<<\n /Size 4\n /Root 1 0 R\n>>\nstartxref\n%d\n%%%%EOF\n", x)
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func mustCompliantPDF(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
out, err := pdfa.Finalize(basePDF(), pdfa.Meta{Title: "SRC-14 Referenz", Created: time.Now()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Pruefung 1: Referenzsatz aus synthetischen gueltigen PDF/A-2b und mutierten
|
||||
// ungueltigen PDFs liefert je den erwarteten Status.
|
||||
func TestQA_ReferenceSetGetsExpectedStatus(t *testing.T) {
|
||||
v := realValidator(t)
|
||||
ctx := context.Background()
|
||||
|
||||
t.Run("dms-eigenes derivat ist konform", func(t *testing.T) {
|
||||
out := v.Validate(ctx, mustCompliantPDF(t))
|
||||
if out.Status != StatusCompliant {
|
||||
t.Fatalf("status = %q, reason=%q, messages=%+v", out.Status, out.Reason, out.Messages)
|
||||
}
|
||||
if out.ValidatorVersion == "" || out.ReportSHA256 == "" {
|
||||
t.Errorf("version=%q hash=%q leer", out.ValidatorVersion, out.ReportSHA256)
|
||||
}
|
||||
if out.Profile == "" {
|
||||
t.Error("profil leer")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("pdf ohne pdf/a-merkmale ist nicht konform", func(t *testing.T) {
|
||||
out := v.Validate(ctx, basePDF())
|
||||
if out.Status != StatusNonCompliant {
|
||||
t.Fatalf("status = %q, want non_compliant", out.Status)
|
||||
}
|
||||
if len(out.Messages) == 0 {
|
||||
t.Error("erwartet mindestens eine regelverletzung")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("mutiertes derivat mit zerstoerter struktur ist nicht pruefbar", func(t *testing.T) {
|
||||
good := mustCompliantPDF(t)
|
||||
mutated := append([]byte(nil), good...)
|
||||
// xref-Tabelle unbrauchbar machen, damit veraPDF nicht parsen kann.
|
||||
mutated = bytes.Replace(mutated, []byte("\nxref\n"), []byte("\nXXXX\n"), 1)
|
||||
out := v.Validate(ctx, mutated)
|
||||
if out.Status != StatusNotChecked {
|
||||
t.Fatalf("status = %q, want not_checked (reason=%q)", out.Status, out.Reason)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Pruefung 2: Ausfall und Blockade des Dienstes ergeben nie compliant.
|
||||
func TestQA_ServiceFailureIsNeverCompliant(t *testing.T) {
|
||||
realValidator(t) // stellt sicher, dass der Sidecar grundsaetzlich lauffaehig ist
|
||||
ctx := context.Background()
|
||||
pdf := mustCompliantPDF(t)
|
||||
|
||||
t.Run("falsche adresse", func(t *testing.T) {
|
||||
v, err := NewValidator("http://127.0.0.1:1", "2b", 2*time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out := v.Validate(ctx, pdf); out.Status != StatusNotChecked {
|
||||
t.Fatalf("status = %q, want not_checked", out.Status)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("zeitueberschreitung", func(t *testing.T) {
|
||||
v, err := NewValidator(os.Getenv("TEST_VERAPDF_ADDR"), "2b", 1*time.Nanosecond)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out := v.Validate(ctx, pdf); out.Status != StatusNotChecked {
|
||||
t.Fatalf("status = %q, want not_checked", out.Status)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("wiederanlauf ohne neustart der anwendung", func(t *testing.T) {
|
||||
v := realValidator(t)
|
||||
out := v.Validate(ctx, pdf)
|
||||
if out.Status != StatusCompliant {
|
||||
t.Fatalf("status nach fehlversuchen = %q, want compliant", out.Status)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
// Package pdfaconform bindet den veraPDF-Sidecar (SRC-14) an und prueft die
|
||||
// PDF/A-Derivate aus SRC-06 gegen einen anerkannten Validator. Ausfall,
|
||||
// Zeitueberschreitung oder unklare Antwort ergeben immer not_checked, nie
|
||||
// compliant.
|
||||
package pdfaconform
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
StatusCompliant = "compliant"
|
||||
StatusNonCompliant = "non_compliant"
|
||||
StatusNotChecked = "not_checked"
|
||||
|
||||
maxReportBytes = 16 << 20
|
||||
maxKeptMessages = 100
|
||||
)
|
||||
|
||||
// Message ist ein einzelner Regelbefund aus dem veraPDF-Bericht.
|
||||
type Message struct {
|
||||
Clause string `json:"clause"`
|
||||
Rule string `json:"rule"`
|
||||
Text string `json:"text"`
|
||||
Failed int `json:"failed_checks"`
|
||||
}
|
||||
|
||||
// Outcome ist das Ergebnis einer Pruefung.
|
||||
type Outcome struct {
|
||||
Status string
|
||||
Profile string
|
||||
ValidatorVersion string
|
||||
ReportSHA256 string
|
||||
Reason string
|
||||
Messages []Message
|
||||
}
|
||||
|
||||
// Validator spricht den veraPDF-Sidecar (dms/cmd/verapdf-sidecar) an, der
|
||||
// intern je Anfrage einen eigenen veraPDF-Prozess startet.
|
||||
type Validator struct {
|
||||
endpoint string
|
||||
Flavour string
|
||||
client *http.Client
|
||||
}
|
||||
|
||||
// NewValidator liefert nil, wenn addr leer ist (Validierung aus).
|
||||
func NewValidator(addr, flavour string, timeout time.Duration) (*Validator, error) {
|
||||
addr = strings.TrimSpace(addr)
|
||||
if addr == "" {
|
||||
return nil, nil
|
||||
}
|
||||
u, err := url.Parse(addr)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return nil, fmt.Errorf("pdfaconform: sidecar-adresse %q ungueltig (erwartet http(s)://host:port)", addr)
|
||||
}
|
||||
if timeout <= 0 {
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
if strings.TrimSpace(flavour) == "" {
|
||||
flavour = "2b"
|
||||
}
|
||||
return &Validator{endpoint: u.String(), Flavour: flavour, client: &http.Client{
|
||||
Timeout: timeout,
|
||||
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
|
||||
}}, nil
|
||||
}
|
||||
|
||||
func notChecked(reason string) Outcome {
|
||||
return Outcome{Status: StatusNotChecked, Reason: reason}
|
||||
}
|
||||
|
||||
func shortErr(err error) string {
|
||||
s := err.Error()
|
||||
if len(s) > 200 {
|
||||
s = s[:200]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Validate reicht das PDF/A-Derivat an den Sidecar weiter und wertet das
|
||||
// Ergebnis aus. Ein nicht erreichbarer Dienst, eine Zeitueberschreitung, ein
|
||||
// zu grosser Bericht oder eine unbekannte Antwort ergeben not_checked.
|
||||
func (v *Validator) Validate(ctx context.Context, pdf []byte) Outcome {
|
||||
endpoint := v.endpoint + "/validate?flavour=" + url.QueryEscape(v.Flavour)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(pdf))
|
||||
if err != nil {
|
||||
return notChecked("anfrage nicht erstellbar")
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/pdf")
|
||||
resp, err := v.client.Do(req)
|
||||
if err != nil {
|
||||
return notChecked("validator nicht erreichbar: " + shortErr(err))
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, maxReportBytes+1))
|
||||
if err != nil {
|
||||
return notChecked("bericht nicht lesbar: " + shortErr(err))
|
||||
}
|
||||
if len(body) > maxReportBytes {
|
||||
return notChecked("bericht groesser als das Limit")
|
||||
}
|
||||
if resp.StatusCode == http.StatusRequestEntityTooLarge {
|
||||
return notChecked("derivat groesser als das Limit des Sidecars")
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return notChecked(fmt.Sprintf("unerwartete Antwort des Validators (HTTP %d)", resp.StatusCode))
|
||||
}
|
||||
sum := sha256.Sum256(body)
|
||||
reportSHA := hex.EncodeToString(sum[:])
|
||||
|
||||
var rep veraPDFReport
|
||||
if err := json.Unmarshal(body, &rep); err != nil {
|
||||
return notChecked("bericht nicht auswertbar: " + shortErr(err))
|
||||
}
|
||||
version := ""
|
||||
for _, r := range rep.Report.BuildInformation.ReleaseDetails {
|
||||
if r.ID == "apps" {
|
||||
version = r.Version
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(rep.Report.Jobs) != 1 {
|
||||
return Outcome{Status: StatusNotChecked, Reason: "bericht enthaelt nicht genau einen auftrag", ValidatorVersion: version, ReportSHA256: reportSHA}
|
||||
}
|
||||
job := rep.Report.Jobs[0]
|
||||
if job.TaskException != nil {
|
||||
reason := job.TaskException.ExceptionMessage
|
||||
if reason == "" {
|
||||
reason = job.TaskException.Exception
|
||||
}
|
||||
return Outcome{Status: StatusNotChecked, Reason: "veraPDF konnte das Derivat nicht verarbeiten: " + reason, ValidatorVersion: version, ReportSHA256: reportSHA}
|
||||
}
|
||||
if len(job.ValidationResult) != 1 {
|
||||
return Outcome{Status: StatusNotChecked, Reason: "kein eindeutiges Validierungsergebnis im Bericht", ValidatorVersion: version, ReportSHA256: reportSHA}
|
||||
}
|
||||
vr := job.ValidationResult[0]
|
||||
out := Outcome{ValidatorVersion: version, ReportSHA256: reportSHA, Profile: vr.ProfileName}
|
||||
if vr.JobEndStatus != "" && vr.JobEndStatus != "normal" {
|
||||
out.Status = StatusNotChecked
|
||||
out.Reason = "veraPDF-Auftrag endete abweichend: " + vr.JobEndStatus
|
||||
return out
|
||||
}
|
||||
if vr.Compliant {
|
||||
out.Status = StatusCompliant
|
||||
} else {
|
||||
out.Status = StatusNonCompliant
|
||||
}
|
||||
for _, rs := range vr.Details.RuleSummaries {
|
||||
if len(out.Messages) >= maxKeptMessages {
|
||||
break
|
||||
}
|
||||
if rs.RuleStatus != "FAILED" {
|
||||
continue
|
||||
}
|
||||
out.Messages = append(out.Messages, Message{
|
||||
Clause: rs.Clause,
|
||||
Rule: rs.Specification + " " + rs.Clause,
|
||||
Text: truncate(rs.Description, 500),
|
||||
Failed: rs.FailedChecks,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
|
||||
// veraPDFReport bildet nur die Felder ab, die diese Anbindung nutzt.
|
||||
type veraPDFReport struct {
|
||||
Report struct {
|
||||
BuildInformation struct {
|
||||
ReleaseDetails []struct {
|
||||
ID string `json:"id"`
|
||||
Version string `json:"version"`
|
||||
} `json:"releaseDetails"`
|
||||
} `json:"buildInformation"`
|
||||
Jobs []struct {
|
||||
TaskException *struct {
|
||||
Exception string `json:"exception"`
|
||||
ExceptionMessage string `json:"exceptionMessage"`
|
||||
} `json:"taskException"`
|
||||
ValidationResult []struct {
|
||||
Compliant bool `json:"compliant"`
|
||||
ProfileName string `json:"profileName"`
|
||||
JobEndStatus string `json:"jobEndStatus"`
|
||||
Details struct {
|
||||
RuleSummaries []struct {
|
||||
RuleStatus string `json:"ruleStatus"`
|
||||
Specification string `json:"specification"`
|
||||
Clause string `json:"clause"`
|
||||
Description string `json:"description"`
|
||||
FailedChecks int `json:"failedChecks"`
|
||||
} `json:"ruleSummaries"`
|
||||
} `json:"details"`
|
||||
} `json:"validationResult"`
|
||||
} `json:"jobs"`
|
||||
} `json:"report"`
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package pdfaconform
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const compliantReport = `{"report":{"buildInformation":{"releaseDetails":[{"id":"apps","version":"1.30.2"}]},"jobs":[{"itemDetails":{"name":"x.pdf","size":100},"validationResult":[{"compliant":true,"profileName":"PDF/A-2b validation profile","jobEndStatus":"normal","details":{"passedRules":144,"failedRules":0,"ruleSummaries":[]}}],"processingTime":{}}]}}`
|
||||
|
||||
const nonCompliantReport = `{"report":{"buildInformation":{"releaseDetails":[{"id":"apps","version":"1.30.2"}]},"jobs":[{"itemDetails":{"name":"x.pdf","size":100},"validationResult":[{"compliant":false,"profileName":"PDF/A-2b validation profile","jobEndStatus":"normal","details":{"passedRules":142,"failedRules":2,"ruleSummaries":[{"ruleStatus":"FAILED","specification":"ISO 19005-2:2011","clause":"6.1.3","description":"The file trailer dictionary shall contain the ID keyword","failedChecks":1}]}}],"processingTime":{}}]}}`
|
||||
|
||||
const parseFailureReport = `{"report":{"buildInformation":{"releaseDetails":[{"id":"apps","version":"1.30.2"}]},"jobs":[{"itemDetails":{"name":"x.pdf","size":10},"taskException":{"exception":"Couldn't parse stream","exceptionMessage":"can not locate xref table"},"processingTime":{}}]}}`
|
||||
|
||||
func stub(t *testing.T, status int, body string) *Validator {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost || r.URL.Path != "/validate" {
|
||||
t.Errorf("anfrage: %s %s", r.Method, r.URL.Path)
|
||||
}
|
||||
if r.URL.Query().Get("flavour") != "2b" {
|
||||
t.Errorf("profil: %q", r.URL.Query().Get("flavour"))
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_, _ = w.Write([]byte(body))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
v, err := NewValidator(srv.URL, "2b", 2*time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func TestValidate_StatusMapping(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
status int
|
||||
body string
|
||||
want string
|
||||
}{
|
||||
{"konform", 200, compliantReport, StatusCompliant},
|
||||
{"nicht konform", 200, nonCompliantReport, StatusNonCompliant},
|
||||
{"nicht verarbeitbar", 200, parseFailureReport, StatusNotChecked},
|
||||
{"sidecar-fehler", 502, `{"error":"veraPDF-lauf fehlgeschlagen"}`, StatusNotChecked},
|
||||
{"zeitueberschreitung des sidecars", 504, `{"error":"zeituberschreitung"}`, StatusNotChecked},
|
||||
{"zu gross", 413, `{"error":"zu gross"}`, StatusNotChecked},
|
||||
{"kaputtes json", 200, "{not json", StatusNotChecked},
|
||||
{"leerer koerper", 200, "", StatusNotChecked},
|
||||
{"leere job-liste", 200, `{"report":{"buildInformation":{"releaseDetails":[]},"jobs":[]}}`, StatusNotChecked},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
out := stub(t, tc.status, tc.body).Validate(context.Background(), []byte("%PDF-1.4"))
|
||||
if out.Status != tc.want {
|
||||
t.Fatalf("status = %q, want %q (reason=%q)", out.Status, tc.want, out.Reason)
|
||||
}
|
||||
if tc.want != StatusNotChecked && out.Reason != "" {
|
||||
t.Errorf("unerwarteter grund bei %s: %q", tc.want, out.Reason)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate_CompliantCarriesVersionAndHash(t *testing.T) {
|
||||
out := stub(t, 200, compliantReport).Validate(context.Background(), []byte("%PDF-1.4"))
|
||||
if out.ValidatorVersion != "1.30.2" {
|
||||
t.Errorf("validator-version = %q", out.ValidatorVersion)
|
||||
}
|
||||
if out.Profile != "PDF/A-2b validation profile" {
|
||||
t.Errorf("profil = %q", out.Profile)
|
||||
}
|
||||
if out.ReportSHA256 == "" || len(out.ReportSHA256) != 64 {
|
||||
t.Errorf("report-hash = %q", out.ReportSHA256)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate_NonCompliantCarriesMessages(t *testing.T) {
|
||||
out := stub(t, 200, nonCompliantReport).Validate(context.Background(), []byte("%PDF-1.4"))
|
||||
if len(out.Messages) != 1 {
|
||||
t.Fatalf("meldungen = %d, want 1", len(out.Messages))
|
||||
}
|
||||
if out.Messages[0].Clause != "6.1.3" {
|
||||
t.Errorf("klausel = %q", out.Messages[0].Clause)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate_UnreachableSidecar(t *testing.T) {
|
||||
v, err := NewValidator("http://127.0.0.1:1", "2b", 200*time.Millisecond)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out := v.Validate(context.Background(), []byte("%PDF-1.4"))
|
||||
if out.Status != StatusNotChecked {
|
||||
t.Fatalf("status = %q, want not_checked", out.Status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewValidator_EmptyAddrDisables(t *testing.T) {
|
||||
v, err := NewValidator("", "2b", time.Second)
|
||||
if err != nil || v != nil {
|
||||
t.Fatalf("v=%v err=%v, want nil,nil", v, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewValidator_InvalidAddr(t *testing.T) {
|
||||
if _, err := NewValidator("not-a-url", "2b", time.Second); err == nil {
|
||||
t.Fatal("erwartet fehler bei ungueltiger adresse")
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/textcrypt"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -76,11 +77,22 @@ func (s TextStore) TextOf(ctx context.Context, documentID string) (string, error
|
||||
if !Supported(mimeType) {
|
||||
return "", nil
|
||||
}
|
||||
if s.Crypto == nil || s.TenantSlug == "" {
|
||||
return "", textcrypt.ErrNoCrypto
|
||||
}
|
||||
|
||||
var text string
|
||||
err = s.Pool.QueryRow(ctx, `SELECT text FROM revision_text_index WHERE revision_id = $1`, revisionID).Scan(&text)
|
||||
var textEnc, wrapped []byte
|
||||
err = s.Pool.QueryRow(ctx, `SELECT text, text_enc, wrapped_dek FROM revision_text_index WHERE revision_id = $1`, revisionID).Scan(&text, &textEnc, &wrapped)
|
||||
if err == nil {
|
||||
return text, nil
|
||||
if textEnc == nil {
|
||||
return text, nil
|
||||
}
|
||||
dek, err := s.vault().Key(ctx, wrapped)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return textcrypt.Open(dek, textcrypt.AADIndex(revisionID), textEnc)
|
||||
}
|
||||
if !errors.Is(err, pgx.ErrNoRows) {
|
||||
return "", fmt.Errorf("plaintext: ablage lesen: %w", err)
|
||||
@@ -126,12 +138,27 @@ func (s TextStore) read(ctx context.Context, storageKey string, wrappedDEK []byt
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func (s TextStore) vault() textcrypt.Vault {
|
||||
return textcrypt.Vault{Crypto: s.Crypto, TenantSlug: s.TenantSlug}
|
||||
}
|
||||
|
||||
func (s TextStore) store(ctx context.Context, revisionID, status, reason, charset string, sizeBytes int64, text string) error {
|
||||
var sealed, wrapped []byte
|
||||
if text != "" {
|
||||
dek, fresh, err := s.vault().NewKey(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plaintext: %w", err)
|
||||
}
|
||||
if sealed, err = textcrypt.Seal(dek, textcrypt.AADIndex(revisionID), text); err != nil {
|
||||
return err
|
||||
}
|
||||
wrapped = fresh
|
||||
}
|
||||
_, err := s.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_text_index (revision_id, status, reason, charset, source_bytes, indexed_chars, text)
|
||||
VALUES ($1, $2, NULLIF($3, ''), NULLIF($4, ''), $5, $6, $7)
|
||||
INSERT INTO revision_text_index (revision_id, status, reason, charset, source_bytes, indexed_chars, text, text_enc, wrapped_dek)
|
||||
VALUES ($1, $2, NULLIF($3, ''), NULLIF($4, ''), $5, $6, '', $7, $8)
|
||||
ON CONFLICT (revision_id) DO NOTHING
|
||||
`, revisionID, status, reason, charset, sizeBytes, utf8.RuneCountInString(text), text)
|
||||
`, revisionID, status, reason, charset, sizeBytes, utf8.RuneCountInString(text), sealed, wrapped)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plaintext: ablage schreiben: %w", err)
|
||||
}
|
||||
|
||||
@@ -139,7 +139,7 @@ func TestTextOf_IgnoresOtherTypesAndUsesCache(t *testing.T) {
|
||||
if _, err := f.store.TextOf(ctx, docID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.env.Pool.Exec(ctx, `UPDATE revision_text_index SET text = 'aus der ablage' WHERE revision_id = $1`, revID); err != nil {
|
||||
if _, err := f.env.Pool.Exec(ctx, `UPDATE revision_text_index SET text = 'aus der ablage', text_enc = NULL, wrapped_dek = NULL WHERE revision_id = $1`, revID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := f.store.TextOf(ctx, docID); got != "aus der ablage" {
|
||||
@@ -260,3 +260,41 @@ func TestSearch_FindsWordOnlyInPlainTextContent(t *testing.T) {
|
||||
t.Errorf("neuer inhalt nicht auffindbar: %v", ids)
|
||||
}
|
||||
}
|
||||
|
||||
// FDN-22 Pruefung 1: nach dem Ablegen liegt kein Klartext in der Tabelle, weder in der Textspalte noch im Chiffretext.
|
||||
func TestTextIsStoredEncrypted(t *testing.T) {
|
||||
f := newFixture(t, 0, 0)
|
||||
ctx := context.Background()
|
||||
docID, revID := f.upload(t, "vertrauliches.txt", "text/plain", []byte("Zebrafinken bauen Lehmziegel"))
|
||||
if got, err := f.store.TextOf(ctx, docID); err != nil || got != "Zebrafinken bauen Lehmziegel" {
|
||||
t.Fatalf("textof: %q %v", got, err)
|
||||
}
|
||||
var text string
|
||||
var enc, wrapped []byte
|
||||
if err := f.env.Pool.QueryRow(ctx, `SELECT text, text_enc, wrapped_dek FROM revision_text_index WHERE revision_id = $1`, revID).Scan(&text, &enc, &wrapped); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if text != "" || len(enc) == 0 || len(wrapped) == 0 || strings.Contains(string(enc), "Zebrafinken") {
|
||||
t.Fatalf("text=%q enc=%d wrapped=%d", text, len(enc), len(wrapped))
|
||||
}
|
||||
// zweiter Aufruf liest aus der Ablage und entschluesselt
|
||||
if got, err := f.store.TextOf(ctx, docID); err != nil || got != "Zebrafinken bauen Lehmziegel" {
|
||||
t.Fatalf("aus ablage: %q %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ohne Schluesseldienst wird nichts im Klartext abgelegt oder geliefert.
|
||||
func TestTextStoreWithoutCryptoFailsClosed(t *testing.T) {
|
||||
f := newFixture(t, 0, 0)
|
||||
docID, revID := f.upload(t, "x.txt", "text/plain", []byte("Zebrafinken"))
|
||||
blind := f.store
|
||||
blind.Crypto = nil
|
||||
if got, err := blind.TextOf(context.Background(), docID); err == nil || got != "" {
|
||||
t.Fatalf("ohne dienst: %q %v", got, err)
|
||||
}
|
||||
var n int
|
||||
_ = f.env.Pool.QueryRow(context.Background(), `SELECT count(*) FROM revision_text_index WHERE revision_id = $1`, revID).Scan(&n)
|
||||
if n != 0 {
|
||||
t.Fatalf("ohne dienst wurde eine zeile abgelegt: %d", n)
|
||||
}
|
||||
}
|
||||
@@ -187,7 +187,7 @@ func newFixture(t *testing.T, perms searchapi.PermissionChecker) *fixture {
|
||||
}
|
||||
|
||||
f := &fixture{env: e, index: index, docIDs: map[string]string{}, nameByID: map[string]string{}, accuracy: map[string]float64{}}
|
||||
f.syncer = &searchsync.Syncer{Pool: e.Pool, Index: index, Text: ocr.TextStore{Pool: e.Pool}}
|
||||
f.syncer = &searchsync.Syncer{Pool: e.Pool, Index: index, Text: ocr.TextStore{Pool: e.Pool, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug}}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
searchapi.NewServer(index, perms).RegisterRoutes(mux)
|
||||
@@ -236,7 +236,7 @@ func newFixture(t *testing.T, perms searchapi.PermissionChecker) *fixture {
|
||||
break
|
||||
}
|
||||
}
|
||||
pages, err := ocr.Pages(ctx, e.Pool, revID)
|
||||
pages, err := ocr.Pages(ctx, e.Pool, e.Crypto, testenv.TenantSlug, revID)
|
||||
if err != nil || len(pages) != 1 {
|
||||
t.Fatalf("%s: seiten = %v, %v", d.Name, pages, err)
|
||||
}
|
||||
|
||||
@@ -120,7 +120,7 @@ func TestClassificationAccuracyOnOCRScanset(t *testing.T) {
|
||||
eng.DPI = 200
|
||||
proc := &ocr.Processor{Pool: e.Pool, Storage: e.Storage, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug, Queue: queue, Engine: eng, WorkerName: "qa05",
|
||||
OnText: func(ctx context.Context, documentID, _ string) error {
|
||||
_, err := autorule.Evaluate(ctx, e.Pool, ocr.TextStore{Pool: e.Pool}, documentID)
|
||||
_, err := autorule.Evaluate(ctx, e.Pool, ocr.TextStore{Pool: e.Pool, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug}, documentID)
|
||||
return err
|
||||
}}
|
||||
worker := jobqueue.NewWorker(queue, "qa05", ocr.JobTypes, time.Millisecond, proc.Handle)
|
||||
@@ -176,7 +176,7 @@ func TestClassificationAccuracyOnOCRScanset(t *testing.T) {
|
||||
if predicted == s.want {
|
||||
correct++
|
||||
} else {
|
||||
pages, _ := ocr.Pages(ctx, e.Pool, docs[i].rev)
|
||||
pages, _ := ocr.Pages(ctx, e.Pool, e.Crypto, testenv.TenantSlug, docs[i].rev)
|
||||
txt := ""
|
||||
if len(pages) > 0 {
|
||||
txt = pages[0].Text
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
// Package sharelink implementiert DOC-22: zeitlich begrenzte, widerrufbare
|
||||
// Freigabelinks fuer ein einzelnes Dokument ohne DMS-Konto. Ein Link ist die
|
||||
// kontrollierte Ausnahme von "nur mit Konto" -- deshalb Pflichtablauf,
|
||||
// optionales Passwort (nur als bcrypt-Hash gespeichert), Sperre nach
|
||||
// wiederholten Fehlversuchen und ein hochentropischer Token, der selbst nur
|
||||
// als SHA-256-Hash in der Datenbank liegt.
|
||||
package sharelink
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrNotFound = errors.New("sharelink: link nicht gefunden, abgelaufen, widerrufen oder ausgeschoepft")
|
||||
ErrLocked = errors.New("sharelink: zu viele fehlversuche, vorruebergehend gesperrt")
|
||||
ErrNeedPassword = errors.New("sharelink: passwort erforderlich")
|
||||
ErrWrongPassword = errors.New("sharelink: falsches passwort")
|
||||
ErrTTLTooLong = errors.New("sharelink: gueltigkeitsdauer ueberschreitet die erlaubte obergrenze")
|
||||
)
|
||||
|
||||
// Link ist die oeffentlich sichtbare Beschreibung eines Freigabelinks (ohne
|
||||
// Token/Passwort-Hash -- die duerfen die Ausgabe nie verlassen).
|
||||
type Link struct {
|
||||
ID string
|
||||
DocumentID string
|
||||
CreatedBy string
|
||||
CreatedAt time.Time
|
||||
ExpiresAt time.Time
|
||||
DownloadLimit *int
|
||||
DownloadCount int
|
||||
HasPassword bool
|
||||
RevokedAt *time.Time
|
||||
}
|
||||
|
||||
func hashToken(token string) []byte {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
return sum[:]
|
||||
}
|
||||
|
||||
// generateToken erzeugt einen hochentropischen (256 Bit) Token; die
|
||||
// URL-taugliche Kodierung macht ihn direkt als Pfadsegment nutzbar.
|
||||
func generateToken() (string, error) {
|
||||
raw := make([]byte, 32)
|
||||
if _, err := rand.Read(raw); err != nil {
|
||||
return "", fmt.Errorf("sharelink: token erzeugen: %w", err)
|
||||
}
|
||||
return base64.RawURLEncoding.EncodeToString(raw), nil
|
||||
}
|
||||
|
||||
// Create legt einen neuen Freigabelink an (Akzeptanzkriterium 1). ttl darf
|
||||
// maxTTL nicht ueberschreiten -- die Obergrenze ist Betriebsvorgabe, nicht
|
||||
// pro Aufruf aushebelbar. Der Klartext-Token wird nur hier, beim Anlegen,
|
||||
// einmalig zurueckgegeben.
|
||||
func Create(ctx context.Context, pool *pgxpool.Pool, documentID, createdBy string, ttl, maxTTL time.Duration, password string, downloadLimit *int) (token string, link Link, err error) {
|
||||
if ttl <= 0 || ttl > maxTTL {
|
||||
return "", Link{}, ErrTTLTooLong
|
||||
}
|
||||
token, err = generateToken()
|
||||
if err != nil {
|
||||
return "", Link{}, err
|
||||
}
|
||||
var passwordHash *string
|
||||
if password != "" {
|
||||
h, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
|
||||
if err != nil {
|
||||
return "", Link{}, fmt.Errorf("sharelink: passwort hashen: %w", err)
|
||||
}
|
||||
s := string(h)
|
||||
passwordHash = &s
|
||||
}
|
||||
expiresAt := time.Now().Add(ttl)
|
||||
err = pool.QueryRow(ctx, `
|
||||
INSERT INTO share_links (document_id, token_hash, password_hash, download_limit, created_by, expires_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
RETURNING id, created_at
|
||||
`, documentID, hashToken(token), passwordHash, downloadLimit, createdBy, expiresAt).Scan(&link.ID, &link.CreatedAt)
|
||||
if err != nil {
|
||||
return "", Link{}, fmt.Errorf("sharelink: anlegen: %w", err)
|
||||
}
|
||||
link.DocumentID = documentID
|
||||
link.CreatedBy = createdBy
|
||||
link.ExpiresAt = expiresAt
|
||||
link.DownloadLimit = downloadLimit
|
||||
link.HasPassword = passwordHash != nil
|
||||
return token, link, nil
|
||||
}
|
||||
|
||||
// List liefert die Freigabelinks eines Dokuments (ohne Token/Passwort-Hash).
|
||||
func List(ctx context.Context, pool *pgxpool.Pool, documentID string) ([]Link, error) {
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT id, created_by, created_at, expires_at, download_limit, download_count,
|
||||
password_hash IS NOT NULL, revoked_at
|
||||
FROM share_links WHERE document_id = $1 ORDER BY created_at DESC
|
||||
`, documentID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("sharelink: liste lesen: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []Link
|
||||
for rows.Next() {
|
||||
var l Link
|
||||
if err := rows.Scan(&l.ID, &l.CreatedBy, &l.CreatedAt, &l.ExpiresAt, &l.DownloadLimit, &l.DownloadCount, &l.HasPassword, &l.RevokedAt); err != nil {
|
||||
return nil, fmt.Errorf("sharelink: zeile lesen: %w", err)
|
||||
}
|
||||
l.DocumentID = documentID
|
||||
out = append(out, l)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// Revoke widerruft einen Link sofort (Akzeptanzkriterium 1). Ein bereits
|
||||
// widerrufener oder fremder Link gilt als nicht gefunden -- idempotent und
|
||||
// ohne Information ueber fremde Dokumente preiszugeben.
|
||||
func Revoke(ctx context.Context, pool *pgxpool.Pool, documentID, linkID string) error {
|
||||
tag, err := pool.Exec(ctx, `
|
||||
UPDATE share_links SET revoked_at = now()
|
||||
WHERE id = $1 AND document_id = $2 AND revoked_at IS NULL
|
||||
`, linkID, documentID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("sharelink: widerrufen: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type row struct {
|
||||
id string
|
||||
documentID string
|
||||
passwordHash *string
|
||||
downloadLimit *int
|
||||
downloadCount int
|
||||
failedAttempts int
|
||||
lockedUntil *time.Time
|
||||
expiresAt time.Time
|
||||
revokedAt *time.Time
|
||||
}
|
||||
|
||||
func lookup(ctx context.Context, pool *pgxpool.Pool, token string) (row, error) {
|
||||
var r row
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT id, document_id, password_hash, download_limit, download_count,
|
||||
failed_attempts, locked_until, expires_at, revoked_at
|
||||
FROM share_links WHERE token_hash = $1
|
||||
`, hashToken(token)).Scan(&r.id, &r.documentID, &r.passwordHash, &r.downloadLimit, &r.downloadCount,
|
||||
&r.failedAttempts, &r.lockedUntil, &r.expiresAt, &r.revokedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return row{}, ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return row{}, fmt.Errorf("sharelink: token nachschlagen: %w", err)
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func (r row) usable() bool {
|
||||
if r.revokedAt != nil {
|
||||
return false
|
||||
}
|
||||
if time.Now().After(r.expiresAt) {
|
||||
return false
|
||||
}
|
||||
if r.downloadLimit != nil && r.downloadCount >= *r.downloadLimit {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Resolve prueft Token und (falls gesetzt) Passwort gegen den gespeicherten
|
||||
// Stand und liefert bei Erfolg die Dokument-ID; bei jedem sonstigen Ausgang
|
||||
// ist der Fehler einer der exportierten Sentinel-Fehler dieses Pakets
|
||||
// (Akzeptanzkriterium 2/3: kein unterscheidbarer Ausgang zwischen
|
||||
// unbekanntem, abgelaufenem, widerrufenem oder ausgeschoepftem Link).
|
||||
func Resolve(ctx context.Context, pool *pgxpool.Pool, token, password string, maxAttempts int, lockDuration time.Duration) (documentID, linkID string, err error) {
|
||||
r, err := lookup(ctx, pool, token)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
if !r.usable() {
|
||||
return "", "", ErrNotFound
|
||||
}
|
||||
if r.passwordHash == nil {
|
||||
return r.documentID, r.id, nil
|
||||
}
|
||||
if r.lockedUntil != nil && time.Now().Before(*r.lockedUntil) {
|
||||
return "", "", ErrLocked
|
||||
}
|
||||
if password == "" {
|
||||
return "", "", ErrNeedPassword
|
||||
}
|
||||
if err := bcrypt.CompareHashAndPassword([]byte(*r.passwordHash), []byte(password)); err != nil {
|
||||
locked, ferr := recordFailure(ctx, pool, r.id, maxAttempts, lockDuration)
|
||||
if ferr != nil {
|
||||
return "", "", ferr
|
||||
}
|
||||
if locked {
|
||||
return "", "", ErrLocked
|
||||
}
|
||||
return "", "", ErrWrongPassword
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `UPDATE share_links SET failed_attempts = 0 WHERE id = $1`, r.id); err != nil {
|
||||
return "", "", fmt.Errorf("sharelink: fehlversuche zuruecksetzen: %w", err)
|
||||
}
|
||||
return r.documentID, r.id, nil
|
||||
}
|
||||
|
||||
func recordFailure(ctx context.Context, pool *pgxpool.Pool, linkID string, maxAttempts int, lockDuration time.Duration) (locked bool, err error) {
|
||||
err = pool.QueryRow(ctx, `
|
||||
UPDATE share_links SET
|
||||
failed_attempts = failed_attempts + 1,
|
||||
locked_until = CASE WHEN failed_attempts + 1 >= $2 THEN now() + $3::interval ELSE locked_until END
|
||||
WHERE id = $1
|
||||
RETURNING failed_attempts >= $2
|
||||
`, linkID, maxAttempts, lockDuration.String()).Scan(&locked)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("sharelink: fehlversuch zaehlen: %w", err)
|
||||
}
|
||||
return locked, nil
|
||||
}
|
||||
|
||||
// CountDownload erhoeht den Zaehler atomar und nur, wenn das Limit (falls
|
||||
// gesetzt) dadurch nicht ueberschritten wird -- verhindert, dass parallele
|
||||
// Anfragen ein Download-Limit umgehen.
|
||||
func CountDownload(ctx context.Context, pool *pgxpool.Pool, linkID string) error {
|
||||
tag, err := pool.Exec(ctx, `
|
||||
UPDATE share_links SET download_count = download_count + 1
|
||||
WHERE id = $1 AND (download_limit IS NULL OR download_count < download_limit)
|
||||
`, linkID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("sharelink: zugriff zaehlen: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package sharelink
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
var docSeq int
|
||||
|
||||
func newDoc(t *testing.T, e *testenv.Env) string {
|
||||
t.Helper()
|
||||
docSeq++
|
||||
docID, _ := e.UploadDoc(t, "beleg.txt", "text/plain", []byte(fmt.Sprintf("inhalt %d %d", docSeq, time.Now().UnixNano())))
|
||||
return docID
|
||||
}
|
||||
|
||||
// Akzeptanzkriterium 1: Erzeugen mit Pflichtablauf, hochentropischem Token
|
||||
// (nur als Hash gespeichert) und sofortiges Widerrufen.
|
||||
func TestCreate_TokenIsHighEntropyAndOnlyHashIsStored(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
|
||||
token, link, err := Create(context.Background(), e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", nil)
|
||||
if err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
if len(token) < 32 {
|
||||
t.Fatalf("token zu kurz fuer 256 bit entropie: %d zeichen", len(token))
|
||||
}
|
||||
var raw string
|
||||
if err := e.Pool.QueryRow(context.Background(), `SELECT encode(token_hash,'hex') FROM share_links WHERE id = $1`, link.ID).Scan(&raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if raw == "" {
|
||||
t.Fatal("token_hash leer")
|
||||
}
|
||||
|
||||
// Alle Klartext-Spalten der Tabelle enthalten nie den Token selbst.
|
||||
rows, err := e.Pool.Query(context.Background(), `SELECT * FROM share_links WHERE id = $1`, link.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
fields := rows.FieldDescriptions()
|
||||
for rows.Next() {
|
||||
vals, err := rows.Values()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i, v := range vals {
|
||||
if s, ok := v.(string); ok && s == token {
|
||||
t.Fatalf("klartext-token in spalte %q gefunden", fields[i].Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreate_RejectsTTLAboveMax(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
if _, _, err := Create(context.Background(), e.Pool, docID, e.UserID, 48*time.Hour, 24*time.Hour, "", nil); !errors.Is(err, ErrTTLTooLong) {
|
||||
t.Fatalf("erwartet ErrTTLTooLong, habe %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolve_WithoutPassword(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
token, _, err := Create(context.Background(), e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, _, err := Resolve(context.Background(), e.Pool, token, "", 5, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve: %v", err)
|
||||
}
|
||||
if got != docID {
|
||||
t.Errorf("dokument-id = %q, erwartet %q", got, docID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolve_WrongToken_GenericNotFound(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
if _, _, err := Resolve(context.Background(), e.Pool, "erratener-token-existiert-nicht", "", 5, time.Minute); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Akzeptanzkriterium 2/Pruefung 1: Passwortschutz, falsches Passwort und
|
||||
// Sperre nach zu vielen Fehlversuchen.
|
||||
func TestResolve_PasswordProtected(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
token, _, err := Create(context.Background(), e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "geheim123", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if _, _, err := Resolve(context.Background(), e.Pool, token, "", 5, time.Minute); !errors.Is(err, ErrNeedPassword) {
|
||||
t.Fatalf("ohne passwort: erwartet ErrNeedPassword, habe %v", err)
|
||||
}
|
||||
if _, _, err := Resolve(context.Background(), e.Pool, token, "falsch", 5, time.Minute); !errors.Is(err, ErrWrongPassword) {
|
||||
t.Fatalf("falsches passwort: erwartet ErrWrongPassword, habe %v", err)
|
||||
}
|
||||
got, _, err := Resolve(context.Background(), e.Pool, token, "geheim123", 5, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("richtiges passwort: %v", err)
|
||||
}
|
||||
if got != docID {
|
||||
t.Errorf("dokument-id = %q, erwartet %q", got, docID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolve_LocksAfterMaxAttempts(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
token, _, err := Create(context.Background(), e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "geheim123", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
const maxAttempts = 3
|
||||
for i := 0; i < maxAttempts-1; i++ {
|
||||
if _, _, err := Resolve(context.Background(), e.Pool, token, "falsch", maxAttempts, time.Minute); !errors.Is(err, ErrWrongPassword) {
|
||||
t.Fatalf("versuch %d: erwartet ErrWrongPassword, habe %v", i, err)
|
||||
}
|
||||
}
|
||||
// Der Versuch, der die Grenze erreicht, sperrt sofort.
|
||||
if _, _, err := Resolve(context.Background(), e.Pool, token, "falsch", maxAttempts, time.Minute); !errors.Is(err, ErrLocked) {
|
||||
t.Fatalf("erwartet ErrLocked, habe %v", err)
|
||||
}
|
||||
// Waehrend der Sperre wird selbst das richtige Passwort abgelehnt.
|
||||
if _, _, err := Resolve(context.Background(), e.Pool, token, "geheim123", maxAttempts, time.Minute); !errors.Is(err, ErrLocked) {
|
||||
t.Fatalf("gesperrt, richtiges passwort: erwartet ErrLocked, habe %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolve_ExpiredAndRevokedAndLimitReached(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
ctx := context.Background()
|
||||
|
||||
expiredToken, expiredLink, err := Create(ctx, e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE share_links SET expires_at = now() - interval '1 second' WHERE id = $1`, expiredLink.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := Resolve(ctx, e.Pool, expiredToken, "", 5, time.Minute); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("abgelaufen: erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
|
||||
revokedToken, revokedLink, err := Create(ctx, e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := Revoke(ctx, e.Pool, docID, revokedLink.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := Resolve(ctx, e.Pool, revokedToken, "", 5, time.Minute); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("widerrufen: erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
|
||||
limit := 1
|
||||
limitedToken, limitedLink, err := Create(ctx, e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", &limit)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := Resolve(ctx, e.Pool, limitedToken, "", 5, time.Minute); err != nil {
|
||||
t.Fatalf("erster abruf: %v", err)
|
||||
}
|
||||
if err := CountDownload(ctx, e.Pool, limitedLink.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := Resolve(ctx, e.Pool, limitedToken, "", 5, time.Minute); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("limit erreicht: erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRevoke_UnknownOrForeignLink(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
otherDocID := newDoc(t, e)
|
||||
_, link, err := Create(context.Background(), e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := Revoke(context.Background(), e.Pool, otherDocID, link.ID); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("fremdes dokument: erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
if err := Revoke(context.Background(), e.Pool, "00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000000"); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("unbekannt: erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCountDownload_NeverExceedsLimitUnderConcurrency(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID := newDoc(t, e)
|
||||
limit := 5
|
||||
_, link, err := Create(context.Background(), e.Pool, docID, e.UserID, time.Hour, 24*time.Hour, "", &limit)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
errs := make(chan error, 20)
|
||||
for i := 0; i < 20; i++ {
|
||||
go func() { errs <- CountDownload(context.Background(), e.Pool, link.ID) }()
|
||||
}
|
||||
ok := 0
|
||||
for i := 0; i < 20; i++ {
|
||||
if <-errs == nil {
|
||||
ok++
|
||||
}
|
||||
}
|
||||
if ok != limit {
|
||||
t.Errorf("erfolgreiche zaehlungen = %d, erwartet genau %d", ok, limit)
|
||||
}
|
||||
var count int
|
||||
if err := e.Pool.QueryRow(context.Background(), `SELECT download_count FROM share_links WHERE id = $1`, link.ID).Scan(&count); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if count != limit {
|
||||
t.Errorf("download_count = %d, erwartet %d", count, limit)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,322 @@
|
||||
// Package sharelinkapi haengt DOC-22 (externe Freigabelinks) ein: die
|
||||
// verwaltenden Endpunkte unter /documents/{id}/share-links laufen wie alle
|
||||
// dokument-eigenen Routen durch authz.Guard (nur Eigentuemer, Identitaet
|
||||
// zwingend); der Abrufpfad /share/{token} ist bewusst NICHT unter
|
||||
// /documents registriert und in der OpenAPI-Beschreibung als "security": []
|
||||
// markiert -- er ist die kontrollierte Ausnahme von "nur mit Konto".
|
||||
package sharelinkapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/authz"
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/sharelink"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
)
|
||||
|
||||
// ShareLinkActor steht im Audit-Log fuer Zugriffe ueber den oeffentlichen
|
||||
// Pfad -- die Anfrage traegt keine Benutzeridentitaet (analog zu
|
||||
// downloadapi.SignedURLActor).
|
||||
const ShareLinkActor = "share-link"
|
||||
|
||||
const visibleDocument = `d.trashed_at IS NULL AND d.deleted_at IS NULL
|
||||
AND NOT EXISTS (SELECT 1 FROM revision_scans rs WHERE rs.revision_id = fr.id AND rs.status <> 'clean')`
|
||||
|
||||
type Server struct {
|
||||
pool *pgxpool.Pool
|
||||
storageSvc *storage.Service
|
||||
cryptoSvc *dmscrypto.Service
|
||||
tenantSlug string
|
||||
enabled bool
|
||||
defaultTTL time.Duration
|
||||
maxTTL time.Duration
|
||||
maxAttempts int
|
||||
lockDuration time.Duration
|
||||
pathPrefix string
|
||||
}
|
||||
|
||||
func New(pool *pgxpool.Pool, storageSvc *storage.Service, cryptoSvc *dmscrypto.Service, tenantSlug string, enabled bool, defaultTTL, maxTTL time.Duration, maxAttempts int, lockDuration time.Duration) *Server {
|
||||
if defaultTTL <= 0 || defaultTTL > maxTTL {
|
||||
defaultTTL = maxTTL
|
||||
}
|
||||
if maxAttempts <= 0 {
|
||||
maxAttempts = 5
|
||||
}
|
||||
if lockDuration <= 0 {
|
||||
lockDuration = 15 * time.Minute
|
||||
}
|
||||
return &Server{pool: pool, storageSvc: storageSvc, cryptoSvc: cryptoSvc, tenantSlug: tenantSlug, enabled: enabled,
|
||||
defaultTTL: defaultTTL, maxTTL: maxTTL, maxAttempts: maxAttempts, lockDuration: lockDuration}
|
||||
}
|
||||
|
||||
// SetPathPrefix stellt den Routenpraefix (FDN-14) den ausgestellten
|
||||
// Freigabelink-URLs voran.
|
||||
func (s *Server) SetPathPrefix(prefix string) { s.pathPrefix = prefix }
|
||||
|
||||
func (s *Server) RegisterRoutes(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /documents/{id}/share-links", s.handleCreate)
|
||||
mux.HandleFunc("GET /documents/{id}/share-links", s.handleList)
|
||||
mux.HandleFunc("DELETE /documents/{id}/share-links/{linkID}", s.handleRevoke)
|
||||
mux.HandleFunc("GET /share/{token}", s.handleAccess)
|
||||
mux.HandleFunc("POST /share/{token}", s.handleAccess)
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, code, message string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"error": map[string]string{"code": code, "message": message}})
|
||||
}
|
||||
|
||||
func caller(r *http.Request) string {
|
||||
c, _ := authz.Identity(r)
|
||||
return c
|
||||
}
|
||||
|
||||
type createRequest struct {
|
||||
TTLSeconds *int `json:"ttl_seconds"`
|
||||
Password string `json:"password"`
|
||||
DownloadLimit *int `json:"download_limit"`
|
||||
}
|
||||
|
||||
type linkDTO struct {
|
||||
ID string `json:"id"`
|
||||
URL string `json:"url,omitempty"`
|
||||
ExpiresAt string `json:"expires_at"`
|
||||
DownloadLimit *int `json:"download_limit,omitempty"`
|
||||
DownloadCount int `json:"download_count"`
|
||||
HasPassword bool `json:"has_password"`
|
||||
RevokedAt *string `json:"revoked_at,omitempty"`
|
||||
}
|
||||
|
||||
func toDTO(l sharelink.Link) linkDTO {
|
||||
d := linkDTO{ID: l.ID, ExpiresAt: l.ExpiresAt.UTC().Format(time.RFC3339), DownloadLimit: l.DownloadLimit,
|
||||
DownloadCount: l.DownloadCount, HasPassword: l.HasPassword}
|
||||
if l.RevokedAt != nil {
|
||||
s := l.RevokedAt.UTC().Format(time.RFC3339)
|
||||
d.RevokedAt = &s
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func (s *Server) handleCreate(w http.ResponseWriter, r *http.Request) {
|
||||
if !s.enabled {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "freigabelinks sind fuer diesen mandanten deaktiviert")
|
||||
return
|
||||
}
|
||||
documentID := r.PathValue("id")
|
||||
c := caller(r)
|
||||
if c == "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "fehlende identitaet")
|
||||
return
|
||||
}
|
||||
|
||||
var req createRequest
|
||||
if r.ContentLength != 0 {
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "ungueltiger anfragekoerper")
|
||||
return
|
||||
}
|
||||
}
|
||||
ttl := s.defaultTTL
|
||||
if req.TTLSeconds != nil {
|
||||
ttl = time.Duration(*req.TTLSeconds) * time.Second
|
||||
}
|
||||
if req.DownloadLimit != nil && *req.DownloadLimit <= 0 {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "download_limit muss positiv sein")
|
||||
return
|
||||
}
|
||||
|
||||
token, link, err := sharelink.Create(r.Context(), s.pool, documentID, c, ttl, s.maxTTL, req.Password, req.DownloadLimit)
|
||||
if errors.Is(err, sharelink.ErrTTLTooLong) {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "gueltigkeitsdauer ueberschreitet die erlaubte obergrenze")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if err := audittrail.Record(r.Context(), s.pool, c, "share_link_created", "document", documentID, map[string]any{
|
||||
"share_link_id": link.ID, "expires_at": link.ExpiresAt.UTC().Format(time.RFC3339), "has_password": link.HasPassword,
|
||||
}); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
dto := toDTO(link)
|
||||
dto.URL = fmt.Sprintf("%s/share/%s", s.pathPrefix, url.PathEscape(token))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
_ = json.NewEncoder(w).Encode(dto)
|
||||
}
|
||||
|
||||
func (s *Server) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
documentID := r.PathValue("id")
|
||||
links, err := sharelink.List(r.Context(), s.pool, documentID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]linkDTO, 0, len(links))
|
||||
for _, l := range links {
|
||||
out = append(out, toDTO(l))
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(out)
|
||||
}
|
||||
|
||||
func validUUID(s string) bool {
|
||||
if len(s) != 36 {
|
||||
return false
|
||||
}
|
||||
for i, r := range s {
|
||||
switch i {
|
||||
case 8, 13, 18, 23:
|
||||
if r != '-' {
|
||||
return false
|
||||
}
|
||||
default:
|
||||
if !((r >= '0' && r <= '9') || (r >= 'a' && r <= 'f') || (r >= 'A' && r <= 'F')) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *Server) handleRevoke(w http.ResponseWriter, r *http.Request) {
|
||||
documentID, linkID := r.PathValue("id"), r.PathValue("linkID")
|
||||
if !validUUID(linkID) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "freigabelink nicht gefunden")
|
||||
return
|
||||
}
|
||||
c := caller(r)
|
||||
err := sharelink.Revoke(r.Context(), s.pool, documentID, linkID)
|
||||
if errors.Is(err, sharelink.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "freigabelink nicht gefunden")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
if err := audittrail.Record(r.Context(), s.pool, c, "share_link_revoked", "document", documentID, map[string]any{"share_link_id": linkID}); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func passwordFrom(r *http.Request) string {
|
||||
if r.Method != http.MethodPost {
|
||||
return ""
|
||||
}
|
||||
var body struct {
|
||||
Password string `json:"password"`
|
||||
}
|
||||
if r.ContentLength == 0 {
|
||||
return ""
|
||||
}
|
||||
_ = json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&body)
|
||||
return body.Password
|
||||
}
|
||||
|
||||
// handleAccess ist der unauthentifizierte Abrufpfad (Akzeptanzkriterium 2).
|
||||
// Jeder nicht nutzbare Ausgang (unbekannt, abgelaufen, widerrufen,
|
||||
// Limit erreicht) liefert denselben generischen 404 -- keine
|
||||
// Unterscheidung, die Rueckschluesse auf einen erratenen Token zuliesse.
|
||||
func (s *Server) handleAccess(w http.ResponseWriter, r *http.Request) {
|
||||
token := r.PathValue("token")
|
||||
password := passwordFrom(r)
|
||||
|
||||
documentID, linkID, err := sharelink.Resolve(r.Context(), s.pool, token, password, s.maxAttempts, s.lockDuration)
|
||||
switch {
|
||||
case errors.Is(err, sharelink.ErrNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "freigabelink nicht gefunden")
|
||||
return
|
||||
case errors.Is(err, sharelink.ErrLocked):
|
||||
writeError(w, http.StatusLocked, "locked", "zu viele fehlversuche, bitte spaeter erneut versuchen")
|
||||
return
|
||||
case errors.Is(err, sharelink.ErrNeedPassword):
|
||||
writeError(w, http.StatusUnauthorized, "password_required", "passwort erforderlich")
|
||||
return
|
||||
case errors.Is(err, sharelink.ErrWrongPassword):
|
||||
_ = audittrail.Record(r.Context(), s.pool, ShareLinkActor, "share_link_access_denied", "document", "", map[string]any{"reason": "wrong_password"})
|
||||
writeError(w, http.StatusUnauthorized, "invalid_password", "falsches passwort")
|
||||
return
|
||||
case err != nil:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var revisionID, mimeType, storageKey, title string
|
||||
var wrappedDEK []byte
|
||||
err = s.pool.QueryRow(r.Context(), `
|
||||
SELECT fr.id, fr.mime_type, fr.storage_key, fr.wrapped_dek, d.title
|
||||
FROM documents d JOIN file_revisions fr ON fr.id = d.current_revision_id
|
||||
WHERE d.id = $1 AND `+visibleDocument+`
|
||||
`, documentID).Scan(&revisionID, &mimeType, &storageKey, &wrappedDEK, &title)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "freigabelink nicht gefunden")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
if err := sharelink.CountDownload(r.Context(), s.pool, linkID); err != nil {
|
||||
writeError(w, http.StatusNotFound, "not_found", "freigabelink nicht gefunden")
|
||||
return
|
||||
}
|
||||
// Vor der Auslieferung protokollieren: ohne Eintrag kein Zugriff.
|
||||
if err := audittrail.Record(r.Context(), s.pool, ShareLinkActor, "share_link_accessed", "document", documentID, map[string]any{"share_link_id": linkID, "revision_id": revisionID}); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
cipherReader, err := s.storageSvc.Get(r.Context(), storageKey)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "original nicht ladbar: "+err.Error())
|
||||
return
|
||||
}
|
||||
defer func() { _ = cipherReader.Close() }()
|
||||
plainReader, err := s.cryptoSvc.Open(r.Context(), s.tenantSlug, wrappedDEK, cipherReader)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "original nicht entschluesselbar: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", mimeType)
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
if title == "" {
|
||||
title = revisionID
|
||||
}
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf(`attachment; filename="%s"; filename*=UTF-8''%s`,
|
||||
toASCIIFallback(title), url.PathEscape(title)))
|
||||
_, _ = io.Copy(w, plainReader)
|
||||
}
|
||||
|
||||
func toASCIIFallback(s string) string {
|
||||
var b strings.Builder
|
||||
for _, r := range s {
|
||||
if r >= 32 && r < 127 && r != '"' && r != '\\' {
|
||||
b.WriteRune(r)
|
||||
} else {
|
||||
b.WriteByte('_')
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,288 @@
|
||||
package sharelinkapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
func newServer(e *testenv.Env, enabled bool, maxAttempts int, lockDuration time.Duration) (*Server, *httptest.Server) {
|
||||
srv := New(e.Pool, e.Storage, e.Crypto, testenv.TenantSlug, enabled, time.Hour, 24*time.Hour, maxAttempts, lockDuration)
|
||||
mux := http.NewServeMux()
|
||||
srv.RegisterRoutes(mux)
|
||||
ts := httptest.NewServer(mux)
|
||||
srv.SetPathPrefix(ts.URL)
|
||||
return srv, ts
|
||||
}
|
||||
|
||||
func createLink(t *testing.T, ts *httptest.Server, docID, callerID, password string, downloadLimit *int) (id, token string) {
|
||||
t.Helper()
|
||||
body, _ := json.Marshal(map[string]any{"password": password, "download_limit": downloadLimit})
|
||||
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/documents/"+docID+"/share-links", bytes.NewReader(body))
|
||||
req.Header.Set("X-User-Id", callerID)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
raw, _ := io.ReadAll(resp.Body)
|
||||
t.Fatalf("create: status %d, body %s", resp.StatusCode, raw)
|
||||
}
|
||||
var dto struct {
|
||||
ID string `json:"id"`
|
||||
URL string `json:"url"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&dto); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// URL hat die Form "<prefix>/share/<token>".
|
||||
const marker = "/share/"
|
||||
i := bytes.LastIndex([]byte(dto.URL), []byte(marker))
|
||||
if i < 0 {
|
||||
t.Fatalf("unerwartetes url-format: %s", dto.URL)
|
||||
}
|
||||
return dto.ID, dto.URL[i+len(marker):]
|
||||
}
|
||||
|
||||
// Akzeptanzkriterium 1: Downloadinhalt entspricht exakt dem Original.
|
||||
func TestAccess_ServesOriginalContent(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "rechnung.txt", "text/plain", []byte("original inhalt"))
|
||||
_, ts := newServer(e, true, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
|
||||
_, token := createLink(t, ts, docID, e.UserID, "", nil)
|
||||
|
||||
resp, err := http.Get(ts.URL + "/share/" + token)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d", resp.StatusCode)
|
||||
}
|
||||
got, _ := io.ReadAll(resp.Body)
|
||||
if string(got) != "original inhalt" {
|
||||
t.Errorf("inhalt = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccess_UnknownToken_Generic404(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
_, ts := newServer(e, true, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
|
||||
resp, err := http.Get(ts.URL + "/share/erratener-token")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("status = %d, erwartet 404", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccess_PasswordFlow(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "geheim.txt", "text/plain", []byte("geschuetzt"))
|
||||
_, ts := newServer(e, true, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
|
||||
_, token := createLink(t, ts, docID, e.UserID, "geheim123", nil)
|
||||
|
||||
getResp, err := http.Get(ts.URL + "/share/" + token)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
getResp.Body.Close()
|
||||
if getResp.StatusCode != http.StatusUnauthorized {
|
||||
t.Errorf("ohne passwort: status = %d, erwartet 401", getResp.StatusCode)
|
||||
}
|
||||
|
||||
wrong, _ := json.Marshal(map[string]string{"password": "falsch"})
|
||||
wrongResp, err := http.Post(ts.URL+"/share/"+token, "application/json", bytes.NewReader(wrong))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wrongResp.Body.Close()
|
||||
if wrongResp.StatusCode != http.StatusUnauthorized {
|
||||
t.Errorf("falsches passwort: status = %d, erwartet 401", wrongResp.StatusCode)
|
||||
}
|
||||
|
||||
right, _ := json.Marshal(map[string]string{"password": "geheim123"})
|
||||
rightResp, err := http.Post(ts.URL+"/share/"+token, "application/json", bytes.NewReader(right))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer rightResp.Body.Close()
|
||||
if rightResp.StatusCode != http.StatusOK {
|
||||
t.Errorf("richtiges passwort: status = %d, erwartet 200", rightResp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccess_LocksAfterFailedAttempts(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "geheim.txt", "text/plain", []byte("x"))
|
||||
_, ts := newServer(e, true, 2, time.Minute)
|
||||
defer ts.Close()
|
||||
_, token := createLink(t, ts, docID, e.UserID, "geheim123", nil)
|
||||
|
||||
wrong, _ := json.Marshal(map[string]string{"password": "falsch"})
|
||||
for i := 0; i < 2; i++ {
|
||||
resp, err := http.Post(ts.URL+"/share/"+token, "application/json", bytes.NewReader(wrong))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
right, _ := json.Marshal(map[string]string{"password": "geheim123"})
|
||||
resp, err := http.Post(ts.URL+"/share/"+token, "application/json", bytes.NewReader(right))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusLocked {
|
||||
t.Errorf("status = %d, erwartet 423 (gesperrt, auch mit richtigem passwort)", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccess_RevokedLink(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "x.txt", "text/plain", []byte("x"))
|
||||
_, ts := newServer(e, true, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
linkID, token := createLink(t, ts, docID, e.UserID, "", nil)
|
||||
|
||||
req, _ := http.NewRequest(http.MethodDelete, ts.URL+"/documents/"+docID+"/share-links/"+linkID, nil)
|
||||
req.Header.Set("X-User-Id", e.UserID)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
t.Fatalf("widerruf: status = %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
accessResp, err := http.Get(ts.URL + "/share/" + token)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
accessResp.Body.Close()
|
||||
if accessResp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("nach widerruf: status = %d, erwartet 404", accessResp.StatusCode)
|
||||
}
|
||||
|
||||
// Erneuter Widerruf ist idempotent-sicher (kein Leck fremder Existenz).
|
||||
req2, _ := http.NewRequest(http.MethodDelete, ts.URL+"/documents/"+docID+"/share-links/"+linkID, nil)
|
||||
req2.Header.Set("X-User-Id", e.UserID)
|
||||
resp2, err := http.DefaultClient.Do(req2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp2.Body.Close()
|
||||
if resp2.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("erneuter widerruf: status = %d, erwartet 404", resp2.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccess_TrashedDocument_NeverServed(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "x.txt", "text/plain", []byte("x"))
|
||||
_, ts := newServer(e, true, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
_, token := createLink(t, ts, docID, e.UserID, "", nil)
|
||||
|
||||
if _, err := e.Pool.Exec(context.Background(), `UPDATE documents SET trashed_at = now() WHERE id = $1`, docID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
resp, err := http.Get(ts.URL + "/share/" + token)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("papierkorb: status = %d, erwartet 404", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestList_NeverExposesTokenOrPassword(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "x.txt", "text/plain", []byte("x"))
|
||||
_, ts := newServer(e, true, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
_, token := createLink(t, ts, docID, e.UserID, "geheim123", nil)
|
||||
|
||||
req, _ := http.NewRequest(http.MethodGet, ts.URL+"/documents/"+docID+"/share-links", nil)
|
||||
req.Header.Set("X-User-Id", e.UserID)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(resp.Body)
|
||||
if bytes.Contains(raw, []byte(token)) {
|
||||
t.Errorf("liste enthaelt den token: %s", raw)
|
||||
}
|
||||
if bytes.Contains(raw, []byte("geheim123")) {
|
||||
t.Errorf("liste enthaelt das passwort: %s", raw)
|
||||
}
|
||||
var dtos []map[string]any
|
||||
if err := json.Unmarshal(raw, &dtos); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(dtos) != 1 || dtos[0]["has_password"] != true {
|
||||
t.Errorf("unerwartete liste: %s", raw)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreate_DisabledForTenant(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
docID, _ := e.UploadDoc(t, "x.txt", "text/plain", []byte("x"))
|
||||
_, ts := newServer(e, false, 5, time.Minute)
|
||||
defer ts.Close()
|
||||
|
||||
req, _ := http.NewRequest(http.MethodPost, ts.URL+"/documents/"+docID+"/share-links", bytes.NewReader([]byte(`{}`)))
|
||||
req.Header.Set("X-User-Id", e.UserID)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusServiceUnavailable {
|
||||
t.Errorf("status = %d, erwartet 503", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
// Akzeptanzkriterium 3/Pruefung: fremder Mandant. Modell C legt je Mandant
|
||||
// eine eigene Datenbank an -- ein Token aus Umgebung A existiert in der
|
||||
// Datenbank von Umgebung B strukturell nicht.
|
||||
func TestAccess_TokenFromOtherTenantIsUnknown(t *testing.T) {
|
||||
a := testenv.New(t)
|
||||
b := testenv.New(t)
|
||||
docID, _ := a.UploadDoc(t, "x.txt", "text/plain", []byte("x"))
|
||||
_, tsA := newServer(a, true, 5, time.Minute)
|
||||
defer tsA.Close()
|
||||
_, tsB := newServer(b, true, 5, time.Minute)
|
||||
defer tsB.Close()
|
||||
|
||||
_, token := createLink(t, tsA, docID, a.UserID, "", nil)
|
||||
|
||||
resp, err := http.Get(tsB.URL + "/share/" + token)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("fremder mandant: status = %d, erwartet 404", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package textcrypt
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// Querier wird von Pool und Transaktion erfuellt.
|
||||
type Querier interface {
|
||||
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
|
||||
}
|
||||
|
||||
// OCRKey liefert den Text-DEK einer Revision und legt ihn beim ersten Zugriff atomar an;
|
||||
// gleichzeitige Seitenbloecke einigen sich ueber COALESCE auf denselben verpackten DEK.
|
||||
func OCRKey(ctx context.Context, db Querier, v Vault, revisionID string) ([]byte, error) {
|
||||
var wrapped []byte
|
||||
if err := db.QueryRow(ctx, `SELECT text_wrapped_dek FROM revision_ocr WHERE revision_id = $1`, revisionID).Scan(&wrapped); err == nil && len(wrapped) > 0 {
|
||||
return v.Key(ctx, wrapped)
|
||||
}
|
||||
dek, fresh, err := v.NewKey(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var stored []byte
|
||||
if err := db.QueryRow(ctx, `
|
||||
INSERT INTO revision_ocr (revision_id, status, text_wrapped_dek) VALUES ($1, 'processing', $2)
|
||||
ON CONFLICT (revision_id) DO UPDATE SET text_wrapped_dek = COALESCE(revision_ocr.text_wrapped_dek, EXCLUDED.text_wrapped_dek)
|
||||
RETURNING text_wrapped_dek
|
||||
`, revisionID, fresh).Scan(&stored); err != nil {
|
||||
return nil, fmt.Errorf("textcrypt: textschluessel ablegen: %w", err)
|
||||
}
|
||||
if bytes.Equal(stored, fresh) {
|
||||
return dek, nil
|
||||
}
|
||||
return v.Key(ctx, stored)
|
||||
}
|
||||
@@ -0,0 +1,235 @@
|
||||
package textcrypt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// Stats zaehlt Zeilen je Textspeicher.
|
||||
type Stats struct {
|
||||
OCRPages int
|
||||
TextIndex int
|
||||
EInvoice int
|
||||
}
|
||||
|
||||
func (s Stats) Total() int { return s.OCRPages + s.TextIndex + s.EInvoice }
|
||||
|
||||
// CountLegacy zaehlt Zeilen, deren Text noch im Klartext liegt.
|
||||
func CountLegacy(ctx context.Context, pool *pgxpool.Pool) (Stats, error) {
|
||||
var s Stats
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT (SELECT count(*) FROM revision_ocr_pages WHERE text_enc IS NULL AND text <> ''),
|
||||
(SELECT count(*) FROM revision_text_index WHERE text_enc IS NULL AND text <> ''),
|
||||
(SELECT count(*) FROM revision_einvoice WHERE search_text_enc IS NULL AND search_text <> '')
|
||||
`).Scan(&s.OCRPages, &s.TextIndex, &s.EInvoice)
|
||||
return s, err
|
||||
}
|
||||
|
||||
// CountEncrypted zaehlt verschluesselte Zeilen.
|
||||
func CountEncrypted(ctx context.Context, pool *pgxpool.Pool) (Stats, error) {
|
||||
var s Stats
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT (SELECT count(*) FROM revision_ocr_pages WHERE text_enc IS NOT NULL),
|
||||
(SELECT count(*) FROM revision_text_index WHERE text_enc IS NOT NULL),
|
||||
(SELECT count(*) FROM revision_einvoice WHERE search_text_enc IS NOT NULL)
|
||||
`).Scan(&s.OCRPages, &s.TextIndex, &s.EInvoice)
|
||||
return s, err
|
||||
}
|
||||
|
||||
// MigrateLegacy verschluesselt vorhandene Klartextzeilen. Je Revision eine Transaktion, idempotent:
|
||||
// ein zweiter Lauf findet nichts mehr.
|
||||
func MigrateLegacy(ctx context.Context, pool *pgxpool.Pool, v Vault) (Stats, error) {
|
||||
var done Stats
|
||||
if err := v.ready(); err != nil {
|
||||
return done, err
|
||||
}
|
||||
pages, err := pool.Query(ctx, `SELECT DISTINCT revision_id::text FROM revision_ocr_pages WHERE text_enc IS NULL AND text <> ''`)
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
revs, err := pgx.CollectRows(pages, pgx.RowTo[string])
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
for _, rev := range revs {
|
||||
n, err := migrateOCRRevision(ctx, pool, v, rev)
|
||||
done.OCRPages += n
|
||||
if err != nil {
|
||||
return done, fmt.Errorf("textcrypt: ocr-revision %s: %w", rev, err)
|
||||
}
|
||||
}
|
||||
n, err := migrateSingle(ctx, pool, v, "revision_text_index", "text", "text_enc", "wrapped_dek", AADIndex)
|
||||
done.TextIndex = n
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
n, err = migrateSingle(ctx, pool, v, "revision_einvoice", "search_text", "search_text_enc", "wrapped_dek", AADInvoice)
|
||||
done.EInvoice = n
|
||||
return done, err
|
||||
}
|
||||
|
||||
func migrateOCRRevision(ctx context.Context, pool *pgxpool.Pool, v Vault, rev string) (int, error) {
|
||||
tx, err := pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
dek, err := OCRKey(ctx, tx, v, rev)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
rows, err := tx.Query(ctx, `SELECT page, text FROM revision_ocr_pages WHERE revision_id = $1 AND text_enc IS NULL AND text <> '' FOR UPDATE`, rev)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
type row struct {
|
||||
page int
|
||||
text string
|
||||
}
|
||||
todo, err := pgx.CollectRows(rows, func(r pgx.CollectableRow) (row, error) {
|
||||
var x row
|
||||
return x, r.Scan(&x.page, &x.text)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
for _, r := range todo {
|
||||
sealed, err := Seal(dek, AADPage(rev, r.page), r.text)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `UPDATE revision_ocr_pages SET text_enc = $3, text = '' WHERE revision_id = $1 AND page = $2`, rev, r.page, sealed); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
return len(todo), tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func migrateSingle(ctx context.Context, pool *pgxpool.Pool, v Vault, table, textCol, encCol, keyCol string, aad func(string) []byte) (int, error) {
|
||||
rows, err := pool.Query(ctx, fmt.Sprintf(`SELECT revision_id::text, %s FROM %s WHERE %s IS NULL AND %s <> ''`, textCol, table, encCol, textCol))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
type row struct{ rev, text string }
|
||||
todo, err := pgx.CollectRows(rows, func(r pgx.CollectableRow) (row, error) {
|
||||
var x row
|
||||
return x, r.Scan(&x.rev, &x.text)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
n := 0
|
||||
for _, r := range todo {
|
||||
dek, wrapped, err := v.NewKey(ctx)
|
||||
if err != nil {
|
||||
return n, err
|
||||
}
|
||||
sealed, err := Seal(dek, aad(r.rev), r.text)
|
||||
if err != nil {
|
||||
return n, err
|
||||
}
|
||||
tag, err := pool.Exec(ctx, fmt.Sprintf(`UPDATE %s SET %s = $2, %s = $3, %s = '' WHERE revision_id = $1 AND %s IS NULL`, table, encCol, keyCol, textCol, encCol), r.rev, sealed, wrapped)
|
||||
if err != nil {
|
||||
return n, fmt.Errorf("textcrypt: %s %s: %w", table, r.rev, err)
|
||||
}
|
||||
n += int(tag.RowsAffected())
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// DecryptAll stellt den Klartext wieder her (Voraussetzung fuer das Zurueckrollen der Migration 0031).
|
||||
func DecryptAll(ctx context.Context, pool *pgxpool.Pool, v Vault) (Stats, error) {
|
||||
var done Stats
|
||||
if err := v.ready(); err != nil {
|
||||
return done, err
|
||||
}
|
||||
revs, err := collectStrings(ctx, pool, `SELECT DISTINCT revision_id::text FROM revision_ocr_pages WHERE text_enc IS NOT NULL`)
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
for _, rev := range revs {
|
||||
var wrapped []byte
|
||||
if err := pool.QueryRow(ctx, `SELECT text_wrapped_dek FROM revision_ocr WHERE revision_id = $1`, rev).Scan(&wrapped); err != nil {
|
||||
return done, err
|
||||
}
|
||||
dek, err := v.Key(ctx, wrapped)
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
rows, err := pool.Query(ctx, `SELECT page, text_enc FROM revision_ocr_pages WHERE revision_id = $1 AND text_enc IS NOT NULL`, rev)
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
type row struct {
|
||||
page int
|
||||
enc []byte
|
||||
}
|
||||
todo, err := pgx.CollectRows(rows, func(r pgx.CollectableRow) (row, error) {
|
||||
var x row
|
||||
return x, r.Scan(&x.page, &x.enc)
|
||||
})
|
||||
if err != nil {
|
||||
return done, err
|
||||
}
|
||||
for _, r := range todo {
|
||||
text, err := Open(dek, AADPage(rev, r.page), r.enc)
|
||||
if err != nil {
|
||||
return done, fmt.Errorf("textcrypt: ocr-revision %s seite %d: %w", rev, r.page, err)
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `UPDATE revision_ocr_pages SET text = $3, text_enc = NULL WHERE revision_id = $1 AND page = $2`, rev, r.page, text); err != nil {
|
||||
return done, err
|
||||
}
|
||||
done.OCRPages++
|
||||
}
|
||||
}
|
||||
if done.TextIndex, err = decryptSingle(ctx, pool, v, "revision_text_index", "text", "text_enc", "wrapped_dek", AADIndex); err != nil {
|
||||
return done, err
|
||||
}
|
||||
done.EInvoice, err = decryptSingle(ctx, pool, v, "revision_einvoice", "search_text", "search_text_enc", "wrapped_dek", AADInvoice)
|
||||
return done, err
|
||||
}
|
||||
|
||||
func collectStrings(ctx context.Context, pool *pgxpool.Pool, sql string) ([]string, error) {
|
||||
rows, err := pool.Query(ctx, sql)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return pgx.CollectRows(rows, pgx.RowTo[string])
|
||||
}
|
||||
|
||||
func decryptSingle(ctx context.Context, pool *pgxpool.Pool, v Vault, table, textCol, encCol, keyCol string, aad func(string) []byte) (int, error) {
|
||||
rows, err := pool.Query(ctx, fmt.Sprintf(`SELECT revision_id::text, %s, %s FROM %s WHERE %s IS NOT NULL`, encCol, keyCol, table, encCol))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
type row struct {
|
||||
rev string
|
||||
enc, kw []byte
|
||||
}
|
||||
todo, err := pgx.CollectRows(rows, func(r pgx.CollectableRow) (row, error) {
|
||||
var x row
|
||||
return x, r.Scan(&x.rev, &x.enc, &x.kw)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
n := 0
|
||||
for _, r := range todo {
|
||||
dek, err := v.Key(ctx, r.kw)
|
||||
if err != nil {
|
||||
return n, err
|
||||
}
|
||||
text, err := Open(dek, aad(r.rev), r.enc)
|
||||
if err != nil {
|
||||
return n, fmt.Errorf("textcrypt: %s %s: %w", table, r.rev, err)
|
||||
}
|
||||
if _, err := pool.Exec(ctx, fmt.Sprintf(`UPDATE %s SET %s = $2, %s = NULL, %s = NULL WHERE revision_id = $1`, table, textCol, encCol, keyCol), r.rev, text); err != nil {
|
||||
return n, err
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package textcrypt_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/einvoice"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/legalhold/legalholdtest"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/ocr"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/plaintext"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/textcrypt"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trash"
|
||||
)
|
||||
|
||||
type kek struct{ b byte }
|
||||
|
||||
func (k kek) TenantKEK(context.Context, string) ([]byte, error) {
|
||||
return bytes.Repeat([]byte{k.b}, dmscrypto.KEKSize), nil
|
||||
}
|
||||
|
||||
const (
|
||||
pageWord = "Wasserkraftwerk"
|
||||
indexWord = "Lehmziegelbrenner"
|
||||
invoiceWord = "Rechnungskoenig"
|
||||
)
|
||||
|
||||
type seeded struct {
|
||||
e *testenv.Env
|
||||
v textcrypt.Vault
|
||||
ocrDoc, ocrRev string
|
||||
idxDoc, idxRev string
|
||||
invDoc, invRev string
|
||||
}
|
||||
|
||||
// seed legt drei Dokumente mit Klartext-Textzeilen an, wie sie vor FDN-22 entstanden.
|
||||
func seed(t *testing.T) seeded {
|
||||
t.Helper()
|
||||
e := testenv.New(t)
|
||||
s := seeded{e: e, v: textcrypt.Vault{Crypto: e.Crypto, TenantSlug: testenv.TenantSlug}}
|
||||
s.ocrDoc, s.ocrRev = e.UploadDoc(t, "scan.txt", "text/plain", []byte("x"))
|
||||
s.idxDoc, s.idxRev = e.UploadDoc(t, "notiz.txt", "text/plain", []byte("y"))
|
||||
s.invDoc, s.invRev = e.UploadDoc(t, "rechnung.txt", "text/plain", []byte("z"))
|
||||
ctx := context.Background()
|
||||
exec := func(q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := e.Pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
exec(`INSERT INTO revision_ocr (revision_id, status, total_pages) VALUES ($1, 'ready', 2)`, s.ocrRev)
|
||||
exec(`INSERT INTO revision_ocr_pages (revision_id, page, text, source) VALUES ($1, 1, $2, 'ocr'), ($1, 2, $3, 'pdf-text')`, s.ocrRev, pageWord+" erste seite", "zweite seite "+pageWord)
|
||||
exec(`INSERT INTO revision_text_index (revision_id, status, source_bytes, indexed_chars, text) VALUES ($1, 'indexed', 30, 30, $2)`, s.idxRev, indexWord+" im klartext")
|
||||
exec(`INSERT INTO revision_einvoice (revision_id, format, container, status, search_text) VALUES ($1, 'cii', 'xml', 'valid', $2)`, s.invRev, invoiceWord+" 2026-0001")
|
||||
return s
|
||||
}
|
||||
|
||||
func (s seeded) rawContains(t *testing.T, word string) bool {
|
||||
t.Helper()
|
||||
var n int
|
||||
err := s.e.Pool.QueryRow(context.Background(), `
|
||||
SELECT (SELECT count(*) FROM revision_ocr_pages WHERE text ILIKE '%'||$1||'%' OR position(convert_to($1, 'UTF8') in text_enc) > 0)
|
||||
+ (SELECT count(*) FROM revision_text_index WHERE text ILIKE '%'||$1||'%' OR position(convert_to($1, 'UTF8') in text_enc) > 0)
|
||||
+ (SELECT count(*) FROM revision_einvoice WHERE search_text ILIKE '%'||$1||'%' OR position(convert_to($1, 'UTF8') in search_text_enc) > 0)
|
||||
`, word).Scan(&n)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n > 0
|
||||
}
|
||||
|
||||
func (s seeded) readAll(t *testing.T) (pages string, idx, inv string) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
pg, err := ocr.Pages(ctx, s.e.Pool, s.e.Crypto, testenv.TenantSlug, s.ocrRev)
|
||||
if err != nil {
|
||||
t.Fatalf("ocr.Pages: %v", err)
|
||||
}
|
||||
var parts []string
|
||||
for _, p := range pg {
|
||||
parts = append(parts, p.Text)
|
||||
}
|
||||
idxText, err := plaintext.TextStore{Pool: s.e.Pool, Storage: s.e.Storage, Crypto: s.e.Crypto, TenantSlug: testenv.TenantSlug}.TextOf(ctx, s.idxDoc)
|
||||
if err != nil {
|
||||
t.Fatalf("plaintext: %v", err)
|
||||
}
|
||||
invText, err := einvoice.TextStore{Pool: s.e.Pool, Crypto: s.e.Crypto, TenantSlug: testenv.TenantSlug}.TextOf(ctx, s.invDoc)
|
||||
if err != nil {
|
||||
t.Fatalf("einvoice: %v", err)
|
||||
}
|
||||
return strings.Join(parts, "|"), idxText, invText
|
||||
}
|
||||
|
||||
// AK3 und Pruefung 3: Bestandsdaten werden ohne Verlust verschluesselt, die Ergebnisse bleiben gleich, ein zweiter Lauf aendert nichts.
|
||||
func TestMigrateLegacy_EncryptsExistingRowsWithoutLosingData(t *testing.T) {
|
||||
s := seed(t)
|
||||
ctx := context.Background()
|
||||
|
||||
beforePages, beforeIdx, beforeInv := s.readAll(t)
|
||||
if !s.rawContains(t, pageWord) || !s.rawContains(t, indexWord) || !s.rawContains(t, invoiceWord) {
|
||||
t.Fatal("ausgangslage: klartext muss in den tabellen liegen")
|
||||
}
|
||||
legacy, err := textcrypt.CountLegacy(ctx, s.e.Pool)
|
||||
if err != nil || legacy.OCRPages != 2 || legacy.TextIndex != 1 || legacy.EInvoice != 1 {
|
||||
t.Fatalf("legacy-zaehlung: %+v %v", legacy, err)
|
||||
}
|
||||
|
||||
done, err := textcrypt.MigrateLegacy(ctx, s.e.Pool, s.v)
|
||||
if err != nil || done.OCRPages != 2 || done.TextIndex != 1 || done.EInvoice != 1 {
|
||||
t.Fatalf("migration: %+v %v", done, err)
|
||||
}
|
||||
for _, w := range []string{pageWord, indexWord, invoiceWord} {
|
||||
if s.rawContains(t, w) {
|
||||
t.Errorf("%q liegt nach der migration noch in der datenbank", w)
|
||||
}
|
||||
}
|
||||
afterPages, afterIdx, afterInv := s.readAll(t)
|
||||
if afterPages != beforePages || afterIdx != beforeIdx || afterInv != beforeInv {
|
||||
t.Fatalf("ergebnis veraendert: %q/%q/%q vs %q/%q/%q", afterPages, afterIdx, afterInv, beforePages, beforeIdx, beforeInv)
|
||||
}
|
||||
if left, _ := textcrypt.CountLegacy(ctx, s.e.Pool); left.Total() != 0 {
|
||||
t.Fatalf("restbestand: %+v", left)
|
||||
}
|
||||
if again, err := textcrypt.MigrateLegacy(ctx, s.e.Pool, s.v); err != nil || again.Total() != 0 {
|
||||
t.Fatalf("zweiter lauf: %+v %v", again, err)
|
||||
}
|
||||
if enc, _ := textcrypt.CountEncrypted(ctx, s.e.Pool); enc.OCRPages != 2 || enc.TextIndex != 1 || enc.EInvoice != 1 {
|
||||
t.Fatalf("verschluesselte zeilen: %+v", enc)
|
||||
}
|
||||
}
|
||||
|
||||
// AK2 und Pruefung 2: falscher Schluessel, fehlender Dienst und fremder Mandanten-KEK liefern nie Klartext.
|
||||
func TestEncryptedTextFailsClosed(t *testing.T) {
|
||||
s := seed(t)
|
||||
ctx := context.Background()
|
||||
if _, err := textcrypt.MigrateLegacy(ctx, s.e.Pool, s.v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
foreign := dmscrypto.NewService(kek{0x77})
|
||||
|
||||
if _, err := ocr.Pages(ctx, s.e.Pool, foreign, testenv.TenantSlug, s.ocrRev); err == nil {
|
||||
t.Error("ocr.Pages mit fremdem kek muss fehlschlagen")
|
||||
}
|
||||
if _, err := ocr.Pages(ctx, s.e.Pool, nil, testenv.TenantSlug, s.ocrRev); err == nil {
|
||||
t.Error("ocr.Pages ohne schluesseldienst muss fehlschlagen")
|
||||
}
|
||||
if txt, err := (ocr.TextStore{Pool: s.e.Pool, Crypto: foreign, TenantSlug: testenv.TenantSlug}).TextOf(ctx, s.ocrDoc); err == nil || strings.Contains(txt, pageWord) {
|
||||
t.Errorf("ocr.TextStore fremder kek: %q %v", txt, err)
|
||||
}
|
||||
if txt, err := (einvoice.TextStore{Pool: s.e.Pool, Crypto: foreign, TenantSlug: testenv.TenantSlug}).TextOf(ctx, s.invDoc); err == nil || strings.Contains(txt, invoiceWord) {
|
||||
t.Errorf("einvoice.TextStore fremder kek: %q %v", txt, err)
|
||||
}
|
||||
if txt, err := (plaintext.TextStore{Pool: s.e.Pool, Storage: s.e.Storage, Crypto: foreign, TenantSlug: testenv.TenantSlug}).TextOf(ctx, s.idxDoc); err == nil || strings.Contains(txt, indexWord) {
|
||||
t.Errorf("plaintext.TextStore fremder kek: %q %v", txt, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ciphertext einer Seite darf nicht auf eine andere Seite oder Revision verschoben werden (Additional Data).
|
||||
func TestEncryptedPageCannotBeMovedToAnotherPageOrRevision(t *testing.T) {
|
||||
s := seed(t)
|
||||
ctx := context.Background()
|
||||
if _, err := textcrypt.MigrateLegacy(ctx, s.e.Pool, s.v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.e.Pool.Exec(ctx, `
|
||||
INSERT INTO revision_ocr_pages (revision_id, page, text, text_enc, source)
|
||||
SELECT revision_id, 3, '', text_enc, source FROM revision_ocr_pages WHERE revision_id = $1 AND page = 1`, s.ocrRev); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := ocr.Pages(ctx, s.e.Pool, s.e.Crypto, testenv.TenantSlug, s.ocrRev); err == nil {
|
||||
t.Fatal("auf eine andere seite verschobener chiffretext wurde akzeptiert")
|
||||
}
|
||||
}
|
||||
|
||||
// Die Datenbank erlaubt nicht, Klartext neben Chiffretext abzulegen.
|
||||
func TestConstraintForbidsPlaintextNextToCipher(t *testing.T) {
|
||||
s := seed(t)
|
||||
ctx := context.Background()
|
||||
if _, err := s.e.Pool.Exec(ctx, `UPDATE revision_text_index SET text_enc = '\x01'::bytea WHERE revision_id = $1`, s.idxRev); err == nil {
|
||||
t.Fatal("klartext neben chiffretext wurde akzeptiert")
|
||||
}
|
||||
}
|
||||
|
||||
// Zurueckrollen: erst entschluesseln, dann verweigert die Down-Migration nicht mehr.
|
||||
func TestDecryptAllThenDownMigrationAllowed(t *testing.T) {
|
||||
s := seed(t)
|
||||
ctx := context.Background()
|
||||
beforePages, beforeIdx, beforeInv := s.readAll(t)
|
||||
if _, err := textcrypt.MigrateLegacy(ctx, s.e.Pool, s.v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
down, err := os.ReadFile("../../migrations/tenant/0031_text_encryption.down.sql")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.e.Pool.Exec(ctx, string(down)); err == nil || !strings.Contains(err.Error(), "zuerst entschluesseln") {
|
||||
t.Fatalf("down-migration muss bei verschluesselten texten verweigern: %v", err)
|
||||
}
|
||||
if rest, _ := textcrypt.CountEncrypted(ctx, s.e.Pool); rest.Total() != 4 {
|
||||
t.Fatalf("verweigerte down-migration hat daten veraendert: %+v", rest)
|
||||
}
|
||||
back, err := textcrypt.DecryptAll(ctx, s.e.Pool, s.v)
|
||||
if err != nil || back.OCRPages != 2 || back.TextIndex != 1 || back.EInvoice != 1 {
|
||||
t.Fatalf("decrypt: %+v %v", back, err)
|
||||
}
|
||||
gotPages, gotIdx, gotInv := s.readAll(t)
|
||||
if gotPages != beforePages || gotIdx != beforeIdx || gotInv != beforeInv {
|
||||
t.Fatal("text nach dem entschluesseln veraendert")
|
||||
}
|
||||
if _, err := s.e.Pool.Exec(ctx, string(down)); err != nil {
|
||||
t.Fatalf("down-migration nach dem entschluesseln: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// AK3: endgueltiges Loeschen entfernt auch die verschluesselten Textkopien (Kaskade ueber die Revision).
|
||||
func TestPurgeRemovesEncryptedTextCopies(t *testing.T) {
|
||||
s := seed(t)
|
||||
ctx := context.Background()
|
||||
if _, err := textcrypt.MigrateLegacy(ctx, s.e.Pool, s.v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.e.Pool.Exec(ctx, `UPDATE documents SET trashed_at = now() - interval '31 days' WHERE id = ANY($1::uuid[])`, []string{s.ocrDoc, s.idxDoc, s.invDoc}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
purged, _, err := trash.PurgeExpired(ctx, s.e.Pool, s.e.Storage, nil, legalholdtest.Allow{}, 30*24*time.Hour)
|
||||
if err != nil || purged != 3 {
|
||||
t.Fatalf("purge: %d %v", purged, err)
|
||||
}
|
||||
enc, err := textcrypt.CountEncrypted(ctx, s.e.Pool)
|
||||
if err != nil || enc.Total() != 0 {
|
||||
t.Fatalf("textkopien nach dem loeschen: %+v %v", enc, err)
|
||||
}
|
||||
var keys int
|
||||
_ = s.e.Pool.QueryRow(ctx, `SELECT count(*) FROM revision_ocr WHERE text_wrapped_dek IS NOT NULL`).Scan(&keys)
|
||||
if keys != 0 {
|
||||
t.Fatalf("verpackte schluessel bleiben zurueck: %d", keys)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
// Package textcrypt verschluesselt Textkopien von Dokumentinhalten (OCR-Text,
|
||||
// Klartext-Extrakt, E-Rechnungs-Suchtext, FDN-22). Je Revision gibt es einen
|
||||
// DEK, verpackt mit dem Mandanten-KEK; jeder Text ist per Additional Data an
|
||||
// Revision und Seite gebunden. Es gibt keinen Klartext-Rueckfall.
|
||||
package textcrypt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/rand"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
)
|
||||
|
||||
const versionV1 byte = 0x01
|
||||
|
||||
var (
|
||||
ErrNoCrypto = errors.New("textcrypt: kein schluesseldienst konfiguriert, text wird nicht im klartext gespeichert oder gelesen")
|
||||
ErrDecryptText = errors.New("textcrypt: text nicht entschluesselbar (falscher schluessel, fremde revision oder manipulierte daten)")
|
||||
)
|
||||
|
||||
// Vault verbindet den Schluesseldienst mit dem Mandanten.
|
||||
type Vault struct {
|
||||
Crypto *dmscrypto.Service
|
||||
TenantSlug string
|
||||
}
|
||||
|
||||
func (v Vault) ready() error {
|
||||
if v.Crypto == nil || v.TenantSlug == "" {
|
||||
return ErrNoCrypto
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NewKey erzeugt einen DEK samt verpackter Form.
|
||||
func (v Vault) NewKey(ctx context.Context) (dek, wrapped []byte, err error) {
|
||||
if err := v.ready(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return v.Crypto.NewDataKey(ctx, v.TenantSlug)
|
||||
}
|
||||
|
||||
// Key entpackt einen gespeicherten DEK.
|
||||
func (v Vault) Key(ctx context.Context, wrapped []byte) ([]byte, error) {
|
||||
if err := v.ready(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(wrapped) == 0 {
|
||||
return nil, ErrDecryptText
|
||||
}
|
||||
dek, err := v.Crypto.UnwrapDataKey(ctx, v.TenantSlug, wrapped)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", ErrDecryptText, err)
|
||||
}
|
||||
return dek, nil
|
||||
}
|
||||
|
||||
func AADPage(revisionID string, page int) []byte {
|
||||
return []byte(fmt.Sprintf("ocr-page:%s:%d", revisionID, page))
|
||||
}
|
||||
func AADIndex(revisionID string) []byte { return []byte("text-index:" + revisionID) }
|
||||
func AADInvoice(revisionID string) []byte { return []byte("einvoice-text:" + revisionID) }
|
||||
|
||||
func gcm(dek []byte) (cipher.AEAD, error) {
|
||||
block, err := aes.NewCipher(dek)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return cipher.NewGCM(block)
|
||||
}
|
||||
|
||||
// Seal liefert Version || Nonce || Chiffretext.
|
||||
func Seal(dek []byte, aad []byte, text string) ([]byte, error) {
|
||||
g, err := gcm(dek)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("textcrypt: cipher: %w", err)
|
||||
}
|
||||
nonce := make([]byte, g.NonceSize())
|
||||
if _, err := rand.Read(nonce); err != nil {
|
||||
return nil, fmt.Errorf("textcrypt: nonce: %w", err)
|
||||
}
|
||||
out := append([]byte{versionV1}, nonce...)
|
||||
return g.Seal(out, nonce, []byte(text), aad), nil
|
||||
}
|
||||
|
||||
// Open entschluesselt einen mit Seal erzeugten Wert.
|
||||
func Open(dek []byte, aad []byte, sealed []byte) (string, error) {
|
||||
g, err := gcm(dek)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("textcrypt: cipher: %w", err)
|
||||
}
|
||||
if len(sealed) < 1+g.NonceSize() || sealed[0] != versionV1 {
|
||||
return "", ErrDecryptText
|
||||
}
|
||||
nonce, ct := sealed[1:1+g.NonceSize()], sealed[1+g.NonceSize():]
|
||||
plain, err := g.Open(nil, nonce, ct, aad)
|
||||
if err != nil {
|
||||
return "", ErrDecryptText
|
||||
}
|
||||
return string(plain), nil
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package textcrypt
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
)
|
||||
|
||||
type staticKEK struct {
|
||||
kek []byte
|
||||
err error
|
||||
}
|
||||
|
||||
func (s staticKEK) TenantKEK(context.Context, string) ([]byte, error) { return s.kek, s.err }
|
||||
|
||||
func vaultWith(b byte) Vault {
|
||||
return Vault{Crypto: dmscrypto.NewService(staticKEK{kek: bytes.Repeat([]byte{b}, dmscrypto.KEKSize)}), TenantSlug: "acme"}
|
||||
}
|
||||
|
||||
func TestSealOpenRoundtripAndUniqueNonce(t *testing.T) {
|
||||
dek, _ := dmscrypto.GenerateDEK()
|
||||
aad := AADPage("rev-1", 3)
|
||||
a, err := Seal(dek, aad, "Zebrafinken bauen Lehmziegel")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := Seal(dek, aad, "Zebrafinken bauen Lehmziegel")
|
||||
if bytes.Equal(a, b) {
|
||||
t.Fatal("zwei verschluesselungen desselben textes sind identisch (nonce wiederverwendet)")
|
||||
}
|
||||
if bytes.Contains(a, []byte("Zebrafinken")) {
|
||||
t.Fatal("klartext im chiffretext")
|
||||
}
|
||||
got, err := Open(dek, aad, a)
|
||||
if err != nil || got != "Zebrafinken bauen Lehmziegel" {
|
||||
t.Fatalf("open: %q %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenRejectsWrongKeyAADTamperAndVersion(t *testing.T) {
|
||||
dek, _ := dmscrypto.GenerateDEK()
|
||||
other, _ := dmscrypto.GenerateDEK()
|
||||
aad := AADIndex("rev-1")
|
||||
sealed, _ := Seal(dek, aad, "geheim")
|
||||
|
||||
cases := map[string]func() error{
|
||||
"falscher schluessel": func() error { _, err := Open(other, aad, sealed); return err },
|
||||
"andere revision": func() error { _, err := Open(dek, AADIndex("rev-2"), sealed); return err },
|
||||
"andere seite": func() error { _, err := Open(dek, AADPage("rev-1", 2), sealed); return err },
|
||||
"andere textart": func() error { _, err := Open(dek, AADInvoice("rev-1"), sealed); return err },
|
||||
"manipuliert": func() error {
|
||||
m := append([]byte(nil), sealed...)
|
||||
m[len(m)-1] ^= 0xFF
|
||||
_, err := Open(dek, aad, m)
|
||||
return err
|
||||
},
|
||||
"unbekannte version": func() error {
|
||||
m := append([]byte(nil), sealed...)
|
||||
m[0] = 0x7F
|
||||
_, err := Open(dek, aad, m)
|
||||
return err
|
||||
},
|
||||
"zu kurz": func() error { _, err := Open(dek, aad, []byte{versionV1, 1, 2}); return err },
|
||||
}
|
||||
for name, f := range cases {
|
||||
if err := f(); !errors.Is(err, ErrDecryptText) {
|
||||
t.Errorf("%s: erwartet ErrDecryptText, erhalten %v", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestVaultFailsClosedWithoutCryptoOrWithWrongKEK(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
if _, _, err := (Vault{}).NewKey(ctx); !errors.Is(err, ErrNoCrypto) {
|
||||
t.Fatalf("ohne dienst: %v", err)
|
||||
}
|
||||
if _, err := (Vault{Crypto: vaultWith(1).Crypto}).Key(ctx, []byte{1}); !errors.Is(err, ErrNoCrypto) {
|
||||
t.Fatalf("ohne mandant: %v", err)
|
||||
}
|
||||
|
||||
a := vaultWith(0x11)
|
||||
dek, wrapped, err := a.NewKey(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if again, err := a.Key(ctx, wrapped); err != nil || !bytes.Equal(again, dek) {
|
||||
t.Fatalf("schluessel nicht wiederherstellbar: %v", err)
|
||||
}
|
||||
if _, err := vaultWith(0x22).Key(ctx, wrapped); !errors.Is(err, ErrDecryptText) {
|
||||
t.Fatalf("fremder kek: %v", err)
|
||||
}
|
||||
broken := Vault{Crypto: dmscrypto.NewService(staticKEK{err: errors.New("kek-dienst nicht erreichbar")}), TenantSlug: "acme"}
|
||||
if _, err := broken.Key(ctx, wrapped); err == nil || !strings.Contains(err.Error(), "nicht erreichbar") {
|
||||
t.Fatalf("kek-ausfall muss fehlschlagen: %v", err)
|
||||
}
|
||||
if _, _, err := broken.NewKey(ctx); err == nil {
|
||||
t.Fatal("kek-ausfall beim anlegen muss fehlschlagen")
|
||||
}
|
||||
if _, err := a.Key(ctx, nil); !errors.Is(err, ErrDecryptText) {
|
||||
t.Fatalf("leerer verpackter schluessel: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package trigger
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// Adapter bindet Fire an einen festen Pool/Vault/Mandanten, damit es die
|
||||
// schmalen Hook-Schnittstellen aus internal/upload (TriggerFirer) sowie die
|
||||
// OnText-/OnConformanceChecked-Hooks aus internal/ocr und internal/einvoice
|
||||
// erfuellt, ohne dass jene Pakete internal/trigger kennen muessen.
|
||||
type Adapter struct {
|
||||
Pool *pgxpool.Pool
|
||||
Vault Vault
|
||||
TenantSlug string
|
||||
}
|
||||
|
||||
func (a *Adapter) Fire(ctx context.Context, event, documentID string) error {
|
||||
return Fire(ctx, a.Pool, a.Vault, a.TenantSlug, event, documentID)
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
package trigger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"regexp"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/documenttype"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/tag"
|
||||
)
|
||||
|
||||
// Fire wertet alle aktivierten Trigger fuer event gegen documentID aus und
|
||||
// fuehrt die passenden Aktionen aus. Ein Fehler in einer Aktion oder beim
|
||||
// Werten eines einzelnen Triggers wird in trigger_runs und dem Audit-Log
|
||||
// festgehalten, bricht aber die Auswertung der uebrigen Trigger nicht ab und
|
||||
// wird NIE an den Aufrufer zurueckgegeben (siehe Package-Doc) — Fire selbst
|
||||
// gibt nur einen Fehler zurueck, wenn das Dokument nicht lesbar ist.
|
||||
func Fire(ctx context.Context, pool *pgxpool.Pool, v Vault, tenantSlug, event, documentID string) error {
|
||||
var title, typeID string
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT title, COALESCE(document_type_id::text, '') FROM documents
|
||||
WHERE id = $1 AND trashed_at IS NULL AND deleted_at IS NULL
|
||||
`, documentID).Scan(&title, &typeID)
|
||||
if err == pgx.ErrNoRows {
|
||||
return nil // Dokument geloescht/im Papierkorb: kein Trigger feuert.
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("trigger: dokument lesen: %w", err)
|
||||
}
|
||||
|
||||
rows, err := pool.Query(ctx, `SELECT `+columns+`, webhook_secret_wrapped_dek, webhook_secret_enc FROM triggers WHERE event = $1 AND enabled`, event)
|
||||
if err != nil {
|
||||
return fmt.Errorf("trigger: liste lesen: %w", err)
|
||||
}
|
||||
type row struct {
|
||||
Trigger
|
||||
wrappedDEK, secretEnc []byte
|
||||
}
|
||||
var list []row
|
||||
for rows.Next() {
|
||||
var r row
|
||||
if err := rows.Scan(&r.ID, &r.Name, &r.Event, &r.FilenamePattern, &r.DocumentTypeID, &r.ActionType, &r.ActionTagID, &r.ActionDocumentTypeID, &r.WebhookURL, &r.Enabled, &r.CreatedBy, &r.wrappedDEK, &r.secretEnc); err != nil {
|
||||
rows.Close()
|
||||
return fmt.Errorf("trigger: liste lesen: %w", err)
|
||||
}
|
||||
list = append(list, r)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, r := range list {
|
||||
if r.FilenamePattern != nil && !regexp.MustCompile(*r.FilenamePattern).MatchString(title) {
|
||||
continue
|
||||
}
|
||||
if r.DocumentTypeID != nil && *r.DocumentTypeID != typeID {
|
||||
continue
|
||||
}
|
||||
runErr := runAction(ctx, pool, v, tenantSlug, r.Trigger, r.wrappedDEK, r.secretEnc, documentID)
|
||||
status := "ok"
|
||||
errText := ""
|
||||
if runErr != nil {
|
||||
status = "failed"
|
||||
errText = runErr.Error()
|
||||
log.Printf("trigger: aktion fuer %q an dokument %q fehlgeschlagen: %v", r.Name, documentID, runErr)
|
||||
}
|
||||
if _, logErr := pool.Exec(ctx, `
|
||||
INSERT INTO trigger_runs (trigger_id, document_id, event, status, error, fired_at)
|
||||
VALUES ($1, $2, $3, $4, NULLIF($5, ''), $6)
|
||||
`, r.ID, documentID, event, status, errText, time.Now()); logErr != nil {
|
||||
log.Printf("trigger: lauf konnte nicht protokolliert werden: %v", logErr)
|
||||
}
|
||||
if runErr != nil {
|
||||
_ = audittrail.Record(ctx, pool, audittrail.SystemActor, "trigger_failed", "trigger", r.ID, map[string]any{"document_id": documentID, "error": errText})
|
||||
} else {
|
||||
_ = audittrail.Record(ctx, pool, audittrail.SystemActor, "trigger_fired", "trigger", r.ID, map[string]any{"document_id": documentID, "action": r.ActionType})
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runAction(ctx context.Context, pool *pgxpool.Pool, v Vault, tenantSlug string, t Trigger, wrappedDEK, secretEnc []byte, documentID string) error {
|
||||
switch t.ActionType {
|
||||
case ActionSetTag:
|
||||
return tag.Assign(ctx, pool, documentID, *t.ActionTagID)
|
||||
case ActionSetType:
|
||||
_, err := documenttype.ChangeDocumentType(ctx, pool, documentID, *t.ActionDocumentTypeID, false)
|
||||
return err
|
||||
case ActionWebhook:
|
||||
secret := ""
|
||||
if len(secretEnc) > 0 {
|
||||
var err error
|
||||
secret, err = v.Open(ctx, tenantSlug, wrappedDEK, secretEnc)
|
||||
if err != nil {
|
||||
return fmt.Errorf("geheimnis entschluesseln: %w", err)
|
||||
}
|
||||
}
|
||||
return sendWebhook(ctx, *t.WebhookURL, secret, webhookPayload{
|
||||
Event: t.Event,
|
||||
DocumentID: documentID,
|
||||
TriggerID: t.ID,
|
||||
OccurredAt: time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
default:
|
||||
return fmt.Errorf("unbekannte aktion %q", t.ActionType)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
// Package trigger implementiert INT-20: Trigger loesen bei einem Ereignis
|
||||
// (Upload abgeschlossen, Texterkennung fertig, E-Rechnung geprueft) eine
|
||||
// Aktion aus (Webhook, Tag setzen, Dokumententyp setzen). Ein Trigger-Fehler
|
||||
// wird protokolliert, blockiert aber nie die Verarbeitung des Dokuments
|
||||
// selbst (fire-and-forget mit Nachweis in trigger_runs und dem Audit-Log).
|
||||
package trigger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/audittrail"
|
||||
)
|
||||
|
||||
const (
|
||||
EventUploaded = "document.uploaded"
|
||||
EventOCRCompleted = "document.ocr_completed"
|
||||
EventEInvoiceChecked = "document.einvoice_checked"
|
||||
|
||||
ActionWebhook = "webhook"
|
||||
ActionSetTag = "set_tag"
|
||||
ActionSetType = "set_type"
|
||||
|
||||
maxPatternLength = 500
|
||||
maxNameLength = 200
|
||||
)
|
||||
|
||||
var validEvents = map[string]bool{
|
||||
EventUploaded: true,
|
||||
EventOCRCompleted: true,
|
||||
EventEInvoiceChecked: true,
|
||||
}
|
||||
|
||||
var (
|
||||
ErrNotFound = errors.New("trigger: nicht gefunden")
|
||||
ErrInvalid = errors.New("trigger: ungueltig")
|
||||
)
|
||||
|
||||
// Trigger ist eine Datenbankzeile; ActionSecret ist nur beim Anlegen/Aendern
|
||||
// gesetzt (Klartext-Eingabe), nie beim Lesen (siehe Store.List/Get).
|
||||
type Trigger struct {
|
||||
ID string
|
||||
Name string
|
||||
Event string
|
||||
FilenamePattern *string
|
||||
DocumentTypeID *string
|
||||
ActionType string
|
||||
ActionTagID *string
|
||||
ActionDocumentTypeID *string
|
||||
WebhookURL *string
|
||||
Enabled bool
|
||||
CreatedBy string
|
||||
}
|
||||
|
||||
func invalid(format string, a ...any) error {
|
||||
return fmt.Errorf("%w: %s", ErrInvalid, fmt.Sprintf(format, a...))
|
||||
}
|
||||
|
||||
func (t Trigger) validate() error {
|
||||
name := strings.TrimSpace(t.Name)
|
||||
if name == "" || len(name) > maxNameLength {
|
||||
return invalid("name muss 1 bis %d zeichen lang sein", maxNameLength)
|
||||
}
|
||||
if !validEvents[t.Event] {
|
||||
return invalid("unbekanntes ereignis %q", t.Event)
|
||||
}
|
||||
if t.FilenamePattern != nil {
|
||||
if len(*t.FilenamePattern) == 0 || len(*t.FilenamePattern) > maxPatternLength {
|
||||
return invalid("dateinamensmuster muss 1 bis %d zeichen lang sein", maxPatternLength)
|
||||
}
|
||||
if _, err := regexp.Compile(*t.FilenamePattern); err != nil {
|
||||
return invalid("dateinamensmuster ist kein gueltiger regulaerer ausdruck: %v", err)
|
||||
}
|
||||
}
|
||||
switch t.ActionType {
|
||||
case ActionWebhook:
|
||||
if t.WebhookURL == nil || strings.TrimSpace(*t.WebhookURL) == "" {
|
||||
return invalid("webhook-aktion braucht eine ziel-url")
|
||||
}
|
||||
if err := ValidateWebhookURL(*t.WebhookURL); err != nil {
|
||||
return invalid("webhook-url: %v", err)
|
||||
}
|
||||
case ActionSetTag:
|
||||
if t.ActionTagID == nil || strings.TrimSpace(*t.ActionTagID) == "" {
|
||||
return invalid("set_tag-aktion braucht ein tag")
|
||||
}
|
||||
case ActionSetType:
|
||||
if t.ActionDocumentTypeID == nil || strings.TrimSpace(*t.ActionDocumentTypeID) == "" {
|
||||
return invalid("set_type-aktion braucht einen dokumententyp")
|
||||
}
|
||||
default:
|
||||
return invalid("unbekannte aktion %q", t.ActionType)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
const columns = `id, name, event, filename_pattern, document_type_id, action_type, action_tag_id, action_document_type_id, webhook_url, enabled, created_by`
|
||||
|
||||
type scanner interface{ Scan(dest ...any) error }
|
||||
|
||||
func scanTrigger(row scanner) (Trigger, error) {
|
||||
var t Trigger
|
||||
err := row.Scan(&t.ID, &t.Name, &t.Event, &t.FilenamePattern, &t.DocumentTypeID, &t.ActionType, &t.ActionTagID, &t.ActionDocumentTypeID, &t.WebhookURL, &t.Enabled, &t.CreatedBy)
|
||||
return t, err
|
||||
}
|
||||
|
||||
func referenceError(err error) error {
|
||||
if strings.Contains(err.Error(), "SQLSTATE 23503") {
|
||||
return invalid("verweist auf einen unbekannten dokumententyp, tag oder ersteller")
|
||||
}
|
||||
return fmt.Errorf("trigger: speichern: %w", err)
|
||||
}
|
||||
|
||||
// Create legt einen Trigger an. secret ist nur bei action_type=webhook relevant
|
||||
// (optional) und wird ueber Vault.Seal verpackt, nie im Klartext gespeichert.
|
||||
func Create(ctx context.Context, pool *pgxpool.Pool, v Vault, tenantSlug, actor string, t Trigger, secret string) (Trigger, error) {
|
||||
t.Enabled = true
|
||||
t.CreatedBy = actor
|
||||
if err := t.validate(); err != nil {
|
||||
return Trigger{}, err
|
||||
}
|
||||
var wrappedDEK, secretEnc []byte
|
||||
if t.ActionType == ActionWebhook && secret != "" {
|
||||
var err error
|
||||
wrappedDEK, secretEnc, err = v.Seal(ctx, tenantSlug, secret)
|
||||
if err != nil {
|
||||
return Trigger{}, fmt.Errorf("trigger: geheimnis verschluesseln: %w", err)
|
||||
}
|
||||
}
|
||||
out, err := scanTrigger(pool.QueryRow(ctx, `
|
||||
INSERT INTO triggers (name, event, filename_pattern, document_type_id, action_type, action_tag_id, action_document_type_id, webhook_url, webhook_secret_wrapped_dek, webhook_secret_enc, enabled, created_by)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) RETURNING `+columns,
|
||||
t.Name, t.Event, t.FilenamePattern, t.DocumentTypeID, t.ActionType, t.ActionTagID, t.ActionDocumentTypeID, t.WebhookURL, wrappedDEK, secretEnc, t.Enabled, actor))
|
||||
if err != nil {
|
||||
return Trigger{}, referenceError(err)
|
||||
}
|
||||
if err := audittrail.Record(ctx, pool, actor, "trigger_created", "trigger", out.ID, map[string]any{"name": out.Name, "event": out.Event}); err != nil {
|
||||
return Trigger{}, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// SetEnabled aktiviert/deaktiviert einen Trigger, ohne die Konfiguration zu aendern.
|
||||
func SetEnabled(ctx context.Context, pool *pgxpool.Pool, actor, id string, enabled bool) error {
|
||||
tag, err := pool.Exec(ctx, `UPDATE triggers SET enabled = $2, updated_at = now() WHERE id = $1`, id, enabled)
|
||||
if err != nil {
|
||||
return fmt.Errorf("trigger: aktivieren/deaktivieren: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
action := "trigger_disabled"
|
||||
if enabled {
|
||||
action = "trigger_enabled"
|
||||
}
|
||||
return audittrail.Record(ctx, pool, actor, action, "trigger", id, nil)
|
||||
}
|
||||
|
||||
// Delete entfernt einen Trigger (und mit ihm die trigger_runs, ON DELETE CASCADE).
|
||||
func Delete(ctx context.Context, pool *pgxpool.Pool, actor, id string) error {
|
||||
tag, err := pool.Exec(ctx, `DELETE FROM triggers WHERE id = $1`, id)
|
||||
if err != nil {
|
||||
return fmt.Errorf("trigger: loeschen: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return audittrail.Record(ctx, pool, actor, "trigger_deleted", "trigger", id, nil)
|
||||
}
|
||||
|
||||
// Get liefert einen Trigger ohne das Webhook-Geheimnis.
|
||||
func Get(ctx context.Context, pool *pgxpool.Pool, id string) (Trigger, error) {
|
||||
t, err := scanTrigger(pool.QueryRow(ctx, `SELECT `+columns+` FROM triggers WHERE id = $1`, id))
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return Trigger{}, ErrNotFound
|
||||
}
|
||||
return t, err
|
||||
}
|
||||
|
||||
// List liefert alle Trigger, ohne das Webhook-Geheimnis.
|
||||
func List(ctx context.Context, pool *pgxpool.Pool) ([]Trigger, error) {
|
||||
rows, err := pool.Query(ctx, `SELECT `+columns+` FROM triggers ORDER BY name`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trigger: liste lesen: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
out := []Trigger{}
|
||||
for rows.Next() {
|
||||
t, err := scanTrigger(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, t)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
package trigger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/documenttype"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/tag"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
func sp(s string) *string { return &s }
|
||||
|
||||
// memVault ist ein Test-Vault ohne echten KEK-Server: es verpackt reversibel
|
||||
// (Praefix statt Klartext-Passthrough), damit Tests einen echten Seal/Open-
|
||||
// Roundtrip pruefen koennen, ohne einen KEK-Provider hochzufahren.
|
||||
type memVault struct{}
|
||||
|
||||
func (memVault) Seal(_ context.Context, _, plaintext string) ([]byte, []byte, error) {
|
||||
return []byte("dek"), []byte("enc:" + plaintext), nil
|
||||
}
|
||||
func (memVault) Open(_ context.Context, _ string, _, ciphertext []byte) (string, error) {
|
||||
return strings.TrimPrefix(string(ciphertext), "enc:"), nil
|
||||
}
|
||||
|
||||
func newEnv(t *testing.T) *testenv.Env {
|
||||
t.Helper()
|
||||
return testenv.New(t)
|
||||
}
|
||||
|
||||
func TestCreate_ValidationRejectsIncompleteTriggers(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
cases := []struct {
|
||||
name string
|
||||
tr Trigger
|
||||
}{
|
||||
{"kein name", Trigger{Event: EventUploaded, ActionType: ActionSetTag, ActionTagID: sp("00000000-0000-0000-0000-000000000001")}},
|
||||
{"unbekanntes ereignis", Trigger{Name: "x", Event: "document.deleted", ActionType: ActionSetTag, ActionTagID: sp("00000000-0000-0000-0000-000000000001")}},
|
||||
{"webhook ohne url", Trigger{Name: "x", Event: EventUploaded, ActionType: ActionWebhook}},
|
||||
{"set_tag ohne tag", Trigger{Name: "x", Event: EventUploaded, ActionType: ActionSetTag}},
|
||||
{"ungueltiges muster", Trigger{Name: "x", Event: EventUploaded, FilenamePattern: sp("(["), ActionType: ActionSetTag, ActionTagID: sp("00000000-0000-0000-0000-000000000001")}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if _, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", c.tr, ""); err == nil {
|
||||
t.Fatalf("erwartet fehler fuer %q", c.name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFire_FilenamePatternMatchAndNoMatch(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
tg, err := tag.Create(ctx, e.Pool, "auto", "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", Trigger{
|
||||
Name: "Rechnung", Event: EventUploaded, FilenamePattern: sp(`(?i)rechnung`), ActionType: ActionSetTag, ActionTagID: sp(tg.ID),
|
||||
}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
docMatch, _ := e.UploadDoc(t, "Rechnung-2026.pdf", "text/plain", []byte("inhalt rechnung"))
|
||||
if err := Fire(ctx, e.Pool, memVault{}, testenv.TenantSlug, EventUploaded, docMatch); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var n int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM document_tags WHERE document_id = $1 AND tag_id = $2`, docMatch, tg.ID).Scan(&n)
|
||||
if n != 1 {
|
||||
t.Fatalf("tag nicht gesetzt bei passendem dateinamen, n=%d", n)
|
||||
}
|
||||
|
||||
docNoMatch, _ := e.UploadDoc(t, "Kontoauszug.pdf", "text/plain", []byte("inhalt kontoauszug"))
|
||||
if err := Fire(ctx, e.Pool, memVault{}, testenv.TenantSlug, EventUploaded, docNoMatch); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
n = 0
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM document_tags WHERE document_id = $1`, docNoMatch).Scan(&n)
|
||||
if n != 0 {
|
||||
t.Fatalf("tag faelschlich gesetzt bei nicht passendem dateinamen, n=%d", n)
|
||||
}
|
||||
|
||||
var runs int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM trigger_runs WHERE document_id = $1 AND status = 'ok'`, docMatch).Scan(&runs)
|
||||
if runs != 1 {
|
||||
t.Fatalf("erwartet 1 erfolgreichen lauf, habe %d", runs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFire_SetTypeAction(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
typ, err := documenttype.Create(ctx, e.Pool, "Rechnung", nil, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", Trigger{
|
||||
Name: "SetzeTyp", Event: EventUploaded, ActionType: ActionSetType, ActionDocumentTypeID: sp(typ.ID),
|
||||
}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
doc, _ := e.UploadDoc(t, "beleg.txt", "text/plain", []byte("inhalt"))
|
||||
if err := Fire(ctx, e.Pool, memVault{}, testenv.TenantSlug, EventUploaded, doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var typeID string
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT document_type_id::text FROM documents WHERE id = $1`, doc).Scan(&typeID)
|
||||
if typeID != typ.ID {
|
||||
t.Fatalf("dokumententyp nicht gesetzt: %q", typeID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFire_WebhookSendsSignedPayload(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
received := make(chan webhookPayload, 1)
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
sig := r.Header.Get("X-Trigger-Signature")
|
||||
if sig == "" {
|
||||
t.Error("erwartet signatur-header")
|
||||
}
|
||||
var p webhookPayload
|
||||
_ = json.NewDecoder(r.Body).Decode(&p)
|
||||
received <- p
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer srv.Close()
|
||||
t.Setenv("NEXARCH_DMS_APP_TRIGGER_WEBHOOK_ALLOW_PRIVATE", "true")
|
||||
|
||||
url := srv.URL
|
||||
if _, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", Trigger{
|
||||
Name: "Hook", Event: EventUploaded, ActionType: ActionWebhook, WebhookURL: &url,
|
||||
}, "geheim"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
doc, _ := e.UploadDoc(t, "beleg.txt", "text/plain", []byte("inhalt"))
|
||||
if err := Fire(ctx, e.Pool, memVault{}, testenv.TenantSlug, EventUploaded, doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case p := <-received:
|
||||
if p.DocumentID != doc || p.Event != EventUploaded {
|
||||
t.Fatalf("unerwartete nutzlast: %+v", p)
|
||||
}
|
||||
default:
|
||||
t.Fatal("webhook wurde nicht aufgerufen")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFire_WebhookFailureDoesNotBlockFireOrOtherTriggers(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
t.Setenv("NEXARCH_DMS_APP_TRIGGER_WEBHOOK_ALLOW_PRIVATE", "true") // erlaubt das ziel bei der ANLAGE; der Verbindungsfehler entsteht erst beim Senden.
|
||||
tg, err := tag.Create(ctx, e.Pool, "trotzdem", "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
badURL := "http://127.0.0.1:1" // verbindungsfehler garantiert
|
||||
if _, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", Trigger{
|
||||
Name: "KaputterHook", Event: EventUploaded, ActionType: ActionWebhook, WebhookURL: &badURL,
|
||||
}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", Trigger{
|
||||
Name: "FunktionierendesTag", Event: EventUploaded, ActionType: ActionSetTag, ActionTagID: sp(tg.ID),
|
||||
}, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
doc, _ := e.UploadDoc(t, "beleg.txt", "text/plain", []byte("inhalt"))
|
||||
// Fire selbst darf trotz fehlschlagendem Webhook keinen Fehler liefern
|
||||
// (Trigger-Fehler blockiert die Verarbeitung des Dokuments nicht).
|
||||
if err := Fire(ctx, e.Pool, memVault{}, testenv.TenantSlug, EventUploaded, doc); err != nil {
|
||||
t.Fatalf("Fire darf nicht fehlschlagen: %v", err)
|
||||
}
|
||||
var n int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM document_tags WHERE document_id = $1 AND tag_id = $2`, doc, tg.ID).Scan(&n)
|
||||
if n != 1 {
|
||||
t.Fatal("der andere trigger haette trotzdem feuern muessen")
|
||||
}
|
||||
var failed int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM trigger_runs WHERE document_id = $1 AND status = 'failed'`, doc).Scan(&failed)
|
||||
if failed != 1 {
|
||||
t.Fatalf("erwartet 1 fehlgeschlagenen lauf protokolliert, habe %d", failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateWebhookURL_RejectsPrivateAndLoopbackTargets(t *testing.T) {
|
||||
cases := []string{
|
||||
"http://127.0.0.1:8080/hook",
|
||||
"http://localhost/hook",
|
||||
"http://10.0.0.5/hook",
|
||||
"http://169.254.169.254/latest/meta-data",
|
||||
"http://192.168.1.1/hook",
|
||||
"ftp://example.test/hook",
|
||||
"not-a-url",
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c, func(t *testing.T) {
|
||||
if err := ValidateWebhookURL(c); err == nil {
|
||||
t.Fatalf("erwartet ablehnung fuer %q", c)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateWebhookURL_AcceptsPublicHTTPS(t *testing.T) {
|
||||
if err := ValidateWebhookURL("https://example.test/hook"); err != nil {
|
||||
t.Fatalf("oeffentliches ziel sollte erlaubt sein: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateWebhookURL_AllowPrivateOverride(t *testing.T) {
|
||||
t.Setenv("NEXARCH_DMS_APP_TRIGGER_WEBHOOK_ALLOW_PRIVATE", "true")
|
||||
if err := ValidateWebhookURL("http://127.0.0.1:9999/hook"); err != nil {
|
||||
t.Fatalf("mit freigabe sollte loopback erlaubt sein: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFire_DisabledTriggerDoesNotFire(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
tg, err := tag.Create(ctx, e.Pool, "sollte-nicht", "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out, err := Create(ctx, e.Pool, memVault{}, testenv.TenantSlug, "tester", Trigger{
|
||||
Name: "Deaktiviert", Event: EventUploaded, ActionType: ActionSetTag, ActionTagID: sp(tg.ID),
|
||||
}, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SetEnabled(ctx, e.Pool, "tester", out.ID, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
doc, _ := e.UploadDoc(t, "beleg.txt", "text/plain", []byte("inhalt"))
|
||||
if err := Fire(ctx, e.Pool, memVault{}, testenv.TenantSlug, EventUploaded, doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var n int
|
||||
_ = e.Pool.QueryRow(ctx, `SELECT count(*) FROM document_tags WHERE document_id = $1`, doc).Scan(&n)
|
||||
if n != 0 {
|
||||
t.Fatal("deaktivierter trigger hat trotzdem gefeuert")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDelete_UnknownReturnsErrNotFound(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
ctx := context.Background()
|
||||
if err := Delete(ctx, e.Pool, "tester", "00000000-0000-0000-0000-000000000001"); err == nil || !strings.Contains(err.Error(), "nicht gefunden") {
|
||||
t.Fatalf("erwartet ErrNotFound, habe %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package trigger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
dmscrypto "gitea.perlbach24.de/scripte/nexarch/dms/internal/crypto"
|
||||
)
|
||||
|
||||
// Vault verpackt/entpackt das Webhook-Geheimnis mit dem Tenant-KEK
|
||||
// (Envelope-Verschluesselung wie FDN-09/FDN-22, kein Klartext-Fallback).
|
||||
type Vault interface {
|
||||
Seal(ctx context.Context, tenantSlug, plaintext string) (wrappedDEK, ciphertext []byte, err error)
|
||||
Open(ctx context.Context, tenantSlug string, wrappedDEK, ciphertext []byte) (string, error)
|
||||
}
|
||||
|
||||
// CryptoVault implementiert Vault ueber den vorhandenen dmscrypto.Service.
|
||||
type CryptoVault struct {
|
||||
Crypto *dmscrypto.Service
|
||||
}
|
||||
|
||||
func (v CryptoVault) Seal(ctx context.Context, tenantSlug, plaintext string) ([]byte, []byte, error) {
|
||||
env, err := v.Crypto.Seal(ctx, tenantSlug, bytes.NewReader([]byte(plaintext)))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
ciphertext, err := io.ReadAll(env.Ciphertext)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return env.WrappedDEK, ciphertext, nil
|
||||
}
|
||||
|
||||
func (v CryptoVault) Open(ctx context.Context, tenantSlug string, wrappedDEK, ciphertext []byte) (string, error) {
|
||||
r, err := v.Crypto.Open(ctx, tenantSlug, wrappedDEK, bytes.NewReader(ciphertext))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
plaintext, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("trigger: geheimnis entschluesseln: %w", err)
|
||||
}
|
||||
return string(plaintext), nil
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package trigger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const webhookTimeout = 10 * time.Second
|
||||
|
||||
var httpClient = &http.Client{
|
||||
Timeout: webhookTimeout,
|
||||
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
|
||||
}
|
||||
|
||||
// ValidateWebhookURL verlangt http(s) und lehnt Loopback-, private und
|
||||
// Link-Local-Ziele ab, sofern nicht ueber
|
||||
// NEXARCH_DMS_APP_TRIGGER_WEBHOOK_ALLOW_PRIVATE=true ausdruecklich erlaubt
|
||||
// (Entwicklungsbetrieb). Das ist eine Bestwissen-Pruefung anhand der im
|
||||
// Hostnamen angegebenen IP; ein DNS-Rebinding zur Anfragezeit wird dadurch
|
||||
// nicht verhindert (siehe Protokoll, Grenze wie bei F-08/F-22).
|
||||
func ValidateWebhookURL(raw string) error {
|
||||
u, err := url.Parse(strings.TrimSpace(raw))
|
||||
if err != nil {
|
||||
return fmt.Errorf("ungueltige url: %w", err)
|
||||
}
|
||||
if u.Scheme != "http" && u.Scheme != "https" {
|
||||
return fmt.Errorf("nur http/https erlaubt")
|
||||
}
|
||||
if u.Host == "" {
|
||||
return fmt.Errorf("host fehlt")
|
||||
}
|
||||
if allowPrivateTargets() {
|
||||
return nil
|
||||
}
|
||||
host := u.Hostname()
|
||||
if ip := net.ParseIP(host); ip != nil {
|
||||
if isDisallowedIP(ip) {
|
||||
return fmt.Errorf("ziel-ip %s ist ein lokales/privates netz, nicht erlaubt", ip)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
// Hostname statt IP: bestmoegliche Pruefung zur Anlagezeit; siehe Doc-Kommentar oben.
|
||||
ips, err := net.LookupIP(host)
|
||||
if err != nil {
|
||||
return nil // Auflösung schlaegt evtl. erst zur Sendezeit fehl; nicht blockieren.
|
||||
}
|
||||
for _, ip := range ips {
|
||||
if isDisallowedIP(ip) {
|
||||
return fmt.Errorf("host %s loest auf ein lokales/privates netz auf, nicht erlaubt", host)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func allowPrivateTargets() bool {
|
||||
return strings.EqualFold(strings.TrimSpace(os.Getenv("NEXARCH_DMS_APP_TRIGGER_WEBHOOK_ALLOW_PRIVATE")), "true")
|
||||
}
|
||||
|
||||
func isDisallowedIP(ip net.IP) bool {
|
||||
return ip.IsLoopback() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || ip.IsUnspecified()
|
||||
}
|
||||
|
||||
// webhookPayload ist der JSON-Koerper der Anfrage.
|
||||
type webhookPayload struct {
|
||||
Event string `json:"event"`
|
||||
DocumentID string `json:"document_id"`
|
||||
TriggerID string `json:"trigger_id"`
|
||||
OccurredAt string `json:"occurred_at"`
|
||||
}
|
||||
|
||||
// sendWebhook signiert den Koerper mit dem entschluesselten Geheimnis (falls
|
||||
// gesetzt) als Header X-Trigger-Signature (HMAC-SHA256, Hex) und sendet ihn.
|
||||
// Ein Nicht-2xx-Status gilt als Fehlschlag.
|
||||
func sendWebhook(ctx context.Context, targetURL string, secret string, payload webhookPayload) error {
|
||||
if err := ValidateWebhookURL(targetURL); err != nil {
|
||||
return fmt.Errorf("webhook-ziel abgelehnt: %w", err)
|
||||
}
|
||||
body, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, targetURL, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if secret != "" {
|
||||
mac := hmac.New(sha256.New, []byte(secret))
|
||||
mac.Write(body)
|
||||
req.Header.Set("X-Trigger-Signature", hex.EncodeToString(mac.Sum(nil)))
|
||||
}
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("webhook: anfrage fehlgeschlagen: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return fmt.Errorf("webhook: unerwarteter status %d", resp.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
// Package triggerapi haengt INT-20 (Trigger-/Automatisierungssystem) ein:
|
||||
// Verwaltung unter /triggers. Nutzt ausschliesslich internal/trigger, keine
|
||||
// eigene Fachlogik. Wie alle nicht dokument-gebundenen Routen (siehe
|
||||
// internal/autoruleapi) verlangt sie eine Identitaet (moduleauth) und die
|
||||
// Rechte dms.trigger.read/dms.trigger.write ueber die zentrale
|
||||
// Policy-Pruefung (x-permission in openapi.json); es gibt keine
|
||||
// Eigentuemerpruefung, weil Trigger nicht an ein Dokument gebunden sind.
|
||||
package triggerapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/authz"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trigger"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
pool *pgxpool.Pool
|
||||
vault trigger.Vault
|
||||
tenantSlug string
|
||||
}
|
||||
|
||||
func NewServer(pool *pgxpool.Pool, vault trigger.Vault, tenantSlug string) *Server {
|
||||
return &Server{pool: pool, vault: vault, tenantSlug: tenantSlug}
|
||||
}
|
||||
|
||||
func (s *Server) RegisterRoutes(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /triggers", s.handleCreate)
|
||||
mux.HandleFunc("GET /triggers", s.handleList)
|
||||
mux.HandleFunc("GET /triggers/{id}", s.handleGet)
|
||||
mux.HandleFunc("DELETE /triggers/{id}", s.handleDelete)
|
||||
mux.HandleFunc("POST /triggers/{id}/enable", s.handleSetEnabled(true))
|
||||
mux.HandleFunc("POST /triggers/{id}/disable", s.handleSetEnabled(false))
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, code, message string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"error": map[string]string{"code": code, "message": message}})
|
||||
}
|
||||
|
||||
func caller(r *http.Request) string {
|
||||
c, _ := authz.Identity(r)
|
||||
return c
|
||||
}
|
||||
|
||||
type dto struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Event string `json:"event"`
|
||||
FilenamePattern *string `json:"filename_pattern,omitempty"`
|
||||
DocumentTypeID *string `json:"document_type_id,omitempty"`
|
||||
ActionType string `json:"action_type"`
|
||||
ActionTagID *string `json:"action_tag_id,omitempty"`
|
||||
ActionDocumentTypeID *string `json:"action_document_type_id,omitempty"`
|
||||
WebhookURL *string `json:"webhook_url,omitempty"`
|
||||
Enabled bool `json:"enabled"`
|
||||
CreatedBy string `json:"created_by"`
|
||||
}
|
||||
|
||||
func toDTO(t trigger.Trigger) dto {
|
||||
return dto{ID: t.ID, Name: t.Name, Event: t.Event, FilenamePattern: t.FilenamePattern, DocumentTypeID: t.DocumentTypeID,
|
||||
ActionType: t.ActionType, ActionTagID: t.ActionTagID, ActionDocumentTypeID: t.ActionDocumentTypeID,
|
||||
WebhookURL: t.WebhookURL, Enabled: t.Enabled, CreatedBy: t.CreatedBy}
|
||||
}
|
||||
|
||||
type createRequest struct {
|
||||
Name string `json:"name"`
|
||||
Event string `json:"event"`
|
||||
FilenamePattern *string `json:"filename_pattern"`
|
||||
DocumentTypeID *string `json:"document_type_id"`
|
||||
ActionType string `json:"action_type"`
|
||||
ActionTagID *string `json:"action_tag_id"`
|
||||
ActionDocumentTypeID *string `json:"action_document_type_id"`
|
||||
WebhookURL *string `json:"webhook_url"`
|
||||
WebhookSecret string `json:"webhook_secret"`
|
||||
}
|
||||
|
||||
func (s *Server) handleCreate(w http.ResponseWriter, r *http.Request) {
|
||||
c := caller(r)
|
||||
if c == "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "fehlende identitaet")
|
||||
return
|
||||
}
|
||||
var req createRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 8192)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "ungueltiger anfragekoerper")
|
||||
return
|
||||
}
|
||||
t := trigger.Trigger{
|
||||
Name: req.Name, Event: req.Event, FilenamePattern: req.FilenamePattern, DocumentTypeID: req.DocumentTypeID,
|
||||
ActionType: req.ActionType, ActionTagID: req.ActionTagID, ActionDocumentTypeID: req.ActionDocumentTypeID,
|
||||
WebhookURL: req.WebhookURL,
|
||||
}
|
||||
out, err := trigger.Create(r.Context(), s.pool, s.vault, s.tenantSlug, c, t, req.WebhookSecret)
|
||||
if errors.Is(err, trigger.ErrInvalid) {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", err.Error())
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
_ = json.NewEncoder(w).Encode(toDTO(out))
|
||||
}
|
||||
|
||||
func (s *Server) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
list, err := trigger.List(r.Context(), s.pool)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]dto, 0, len(list))
|
||||
for _, t := range list {
|
||||
out = append(out, toDTO(t))
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(out)
|
||||
}
|
||||
|
||||
func (s *Server) handleGet(w http.ResponseWriter, r *http.Request) {
|
||||
t, err := trigger.Get(r.Context(), s.pool, r.PathValue("id"))
|
||||
if errors.Is(err, trigger.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "trigger nicht gefunden")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(toDTO(t))
|
||||
}
|
||||
|
||||
func (s *Server) handleDelete(w http.ResponseWriter, r *http.Request) {
|
||||
c := caller(r)
|
||||
if c == "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "fehlende identitaet")
|
||||
return
|
||||
}
|
||||
err := trigger.Delete(r.Context(), s.pool, c, r.PathValue("id"))
|
||||
if errors.Is(err, trigger.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "trigger nicht gefunden")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (s *Server) handleSetEnabled(enabled bool) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
c := caller(r)
|
||||
if c == "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request", "fehlende identitaet")
|
||||
return
|
||||
}
|
||||
err := trigger.SetEnabled(r.Context(), s.pool, c, r.PathValue("id"), enabled)
|
||||
if errors.Is(err, trigger.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "trigger nicht gefunden")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", err.Error())
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
package triggerapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/tag"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trigger"
|
||||
)
|
||||
|
||||
// memVault verpackt reversibel (Praefix) statt echt zu verschluesseln, damit
|
||||
// der Test einen echten Seal/Open-Roundtrip pruefen kann, ohne einen
|
||||
// KEK-Provider hochzufahren.
|
||||
type memVault struct{}
|
||||
|
||||
var _ trigger.Vault = memVault{}
|
||||
|
||||
func (memVault) Seal(_ context.Context, _, plaintext string) ([]byte, []byte, error) {
|
||||
return []byte("dek"), []byte("enc:" + plaintext), nil
|
||||
}
|
||||
func (memVault) Open(_ context.Context, _ string, _, ciphertext []byte) (string, error) {
|
||||
return strings.TrimPrefix(string(ciphertext), "enc:"), nil
|
||||
}
|
||||
|
||||
func TestAPI_ManageTriggersOverHTTP(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
tg, err := tag.Create(t.Context(), e.Pool, "rechnung", "", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
NewServer(e.Pool, memVault{}, testenv.TenantSlug).RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
do := func(method, path string, body any) (int, []byte) {
|
||||
var r *http.Request
|
||||
var err error
|
||||
if body != nil {
|
||||
b, _ := json.Marshal(body)
|
||||
r, err = http.NewRequest(method, srv.URL+path, bytes.NewReader(b))
|
||||
} else {
|
||||
r, err = http.NewRequest(method, srv.URL+path, nil)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(r)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
var buf bytes.Buffer
|
||||
_, _ = buf.ReadFrom(resp.Body)
|
||||
return resp.StatusCode, buf.Bytes()
|
||||
}
|
||||
caller := "?caller=" + e.UserID
|
||||
trg := map[string]any{"name": "Rechnung", "event": "document.uploaded", "filename_pattern": "(?i)rechnung", "action_type": "set_tag", "action_tag_id": tg.ID}
|
||||
|
||||
if status, _ := do(http.MethodPost, "/triggers", trg); status != http.StatusBadRequest {
|
||||
t.Errorf("ohne caller: %d, want 400", status)
|
||||
}
|
||||
status, body := do(http.MethodPost, "/triggers"+caller, trg)
|
||||
if status != http.StatusCreated {
|
||||
t.Fatalf("anlegen: %d %s", status, body)
|
||||
}
|
||||
var created map[string]any
|
||||
_ = json.Unmarshal(body, &created)
|
||||
id := created["id"].(string)
|
||||
if created["enabled"] != true {
|
||||
t.Errorf("neuer trigger sollte aktiviert sein: %+v", created)
|
||||
}
|
||||
|
||||
bad := map[string]any{"name": "kaputt", "event": "document.uploaded", "filename_pattern": "([", "action_type": "set_tag", "action_tag_id": tg.ID}
|
||||
if status, _ := do(http.MethodPost, "/triggers"+caller, bad); status != http.StatusBadRequest {
|
||||
t.Errorf("ungueltiges muster: %d, want 400", status)
|
||||
}
|
||||
|
||||
status, body = do(http.MethodGet, "/triggers", nil)
|
||||
var list []map[string]any
|
||||
_ = json.Unmarshal(body, &list)
|
||||
if status != http.StatusOK || len(list) != 1 {
|
||||
t.Errorf("liste: %d %s", status, body)
|
||||
}
|
||||
if _, hasSecret := list[0]["webhook_secret"]; hasSecret {
|
||||
t.Error("die liste darf kein geheimnis enthalten")
|
||||
}
|
||||
|
||||
if status, _ := do(http.MethodGet, "/triggers/"+id, nil); status != http.StatusOK {
|
||||
t.Errorf("lesen: %d", status)
|
||||
}
|
||||
if status, _ := do(http.MethodGet, "/triggers/00000000-0000-0000-0000-000000000001", nil); status != http.StatusNotFound {
|
||||
t.Errorf("unbekannt: %d, want 404", status)
|
||||
}
|
||||
|
||||
if status, _ := do(http.MethodPost, "/triggers/"+id+"/disable"+caller, nil); status != http.StatusNoContent {
|
||||
t.Errorf("deaktivieren: %d", status)
|
||||
}
|
||||
_, body = do(http.MethodGet, "/triggers/"+id, nil)
|
||||
var after map[string]any
|
||||
_ = json.Unmarshal(body, &after)
|
||||
if after["enabled"] != false {
|
||||
t.Errorf("trigger sollte deaktiviert sein: %+v", after)
|
||||
}
|
||||
if status, _ := do(http.MethodPost, "/triggers/"+id+"/enable"+caller, nil); status != http.StatusNoContent {
|
||||
t.Errorf("aktivieren: %d", status)
|
||||
}
|
||||
|
||||
if status, _ := do(http.MethodDelete, "/triggers/"+id+caller, nil); status != http.StatusNoContent {
|
||||
t.Errorf("loeschen: %d", status)
|
||||
}
|
||||
if status, _ := do(http.MethodGet, "/triggers/"+id, nil); status != http.StatusNotFound {
|
||||
t.Errorf("nach loeschen: %d, want 404", status)
|
||||
}
|
||||
if status, _ := do(http.MethodDelete, "/triggers/"+id+caller, nil); status != http.StatusNotFound {
|
||||
t.Errorf("doppelt loeschen: %d, want 404", status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPI_WebhookSecretNeverReturned(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
mux := http.NewServeMux()
|
||||
NewServer(e.Pool, memVault{}, testenv.TenantSlug).RegisterRoutes(mux)
|
||||
srv := httptest.NewServer(mux)
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
url := "https://example.test/hook"
|
||||
trg := map[string]any{"name": "Hook", "event": "document.uploaded", "action_type": "webhook", "webhook_url": url, "webhook_secret": "top-secret"}
|
||||
b, _ := json.Marshal(trg)
|
||||
resp, err := http.Post(srv.URL+"/triggers?caller="+e.UserID, "application/json", bytes.NewReader(b))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
var out bytes.Buffer
|
||||
_, _ = out.ReadFrom(resp.Body)
|
||||
if bytes.Contains(out.Bytes(), []byte("top-secret")) {
|
||||
t.Fatalf("antwort enthaelt das geheimnis im klartext: %s", out.String())
|
||||
}
|
||||
|
||||
var wrapped, enc []byte
|
||||
if err := e.Pool.QueryRow(t.Context(), `SELECT webhook_secret_wrapped_dek, webhook_secret_enc FROM triggers WHERE webhook_url = $1`, url).Scan(&wrapped, &enc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if bytes.Equal(enc, []byte("top-secret")) {
|
||||
t.Fatal("geheimnis liegt unverschluesselt in der datenbank")
|
||||
}
|
||||
if len(wrapped) == 0 {
|
||||
t.Fatal("kein verpackter schluessel gespeichert")
|
||||
}
|
||||
}
|
||||
@@ -44,6 +44,12 @@ type AutoRuleApplier interface {
|
||||
Apply(ctx context.Context, documentID string) error
|
||||
}
|
||||
|
||||
// TriggerFirer ist die schmale Schnittstelle zu INT-20 (internal/trigger):
|
||||
// Webhook/Tag/Typ-Aktionen nach dem Ereignis document.uploaded.
|
||||
type TriggerFirer interface {
|
||||
Fire(ctx context.Context, event, documentID string) error
|
||||
}
|
||||
|
||||
// SearchIndexEnqueuer ist die schmale Schnittstelle zu SRC-02
|
||||
// (internal/searchsync) — analoges Muster zu ThumbnailEnqueuer.
|
||||
type SearchIndexEnqueuer interface {
|
||||
@@ -72,6 +78,7 @@ type Service struct {
|
||||
autoRules AutoRuleApplier
|
||||
searchIndex SearchIndexEnqueuer
|
||||
einvoice EInvoiceEnqueuer
|
||||
triggers TriggerFirer
|
||||
}
|
||||
|
||||
func NewService(pool *pgxpool.Pool, sessions *SessionStore, staging *Staging, storageSvc *storage.Service, cryptoSvc *dmscrypto.Service, validator *Validator, tenantSlug string) *Service {
|
||||
@@ -111,6 +118,11 @@ func (s *Service) SetAutoRuleApplier(a AutoRuleApplier) {
|
||||
s.autoRules = a
|
||||
}
|
||||
|
||||
// SetTriggerFirer aktiviert INT-20 nach erfolgreichem Complete (optional).
|
||||
func (s *Service) SetTriggerFirer(t TriggerFirer) {
|
||||
s.triggers = t
|
||||
}
|
||||
|
||||
// SetEInvoiceEnqueuer aktiviert MET-06 nach erfolgreichem Complete (optional).
|
||||
func (s *Service) SetEInvoiceEnqueuer(e EInvoiceEnqueuer) {
|
||||
s.einvoice = e
|
||||
@@ -323,6 +335,13 @@ func (s *Service) Complete(ctx context.Context, sessionID string, confirmDuplica
|
||||
log.Printf("upload: automatische regeln fuer dokument %q fehlgeschlagen: %v", documentID, err)
|
||||
}
|
||||
}
|
||||
// INT-20: Trigger sind eine Nebenwirkung wie die Auto-Regeln; ein Fehler
|
||||
// darf einen abgeschlossenen Upload nicht rueckwirkend scheitern lassen.
|
||||
if s.triggers != nil {
|
||||
if err := s.triggers.Fire(ctx, "document.uploaded", documentID); err != nil {
|
||||
log.Printf("upload: trigger fuer dokument %q fehlgeschlagen: %v", documentID, err)
|
||||
}
|
||||
}
|
||||
// DOC-04: Vorschau-Erzeugung ist eine Nebenwirkung, kein Bestandteil
|
||||
// des Upload-Erfolgs selbst — ein Fehlschlag hier darf einen bereits
|
||||
// abgeschlossenen Upload nicht rückwirkend als fehlgeschlagen melden.
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package workerruntime
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/integrity"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
)
|
||||
|
||||
func TestIntegrityJobIsRegisteredOnlyWhenConfigured(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
queue := jobqueue.NewQueue(e.Pool, 5*time.Minute)
|
||||
noTools := func(string) (string, error) { return "", errors.New("nicht gefunden") }
|
||||
base := Deps{Pool: e.Pool, Storage: e.Storage, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug, Queue: queue, LookPath: noTools}
|
||||
|
||||
rt := New(base)
|
||||
if _, ok := rt.Handlers[integrity.JobType]; ok {
|
||||
t.Fatal("integrity.verify ohne konfiguration registriert")
|
||||
}
|
||||
if !strings.Contains(strings.Join(rt.Notices, "\n"), integrity.JobType) {
|
||||
t.Errorf("hinweis auf %s fehlt: %v", integrity.JobType, rt.Notices)
|
||||
}
|
||||
|
||||
base.Integrity = &integrity.Config{Mode: integrity.ModeFull, MaxDuration: 2 * time.Hour}
|
||||
rt = New(base)
|
||||
if _, ok := rt.Handlers[integrity.JobType]; !ok {
|
||||
t.Fatal("integrity.verify trotz konfiguration nicht registriert")
|
||||
}
|
||||
if l := rt.Limits[integrity.JobType]; l.MaxDuration != 2*time.Hour+time.Minute {
|
||||
t.Fatalf("job-limits: %+v", l)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrityJobRunsThroughDispatcherToFinalState(t *testing.T) {
|
||||
e := testenv.New(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
queue := jobqueue.NewQueue(e.Pool, 5*time.Minute)
|
||||
noTools := func(string) (string, error) { return "", errors.New("nicht gefunden") }
|
||||
rt := New(Deps{Pool: e.Pool, Storage: e.Storage, Crypto: e.Crypto, TenantSlug: testenv.TenantSlug, Queue: queue, LookPath: noTools,
|
||||
Integrity: &integrity.Config{Mode: integrity.ModeFull, MaxDuration: time.Minute}})
|
||||
|
||||
_, rev := e.UploadDoc(t, "a.txt", "text/plain", []byte("inhalt"))
|
||||
if _, err := e.Pool.Exec(ctx, `UPDATE file_revisions SET created_at = now() - interval '1 hour'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok, err := integrity.EnqueueIfDue(ctx, e.Pool, queue, integrity.Schedule{Interval: time.Hour}, time.Now()); err != nil || !ok {
|
||||
t.Fatalf("einreihen: %v %v", ok, err)
|
||||
}
|
||||
go rt.Run(ctx, 1, "test", 20*time.Millisecond)
|
||||
|
||||
waitFor(t, e, "integrity.verify abgeschlossen", 20*time.Second, func() bool {
|
||||
return count(t, e, `SELECT count(*) FROM processing_jobs WHERE job_type = $1 AND status = 'succeeded'`, integrity.JobType) == 1
|
||||
})
|
||||
if n := count(t, e, `SELECT count(*) FROM integrity_results WHERE revision_id = $1 AND status = 'ok'`, rev); n != 1 {
|
||||
t.Fatalf("ergebnis der revision: %d, erwartet ok", n)
|
||||
}
|
||||
}
|
||||
@@ -22,15 +22,18 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/downloadapi"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/einvoice"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/export"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/integrity"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/mailimport"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/malwarescan"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/ocr"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/pdfaconform"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/pdfconvert"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/plaintext"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/searchsync"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/storage"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/thumbnail"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/trigger"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/upload"
|
||||
)
|
||||
|
||||
@@ -57,6 +60,10 @@ type Deps struct {
|
||||
MailImport *mailimport.Importer
|
||||
// EInvoiceValidator ist der KoSIT-Sidecar; nil = Ergebnis not_checked.
|
||||
EInvoiceValidator *einvoice.Validator
|
||||
// PDFAValidator ist der veraPDF-Sidecar (SRC-14); nil = Ergebnis not_checked.
|
||||
PDFAValidator *pdfaconform.Validator
|
||||
// Integrity ist die Objektpruefung (FDN-23); nil = integrity.verify wird nicht abgeholt.
|
||||
Integrity *integrity.Config
|
||||
}
|
||||
|
||||
func (d Deps) lookPath(name string) error {
|
||||
@@ -72,6 +79,8 @@ func (d Deps) lookPath(name string) error {
|
||||
type Runtime struct {
|
||||
Handlers map[string]jobqueue.Handler
|
||||
Notices []string
|
||||
// Limits sind jobtypspezifische Ressourcengrenzen (FDN-11), die Run an die Worker weitergibt.
|
||||
Limits map[string]jobqueue.Limits
|
||||
// Middleware umhuellt den Dispatcher (z. B. Fehlerverfolgung); optional.
|
||||
Middleware func(jobqueue.Handler) jobqueue.Handler
|
||||
queue *jobqueue.Queue
|
||||
@@ -79,10 +88,11 @@ type Runtime struct {
|
||||
|
||||
// New registriert alle Jobtypen, deren Voraussetzungen erfuellt sind.
|
||||
func New(d Deps) *Runtime {
|
||||
rt := &Runtime{Handlers: map[string]jobqueue.Handler{}, queue: d.Queue}
|
||||
rt := &Runtime{Handlers: map[string]jobqueue.Handler{}, Limits: map[string]jobqueue.Limits{}, queue: d.Queue}
|
||||
skip := func(jobType, why string) {
|
||||
rt.Notices = append(rt.Notices, fmt.Sprintf("jobtyp %s wird nicht abgeholt: %s", jobType, why))
|
||||
}
|
||||
triggers := &trigger.Adapter{Pool: d.Pool, Vault: trigger.CryptoVault{Crypto: d.Crypto}, TenantSlug: d.TenantSlug}
|
||||
|
||||
if err := d.lookPath("pdftoppm"); err != nil {
|
||||
skip(thumbnail.JobType, "pdftoppm nicht gefunden")
|
||||
@@ -111,7 +121,7 @@ func New(d Deps) *Runtime {
|
||||
if d.SearchIndex == nil {
|
||||
skip(searchsync.JobType, "kein Suchindex (Manticore) konfiguriert; Dokumente sind nicht ueber die Suche auffindbar")
|
||||
} else {
|
||||
sy := &searchsync.Syncer{Pool: d.Pool, Index: d.SearchIndex, Text: searchsync.MultiText{ocr.TextStore{Pool: d.Pool}, einvoice.TextStore{Pool: d.Pool}, plaintext.TextStore{Pool: d.Pool, Storage: d.Storage, Crypto: d.Crypto, TenantSlug: d.TenantSlug, MaxBytes: d.PlaintextMaxBytes, MaxChars: d.PlaintextMaxChars}}}
|
||||
sy := &searchsync.Syncer{Pool: d.Pool, Index: d.SearchIndex, Text: searchsync.MultiText{ocr.TextStore{Pool: d.Pool, Crypto: d.Crypto, TenantSlug: d.TenantSlug}, einvoice.TextStore{Pool: d.Pool, Crypto: d.Crypto, TenantSlug: d.TenantSlug}, plaintext.TextStore{Pool: d.Pool, Storage: d.Storage, Crypto: d.Crypto, TenantSlug: d.TenantSlug, MaxBytes: d.PlaintextMaxBytes, MaxChars: d.PlaintextMaxChars}}}
|
||||
rt.Handlers[searchsync.JobType] = sy.Handler
|
||||
}
|
||||
|
||||
@@ -121,12 +131,24 @@ func New(d Deps) *Runtime {
|
||||
rt.Handlers[mailimport.JobType] = d.MailImport.Handler
|
||||
}
|
||||
|
||||
if d.Integrity == nil {
|
||||
skip(integrity.JobType, "keine Integritaetspruefung konfiguriert (NEXARCH_DMS_WORKER_INTEGRITY_INTERVAL_SECONDS)")
|
||||
} else {
|
||||
iv := &integrity.Verifier{Pool: d.Pool, Storage: d.Storage, Crypto: d.Crypto, TenantSlug: d.TenantSlug, Config: *d.Integrity}
|
||||
rt.Handlers[integrity.JobType] = iv.Handler
|
||||
rt.Limits[integrity.JobType] = d.Integrity.JobLimits()
|
||||
}
|
||||
|
||||
ei := &einvoice.Processor{Validator: d.EInvoiceValidator, Pool: d.Pool, Storage: d.Storage, Crypto: d.Crypto, TenantSlug: d.TenantSlug}
|
||||
if _, ok := rt.Handlers[searchsync.JobType]; ok {
|
||||
ei.OnRead = func(ctx context.Context, documentID, _ string) error {
|
||||
return searchsync.Enqueue(ctx, d.Queue, documentID)
|
||||
}
|
||||
}
|
||||
// INT-20: document.einvoice_checked nach jeder Konformitaetspruefung (Erst- und Neupruefung).
|
||||
ei.OnConformanceChecked = func(ctx context.Context, documentID, _ string) error {
|
||||
return triggers.Fire(ctx, trigger.EventEInvoiceChecked, documentID)
|
||||
}
|
||||
rt.Handlers[einvoice.JobType] = ei.Handler
|
||||
rt.Handlers[einvoice.JobTypeValidate] = ei.HandlerValidate
|
||||
|
||||
@@ -147,15 +169,29 @@ func New(d Deps) *Runtime {
|
||||
proc = &ocr.Processor{Engine: ocr.DefaultEngine()}
|
||||
}
|
||||
proc.Pool, proc.Storage, proc.Crypto, proc.TenantSlug, proc.Queue = d.Pool, d.Storage, d.Crypto, d.TenantSlug, d.Queue
|
||||
if _, ok := rt.Handlers[searchsync.JobType]; ok && proc.OnText == nil {
|
||||
_, syncIndex := rt.Handlers[searchsync.JobType]
|
||||
if proc.OnText == nil {
|
||||
proc.OnText = func(ctx context.Context, documentID, _ string) error {
|
||||
return searchsync.Enqueue(ctx, d.Queue, documentID)
|
||||
if syncIndex {
|
||||
if err := searchsync.Enqueue(ctx, d.Queue, documentID); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// INT-20: document.ocr_completed nach Vorliegen des Volltexts.
|
||||
return triggers.Fire(ctx, trigger.EventOCRCompleted, documentID)
|
||||
}
|
||||
}
|
||||
if proc.OnPDFACreated == nil {
|
||||
proc.OnPDFACreated = func(ctx context.Context, documentID, revisionID string) error {
|
||||
return pdfaconform.Enqueue(ctx, d.Pool, d.Queue, documentID, revisionID)
|
||||
}
|
||||
}
|
||||
for _, jt := range ocr.JobTypes {
|
||||
rt.Handlers[jt] = proc.Handle
|
||||
}
|
||||
}
|
||||
pc := &pdfaconform.Processor{Validator: d.PDFAValidator, Pool: d.Pool, Storage: d.Storage, Crypto: d.Crypto, TenantSlug: d.TenantSlug}
|
||||
rt.Handlers[pdfaconform.JobType] = pc.Handler
|
||||
return rt
|
||||
}
|
||||
|
||||
@@ -194,9 +230,13 @@ func (r *Runtime) Run(ctx context.Context, n int, workerPrefix string, poll time
|
||||
if r.Middleware != nil {
|
||||
handler = r.Middleware(handler)
|
||||
}
|
||||
var opts []jobqueue.WorkerOption
|
||||
for jt, l := range r.Limits {
|
||||
opts = append(opts, jobqueue.WithJobTypeLimits(jt, l))
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < n; i++ {
|
||||
w := jobqueue.NewWorker(r.queue, fmt.Sprintf("%s-%d", workerPrefix, i+1), types, poll, handler)
|
||||
w := jobqueue.NewWorker(r.queue, fmt.Sprintf("%s-%d", workerPrefix, i+1), types, poll, handler, opts...)
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); w.Run(ctx) }()
|
||||
}
|
||||
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -16,6 +18,7 @@ import (
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/jobqueue"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/malwarescan"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/ocr"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/pdfaconform"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testcorpus"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/testenv"
|
||||
"gitea.perlbach24.de/scripte/nexarch/dms/internal/thumbnail"
|
||||
@@ -149,8 +152,10 @@ func TestMissingToolsAndConfigurationAreReportedAndNotEnqueued(t *testing.T) {
|
||||
t.Errorf("hinweis auf %q fehlt:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
if types := rt.JobTypes(); len(types) != 3 || types[0] != einvoice.JobType || types[1] != einvoice.JobTypeValidate || types[2] != export.JobType {
|
||||
t.Errorf("registrierte jobtypen: %v, erwartet nur %s, %s und %s", types, einvoice.JobType, einvoice.JobTypeValidate, export.JobType)
|
||||
want := []string{einvoice.JobType, einvoice.JobTypeValidate, export.JobType, pdfaconform.JobType}
|
||||
sort.Strings(want)
|
||||
if types := rt.JobTypes(); !reflect.DeepEqual(types, want) {
|
||||
t.Errorf("registrierte jobtypen: %v, erwartet nur %v", types, want)
|
||||
}
|
||||
|
||||
uploadPDF(t, e, rt, queue, "x.pdf", testcorpus.GeneratePDF("x", []string{"Testinhalt Zeile eins", "Zeile zwei"}))
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Chiffretext laesst sich in SQL nicht entschluesseln: die Rueckwaertsmigration verweigert sich,
|
||||
-- solange verschluesselte Texte vorhanden sind (sonst gingen sie stillschweigend verloren).
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM revision_ocr_pages WHERE text_enc IS NOT NULL)
|
||||
OR EXISTS (SELECT 1 FROM revision_text_index WHERE text_enc IS NOT NULL)
|
||||
OR EXISTS (SELECT 1 FROM revision_einvoice WHERE search_text_enc IS NOT NULL) THEN
|
||||
RAISE EXCEPTION 'FDN-22: verschluesselte textkopien vorhanden, zuerst entschluesseln (textcrypt-migrate -decrypt)';
|
||||
END IF;
|
||||
END
|
||||
$$;
|
||||
|
||||
ALTER TABLE revision_einvoice DROP CONSTRAINT revision_einvoice_no_plaintext_with_cipher,
|
||||
DROP COLUMN search_text_enc, DROP COLUMN wrapped_dek;
|
||||
ALTER TABLE revision_text_index DROP CONSTRAINT revision_text_index_no_plaintext_with_cipher,
|
||||
DROP COLUMN text_enc, DROP COLUMN wrapped_dek;
|
||||
ALTER TABLE revision_ocr_pages DROP CONSTRAINT revision_ocr_pages_no_plaintext_with_cipher,
|
||||
DROP COLUMN text_enc;
|
||||
ALTER TABLE revision_ocr DROP COLUMN text_wrapped_dek;
|
||||
@@ -0,0 +1,18 @@
|
||||
-- FDN-22: Textkopien verschluesselt ablegen. Bestehende Klartextzeilen bleiben lesbar
|
||||
-- (text_enc IS NULL) und werden per textcrypt-migrate nachverschluesselt; neue Zeilen
|
||||
-- tragen nur Chiffretext, die Klartextspalte bleibt leer.
|
||||
ALTER TABLE revision_ocr ADD COLUMN text_wrapped_dek BYTEA;
|
||||
|
||||
ALTER TABLE revision_ocr_pages
|
||||
ADD COLUMN text_enc BYTEA,
|
||||
ADD CONSTRAINT revision_ocr_pages_no_plaintext_with_cipher CHECK (text_enc IS NULL OR text = '');
|
||||
|
||||
ALTER TABLE revision_text_index
|
||||
ADD COLUMN wrapped_dek BYTEA,
|
||||
ADD COLUMN text_enc BYTEA,
|
||||
ADD CONSTRAINT revision_text_index_no_plaintext_with_cipher CHECK (text_enc IS NULL OR text = '');
|
||||
|
||||
ALTER TABLE revision_einvoice
|
||||
ADD COLUMN wrapped_dek BYTEA,
|
||||
ADD COLUMN search_text_enc BYTEA,
|
||||
ADD CONSTRAINT revision_einvoice_no_plaintext_with_cipher CHECK (search_text_enc IS NULL OR search_text = '');
|
||||
@@ -0,0 +1,2 @@
|
||||
DROP TABLE IF EXISTS integrity_results;
|
||||
DROP TABLE IF EXISTS integrity_runs;
|
||||
@@ -0,0 +1,25 @@
|
||||
CREATE TABLE integrity_runs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
finished_at TIMESTAMPTZ,
|
||||
mode TEXT NOT NULL CHECK (mode IN ('sample', 'full')),
|
||||
status TEXT NOT NULL DEFAULT 'running' CHECK (status IN ('running', 'completed', 'partial', 'failed')),
|
||||
checked INT NOT NULL DEFAULT 0,
|
||||
ok INT NOT NULL DEFAULT 0,
|
||||
mismatch INT NOT NULL DEFAULT 0,
|
||||
missing INT NOT NULL DEFAULT 0,
|
||||
unreadable INT NOT NULL DEFAULT 0,
|
||||
error INT NOT NULL DEFAULT 0,
|
||||
note TEXT NOT NULL DEFAULT ''
|
||||
);
|
||||
CREATE INDEX idx_integrity_runs_started_at ON integrity_runs(started_at);
|
||||
|
||||
CREATE TABLE integrity_results (
|
||||
revision_id UUID PRIMARY KEY REFERENCES file_revisions(id) ON DELETE CASCADE,
|
||||
run_id UUID REFERENCES integrity_runs(id) ON DELETE SET NULL,
|
||||
status TEXT NOT NULL CHECK (status IN ('ok', 'mismatch', 'missing', 'unreadable', 'error')),
|
||||
detail TEXT NOT NULL DEFAULT '',
|
||||
checked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
last_ok_at TIMESTAMPTZ
|
||||
);
|
||||
CREATE INDEX idx_integrity_results_checked_at ON integrity_results(checked_at);
|
||||
@@ -0,0 +1 @@
|
||||
DROP TABLE IF EXISTS share_links;
|
||||
@@ -0,0 +1,21 @@
|
||||
-- DOC-22: externe Freigabelinks. token_hash/password_hash tragen nie den
|
||||
-- Klartext (Token bzw. Passwort) -- nur SHA-256 des Tokens bzw. ein
|
||||
-- bcrypt-Hash des Passworts liegen in der Datenbank.
|
||||
CREATE TABLE share_links (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
|
||||
token_hash BYTEA NOT NULL,
|
||||
password_hash TEXT,
|
||||
download_limit INT,
|
||||
download_count INT NOT NULL DEFAULT 0,
|
||||
failed_attempts INT NOT NULL DEFAULT 0,
|
||||
locked_until TIMESTAMPTZ,
|
||||
created_by UUID NOT NULL REFERENCES users(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
revoked_at TIMESTAMPTZ,
|
||||
CHECK (download_limit IS NULL OR download_limit > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX idx_share_links_token_hash ON share_links(token_hash);
|
||||
CREATE INDEX idx_share_links_document_id ON share_links(document_id);
|
||||
@@ -0,0 +1 @@
|
||||
DROP TABLE IF EXISTS revision_pdfa_conformance;
|
||||
@@ -0,0 +1,17 @@
|
||||
-- SRC-14: Konformitaetsergebnis des PDF/A-Derivats je Revision (veraPDF).
|
||||
-- Jede Pruefung ist eine eigene Zeile (Muster wie 0029_einvoice_conformance);
|
||||
-- eine Neupruefung nach Werkzeug-Update haengt an, das aeltere Ergebnis bleibt lesbar.
|
||||
CREATE TABLE revision_pdfa_conformance (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
revision_id UUID NOT NULL REFERENCES file_revisions(id) ON DELETE CASCADE,
|
||||
status TEXT NOT NULL CHECK (status IN ('compliant', 'non_compliant', 'not_checked')),
|
||||
profile TEXT NOT NULL DEFAULT '',
|
||||
validator_version TEXT NOT NULL DEFAULT '',
|
||||
report_sha256 TEXT NOT NULL DEFAULT '',
|
||||
reason TEXT NOT NULL DEFAULT '',
|
||||
messages JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
checked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CHECK (status = 'not_checked' OR (report_sha256 <> '' AND validator_version <> ''))
|
||||
);
|
||||
|
||||
CREATE INDEX idx_revision_pdfa_conformance_latest ON revision_pdfa_conformance (revision_id, checked_at DESC);
|
||||
@@ -0,0 +1,2 @@
|
||||
DROP TABLE trigger_runs;
|
||||
DROP TABLE triggers;
|
||||
@@ -0,0 +1,39 @@
|
||||
-- INT-20: Trigger-/Automatisierungssystem. Ereignis (Upload abgeschlossen,
|
||||
-- Texterkennung fertig, E-Rechnung geprueft) plus optionale Dateinamen-
|
||||
-- Bedingung loesen eine Aktion aus (Webhook, Tag setzen, Dokumententyp
|
||||
-- setzen). Das Webhook-Geheimnis liegt wie der Text aus FDN-22 nur
|
||||
-- verschluesselt vor (Envelope mit Tenant-KEK).
|
||||
CREATE TABLE triggers (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name TEXT NOT NULL,
|
||||
event TEXT NOT NULL
|
||||
CHECK (event IN ('document.uploaded', 'document.ocr_completed', 'document.einvoice_checked')),
|
||||
filename_pattern TEXT,
|
||||
document_type_id UUID REFERENCES document_types(id) ON DELETE SET NULL,
|
||||
action_type TEXT NOT NULL CHECK (action_type IN ('webhook', 'set_tag', 'set_type')),
|
||||
action_tag_id UUID REFERENCES tags(id) ON DELETE CASCADE,
|
||||
action_document_type_id UUID REFERENCES document_types(id) ON DELETE CASCADE,
|
||||
webhook_url TEXT,
|
||||
webhook_secret_wrapped_dek BYTEA,
|
||||
webhook_secret_enc BYTEA,
|
||||
enabled BOOLEAN NOT NULL DEFAULT true,
|
||||
-- created_by ist wie der Audit-Akteur (audittrail) ein Freitext-Interimsakteur,
|
||||
-- kein Fremdschluessel auf users: Systemvorgaenge/Tests kennen keine Benutzer-UUID.
|
||||
created_by TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_triggers_event_enabled ON triggers(event) WHERE enabled;
|
||||
|
||||
CREATE TABLE trigger_runs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
trigger_id UUID NOT NULL REFERENCES triggers(id) ON DELETE CASCADE,
|
||||
document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
|
||||
event TEXT NOT NULL,
|
||||
status TEXT NOT NULL CHECK (status IN ('ok', 'failed')),
|
||||
error TEXT,
|
||||
fired_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_trigger_runs_trigger ON trigger_runs(trigger_id, fired_at DESC);
|
||||
Reference in new issue
Block a user