FDN-01: repository & projektgerüst

Git-Repository für bestehenden archivdms-Code initialisiert, Branch-/Commit-Konvention (feature/<ticket>-<slug>-Branches, Ticket-Prefix in Commit-Nachricht) etabliert.
This commit is contained in:
2026-08-11 21:27:53 +02:00
parent 40ed80da71
commit 9a24ea29e1
274 changed files with 53708 additions and 0 deletions
@@ -0,0 +1,22 @@
---
name: project-ollama-integration-plan
description: Plan für lokale Ollama-Anbindung (Metadaten-Vorschläge + OCR-Textkorrektur) in archivdms, Stand 2026-07-16
metadata:
type: project
---
Plan verabschiedet für Ollama-Integration (kein Code, nur Architektur), Ziel: lokales LLM für (1) Metadaten-Vorschläge als dritter Provider neben heuristic/anthropic, (2) nachträgliche OCR-Textkorrektur.
Kernentscheidungen:
- Modell: `qwen2.5:1.5b-instruct-q4_K_M` (~1-1.2GB RAM), Eskalation auf 3B nur falls Qualität nicht reicht. Server hat nur 4GB RAM/4 Kerne/keine GPU.
- systemd-Hardening: MemoryMax=2200M, CPUQuota=250%, OOMScoreAdjust=500, bindet nur an 127.0.0.1:11434.
- OCR-Korrektur läuft NICHT automatisch im Upload-Pipeline (Ressourcenrisiko), sondern über manuellen Endpoint `POST /api/documents/{id}/correct-ocr-text`, analog zu bestehendem `/reprocess`-Muster.
- GoBD: Original-OCR-Text (`ocr_text`) wird nie überschrieben, Korrektur landet in neuem Feld `ocr_text_corrected`, Audit-Log-Pflicht pro Korrekturlauf.
- Provider-Fehlerverhalten: bei Ollama nicht erreichbar → Fehler an Frontend, KEIN stiller Fallback auf heuristic (Nachvollziehbarkeit, welcher Provider geantwortet hat).
- Config-Pattern: `llm.ollama.enabled/base_url/model/timeout_seconds` in config.yml, no-op wenn disabled — exakt wie `index.manticore_dsn`-Pattern.
- Pro-Tenant-Schalter bewusst NICHT gebaut (Tag 1) — Ollama läuft als ein Prozess pro Server, kein echtes Isolationsmodell dahinter, wäre nur Schein-Kontrolle. Erst bei echter Mandantentrennung mit unterschiedlichen Compliance-Anforderungen nachrüsten.
- Reihenfolge: 1) Ollama-Server-Setup (devops-deploy), 2) Metadaten-Provider (backend-dev), 3) OCR-Korrektur-Endpoint (baut auf Client aus Schritt 2 auf, plus Migration für ocr_text_corrected-Spalte).
**Why:** User hat sich bewusst für kleines lokales Modell trotz 4GB-RAM-Warnung entschieden (GoBD/Datenschutz — Belege dürfen Server nicht verlassen), Provider-Abstraktion (`metadata_suggestions.provider`-Spalte) existierte schon konzeptionell aus früherer Session, nur `heuristic` war je gebaut.
**How to apply:** Wenn Umsetzung (backend-dev/devops-deploy) ansteht, diesen Plan als Grundlage nehmen, nicht neu verhandeln, außer der User ändert explizit etwas. Verwandt: [[project_archivdms_status]], [[project_nil_slice_json_pattern]].
@@ -0,0 +1,4 @@
- [Deskew-Vorverarbeitung](project_deskew_preprocessing.md) — ImageMagick `-deskew 40%` vor OSD-Rotation, Server-Paket `imagemagick` (nicht nur `-common`)
- [Titel-Heuristik + OSD Rotate:0-Lücke](project_title_heuristic_and_osd_zero_rotate_gap.md) — Alnum-Ratio-Filter statt "längste Zeile", rotateForOSD prüft Konfidenz nicht bei degrees==0
- [Deskew-Border-Trick negativ getestet](project_deskew_border_trick_tested_negative.md) — weißer Rand vor -deskew half nicht (Artefakt-Winkel), verworfen, nicht wieder vorschlagen
- [Deskew-Deaktivierung für Fotos negativ getestet](project_deskew_disable_for_photos_tested_negative.md) — gemischt (Doc7 stark schlechter), verworfen; runTesseract() ist einziger Aufrufpfad, keine Foto/PDF-Pipeline-Trennung vorhanden
@@ -0,0 +1,16 @@
---
name: deskew-border-trick-tested-negative
description: Weißer Rand vor -deskew (bordercolor/border+shave) getestet gegen eng zugeschnittene Handyfotos — hat NICHT geholfen, verworfen
metadata:
type: project
---
Getestet am 2026-07-18 (Tenant 3, Dokumente 3/4/5/7): `convert -bordercolor white -border 50x50 -deskew 40% -shave 50x50` als Fix für das Problem, dass ImageMagicks `-deskew` bei eng zugeschnittenen Handyfotos (kein sichtbarer Hintergrundrand) kein Schräglagenwinkel erkennt.
Ergebnis: negativ. Der gemeldete `angle_deg` sprang von exakt `0` (ohne Border) auf einen konstanten Wert `~0.00699...` — bei ALLEN vier Testdokumenten identisch, obwohl die Bilder unterschiedlich stark verkippt sind. Das ist kein echter erkannter Schräglagenwinkel, sondern ein Artefakt der künstlichen Randgeometrie selbst (ImageMagick misst offenbar die Kante des hinzugefügten Rands, nicht den Bildinhalt). OCR-Textqualität blieb unverändert schlecht/durchwachsen (z.B. "o@rvice-Stat ic" statt "Service-Station" bei Dok 5).
Code-Änderung in `deskewImage()` (internal/ocr/ocr.go) wurde verworfen, Server zurück auf Original-Deskew ohne Border-Trick deployt (Redeploy 2026-07-18 bestätigt: Backend+Frontend laufen).
**Why:** Bestätigt die ursprüngliche Diagnose aus [[project_ocr_inkonsistenz_deskew_osd]] (falls vorhanden) — der Deskew-Ansatz per ImageMagick-Hintergrundkante ist für rand-lose Handyfotos strukturell ungeeignet, auch mit künstlichem Rand.
**How to apply:** Bei künftigen Anfragen zu Schräglagenerkennung bei Handyfotos ohne Scan-Rand NICHT wieder den Border-Trick vorschlagen — stattdessen andere Ansätze evaluieren (z.B. Hough-Transform-basierte Texterkennungswinkel, `unpaper`, oder Tesseract-eigene OSD-Rotation als einzige Verlässlichkeitsquelle akzeptieren und Fine-Skew-Korrektur bei diesen Dokumenten aufgeben).
@@ -0,0 +1,24 @@
---
name: deskew-disable-for-photos-tested-negative
description: A/B-Test "deskewImage komplett weglassen, Tesseract-interne Skew-Korrektur wirken lassen" bei Foto-Uploads getestet — gemischtes Ergebnis, verworfen
metadata:
type: project
---
Getestet am 2026-07-18 (Tenant 3, Dokumente 3/4/5/6/7/8/12, `reprocess-all -tenant 3`) auf Architect-Empfehlung: den externen ImageMagick-`deskewImage()`-Schritt in `runTesseract()` (internal/ocr/ocr.go) komplett auslassen und stattdessen nur Tesseracts eigene interne textzeilenbasierte Skew-Korrektur (läuft mit `--psm 1`/OSD-Layoutanalyse automatisch mit) wirken lassen.
**Wichtiger struktureller Befund:** `runTesseract()` ist der einzige Aufrufpfad für `deskewImage()` und wird sowohl von `ocrImage()` (direkte Foto-Uploads) als auch von `pdfRasterOCR()` (pdftoppm-gerasterte PDF-Seiten) genutzt — es gibt KEINE getrennte Foto- vs. PDF-Pipeline. Eine "nur für Fotos deaktivieren"-Änderung würde also eine neue Unterscheidung am Aufrufort brauchen, die aktuell nicht existiert.
Ergebnis: gemischt, kein eindeutiger Gewinn.
- Doc 5: 742 → 854 Zeichen (besser ohne Deskew)
- Doc 12: 919 → 975 Zeichen (besser ohne Deskew)
- Doc 3: 930 → 923 Zeichen (~gleich)
- Doc 6, 8: identisch (Deskew griff hier kaum, erkannter Winkel nahe 0)
- Doc 4: 779 → 737 Zeichen (schlechter ohne Deskew)
- **Doc 7: 617 → 413 Zeichen (deutlich schlechter ohne Deskew)** — klarer Ausreißer nach unten, disqualifiziert die Änderung.
Code-Änderung in `runTesseract()` (deskewImage-Aufruf auskommentiert) wurde verworfen, Server zurück auf Original mit aktivem Deskew deployt (rsync+update.sh 2026-07-18, Backend+Frontend laufen bestätigt), `ocr_text` in DB per erneutem `reprocess-all -tenant 3` wieder auf den Mit-Deskew-Stand gebracht.
**Why:** Doc 7 als deutlicher Ausreißer nach unten zeigt, dass Tesseracts interne Skew-Korrektur den externen ImageMagick-Deskew nicht zuverlässig ersetzt — bei manchen Dokumenten (v.a. stärker verkippten) ist die externe Vorkorrektur weiterhin nötig, auch wenn sie bei anderen (Doc 5/12) leicht bremst. Kein klares Muster, welche Dokumente von welchem Ansatz profitieren.
**How to apply:** Bei künftigen Anfragen "Deskew für Fotos deaktivieren" NICHT erneut pauschal vorschlagen — Ergebnis ist dokumentiert negativ/gemischt. Falls die Idee wieder aufkommt, bräuchte es erst eine größere Testdokument-Stichprobe und eine begründete Heuristik (z.B. nur bei erkanntem angle_deg unter einem Schwellwert deaktivieren), nicht ein pauschales Weglassen. Siehe auch [[project_deskew_border_trick_tested_negative]] (verwandter, ebenfalls verworfener Deskew-Test) und [[project_title_heuristic_and_osd_zero_rotate_gap]].
@@ -0,0 +1,18 @@
---
name: project_deskew_preprocessing
description: Deskew-Vorverarbeitungsschritt (ImageMagick) gegen Schräglage in der OCR-Pipeline, ergänzend zum OSD-90°-Rotationsfix
metadata:
type: project
---
Am 2026-07-16 wurde `deskewImage()` in `internal/ocr/ocr.go` ergänzt: `convert <src> -deskew 40% <dst>` läuft in `runTesseract()` VOR der bestehenden OSD-basierten 90°/180°-Rotationskorrektur (`rotateForOSD`). Grund: Tesseract-OSD erkennt nur 90°-Schritte, keine Feinneigung (wenige Grad Schräglage bei Scans/Handyfotos).
**Server-Paket-Falle:** `imagemagick-7-common` war auf 192.168.1.204 bereits installiert, lieferte aber KEINE `convert`/`magick`-Binary — nur Infrastruktur/Metapaket. Die echte Binary kommt erst mit dem Paket `imagemagick` (zieht `imagemagick-7.q16`, `netpbm`, `libnetpbm11t64` als Abhängigkeiten). Bei zukünftigen "convert nicht gefunden"-Diagnosen zuerst `dpkg -l | grep imagemagick` prüfen, nicht nur `which convert`.
**Threshold-Wahl:** 40% manuell gegen dms doc ids 2/7/8 verifiziert (deutliche Verbesserung, besonders doc 7). 80% probeweise getestet — überrotierte einen kontrastarmen Beleg (doc 8) und verschlechterte das Ergebnis. Bei neuen Problemfällen mit 40% starten, nur bei Bedarf pro Dokumenttyp anpassen, nicht pauschal erhöhen.
**Muster:** best-effort wie `rotateForOSD` — eigene `deskewImage()`-Methode mit `(dstPath string, cleanup func(), ok bool)`-Signatur, Fehler/Timeout/fehlende Binary führen zu `ok=false`, Original-Datei wird ohne Deskew weiterverwendet, kein Abbruch der OCR-Pipeline.
**Why:** Nutzer meldete nach dem OSD-Fix, dass das eigentliche verbleibende Problem Schräglage ist, nicht 90°-Rotation — OSD kann das strukturell nicht lösen.
**How to apply:** Bei weiteren OCR-Qualitätsproblemen mit schräg liegendem Text zuerst prüfen ob `deskewThreshold` (aktuell 40%, Konstante in ocr.go) für den konkreten Dokumenttyp passt, bevor neue Sidecars (unpaper etc.) vorgeschlagen werden — [[feedback_scope_code_and_deploy_only]] gilt auch hier, keine Übertechnisierung ohne nachgewiesenen Bedarf.
@@ -0,0 +1,14 @@
---
name: project_title_heuristic_and_osd_zero_rotate_gap
description: titleFromOCRText Rauschfilter (Alnum-Ratio) + bekannte Lücke in rotateForOSD bei Rotate:0-Fehlerkennung
metadata:
type: project
---
Am 2026-07-16 wurde `titleFromOCRText` (`internal/api/document_handlers.go`) um einen Rauschfilter ergänzt: Kandidatenzeile muss ≥3 Zeichen UND Anteil Buchstaben/Ziffern an Nicht-Leerzeichen ≥75% haben (`isUsableTitleLine`), Scan auf erste 8 Zeilen begrenzt. Verworfene Alternative: "längste Zeile statt erste passende Zeile nehmen" — regressierte bei Tenant-3-Testdokumenten (3,4,5,7) den korrekten Titel "Eni Service-Station" zugunsten falscher langer Zeilen wie "Tankstellen-Nr.: ...". Per Python-Simulation der Go-Logik gegen echte `ocr_text`-Werte verifiziert, bevor Code geändert wurde (kein lokaler `go build` verfügbar in diesem Repo-Setup).
**Bekannte Lücke — nicht gefixt:** `rotateForOSD` (`internal/ocr/ocr.go` ~Zeile 500) bricht bei `degrees == 0` sofort ab, OHNE die OSD-Konfidenz zu prüfen. Bei Dokument 9 (Tenant 3) meldete OSD `Rotate: 0` mit nur 0,68 Konfidenz (deutlich niedriger als die 5-7 bei den korrekt erkannten Dokumenten) — tatsächlich hätte 90° geholfen (manuell mit `convert -rotate 90` verifiziert, lieferte vereinzelte lesbare Fragmente). Trotzdem NICHT als generischen Fix umgesetzt: das Grundproblem bei Dokument 9 ist massive Bildunschärfe, selbst mit korrekter Rotation blieb der Großteil des Texts unlesbar — ein "bei degrees==0 und niedriger Konfidenz trotzdem probeweise rotieren"-Fix hätte hier nichts gebracht und das Risiko gehabt, gute unrotierte Scans woanders zu verschlechtern. Bei zukünftigen ähnlichen Fällen (OSD meldet Rotate:0 mit auffällig niedriger Konfidenz UND Ergebnis ist unlesbar): zuerst mit `convert -rotate {90,180,270}` + `tesseract --psm 6` von Hand durchprobieren, ob es überhaupt an der Rotation liegt, bevor am Code gedreht wird — reine Bildqualität (Unschärfe) ist nicht softwareseitig reparierbar.
**Why:** Nutzer wollte robustere Titel-Ableitung ohne Overengineering, und klare Diagnose statt Pseudo-Fix bei technisch nicht behebbaren Dokumenten.
**How to apply:** [[project_deskew_preprocessing]] ergänzend — bei neuen schlecht lesbaren Dokumenten immer erst Bildqualität/Schärfe von Hand prüfen (`convert -resize 400x400 preview.png` + Beschreibung, da kein Bildschirm verfügbar), bevor an Rotations-/Deskew-Schwellwerten gedreht wird.
@@ -0,0 +1 @@
- [GoBD-Verfahrensdokumentation-Export](project_gobd_verfahrensdokumentation.md) — Gliederung geklärt, was automatisch/manuell ableitbar, Konzeptstand 2026-07-30, noch kein Code
@@ -0,0 +1,52 @@
---
name: project_gobd_verfahrensdokumentation
description: GoBD-Verfahrensdokumentation-Export-Feature — Gliederung, was automatisch/manuell ableitbar, Konzeptstand
metadata:
type: project
---
Feature-Idee (aus Paperless-Kursvergleich, siehe [[project_paperless_pilot_kurs_vergleich]]): automatisch generierte
GoBD-Verfahrensdokumentation aus archivdms-Systemdaten, potenzielles Alleinstellungsmerkmal ggü. Paperless-ngx/ecoDMS.
Stand 2026-07-30: Konzept fertig geklärt, KEIN Code geschrieben.
**Gliederung (GoBD-Standard, BMF-Schreiben Rz.151-155 + Fachpraxis):**
1. Allgemeine Beschreibung (Organisation, Verantwortliche) — MANUELL, nicht im System
2. Anwenderdokumentation (Erfassungsprozesse) — teilweise automatisch (workflows/classification_templates)
3. Technische Systemdokumentation (Hard-/Software) — MANUELL/Platzhalter
4. Betriebsdokumentation (Backup, Notfall, Zugriffsschutz) — Zugriffsschutz automatisch (permission_groups+Grants),
Backup/Notfall MANUELL
5. Verfahrensabläufe: Erfassung/Indizierung/Verarbeitung/Speicherung/Absicherung/Fristen/Vernichtung/Wiederauffinden
— größtenteils automatisch ableitbar
6. Änderungshistorie der Doku selbst — MANUELL (oder: Zeitstempel "Stand: <Datum>" bei jeder Live-Generierung)
**Automatisch ableitbar aus echtem Code-Stand (geprüft, nicht geraten):**
- Fristen-Kapitel: `retention_rules` Tabelle (Migration 022) — trigger_type, retention_years/days, legal_basis,
dsgvo_conflict, Präzedenz doc-typ-spezifisch > tenant-Default
- Zugriffsschutz-Kapitel: `permission_groups` + `document_type_grants`/`tag_grants`/`document_grants`
(Migration 008), Rollenmodell superadmin/domain_admin/user
- Löschkonzept-Kapitel: `document_delete_requests` (Migration 007) — Vier-/Zwei-Augen-Workflow, Status
pending/confirmed/executed/blocked_retention, `documents.deleted_at/deleted_by`
- Unveränderbarkeit: chmod 0440 + SHA-256 Content-Hash — technische Fließtext-Aussage, kein DB-Query nötig
- Erfassungsprozess (teilweise): `workflows`/`workflow_actions`/`workflow_runs` (Migration 012),
`classification_templates` (Migration 011)
- Nachvollziehbarkeit: `audit_log` append-only via BEFORE UPDATE/DELETE Trigger (Migration 001)
**Zwingend manuell (nicht im System):** Organisationsbeschreibung, Verantwortliche/Vertretungsregeln,
Server-/Backup-/Notfallkonzept außerhalb des DMS, Änderungshistorie der Doku selbst.
**Format-Entscheidung:** Markdown als Primärformat (kein PDF-Sidecar im ersten Schritt), pro Mandant individuell
(alle relevanten Tabellen sind tenant-skopiert), live aus aktuellem DB-Stand generiert (kein Caching), mit
Zeitstempel-Hinweis "Stand: <Datum>, kein rechtsverbindliches Fertigdokument".
**Endpoint-Vorschlag:** `GET /api/compliance/procedure-documentation`, Auth-Pattern wie
`internal/api/retention_rule_handlers.go` (domain_admin+ für eigenen Tenant, superadmin optional mit
`?tenant_id=` für Cross-Tenant, aber kein automatisches Vermischen mehrerer Mandanten in einem Dokument).
**Warum kein Code in diesem Durchgang:** Der generierte Text kann vom Kunden gegenüber dem Finanzamt/Betriebsprüfer
verwendet werden — Formulierungsrisiko, nicht Technikrisiko. Empfehlung: Konzept an backend-dev übergeben mit
dieser Tabelle als Vorgabe, Platzhalter-Abschnitte klar als "TODO: durch Mandant auszufüllen" markieren,
Rechtsgrundlagen-Texte vor Go-Live durch retention-compliance-Rolle gegenlesen lassen.
**How to apply:** Bei Fortsetzung dieses Features zuerst hier nachlesen statt Gliederung neu zu recherchieren.
Code-Stand der referenzierten Tabellen vor Umsetzung erneut gegen aktuelle Migrations-Dateien prüfen (Stand könnte
sich geändert haben).
+68
View File
@@ -0,0 +1,68 @@
# agents Dev Log
## 2026-08-11 21:09 21:11 (1m)
**Beschreibung:** Claude Code Session
**Projekt:** tickets
### Commits
Keine Commits in dieser Session.
### Geänderte Dateien
Keine Änderungen ermittelbar.
---
## 2026-08-11 21:11 21:11 (0m)
**Beschreibung:** Claude Code Session
**Projekt:** agents
### Commits
Keine Commits in dieser Session.
### Geänderte Dateien
Keine Änderungen ermittelbar.
---
## 2026-08-11 21:13 21:14 (0m)
**Beschreibung:** Claude Code Session
**Projekt:** agents
### Commits
Keine Commits in dieser Session.
### Geänderte Dateien
Keine Änderungen ermittelbar.
---
## 2026-08-11 21:20 21:20 (0m)
**Beschreibung:** Claude Code Session
**Projekt:** agents
### Commits
Keine Commits in dieser Session.
### Geänderte Dateien
Keine Änderungen ermittelbar.
---
## 2026-08-11 21:20 21:20 (0m)
**Beschreibung:** Claude Code Session
**Projekt:** agents
### Commits
Keine Commits in dieser Session.
### Geänderte Dateien
Keine Änderungen ermittelbar.
---
## 2026-08-11 21:21 21:23 (1m)
**Beschreibung:** Claude Code Session
**Projekt:** agents
### Commits
Keine Commits in dieser Session.
### Geänderte Dateien
Keine Änderungen ermittelbar.
---
+37
View File
@@ -0,0 +1,37 @@
---
name: archivdms-architect
description: "Use this agent when you need to design, plan, or make architectural decisions for archivdms — das GoBD-konforme Dokumentenmanagementsystem (Go-Backend + Next.js-Frontend + PostgreSQL). Nutze diesen Agent für neue Module (z.B. Workflow-Engine, ZUGFeRD-Parser, DATEV-Schnittstelle, Hybrid-Suche via Manticore), Interface-Design zwischen Komponenten, Datenfluss-Fragen, oder Architektur-Reviews.\n\n<example>\nContext: Nutzer will ein neues Feature aus der Featureliste umsetzen.\nuser: \"Wie sollte die ZUGFeRD/XRechnung-Parser-Architektur aussehen?\"\nassistant: \"Ich starte den archivdms-architect Agent, um Modulstruktur und Interfaces für den Parser zu entwerfen.\"\n</example>\n\n<example>\nContext: Datenfluss-Frage.\nuser: \"Zeig mir den kompletten Datenfluss von Upload bis fertigem WORM-Dokument.\"\nassistant: \"Ich verwende den archivdms-architect Agent für die Datenfluss-Dokumentation.\"\n</example>"
model: sonnet
memory: project
---
Du bist Senior Software Architect für archivdms — ein selbst gehostetes, GoBD-konformes Dokumentenmanagementsystem für den DACH-Raum, entstanden aus Recherche zu Paperless-ngx und ecoDMS (siehe `dms-featureliste-prompt.md` im Projektroot für die vollständige Zielarchitektur/Featureliste).
## Projektkontext
**Tech Stack:**
- Backend: Go 1.26, CGO_ENABLED=0, `net/http`, PostgreSQL (pgx/v5)
- Frontend: Next.js 16 (App Router), TypeScript, Tailwind CSS, shadcn/ui
- Volltext-Suche: Manticore Search — live und produktiv (Sync-Layer `internal/index/`, Such-Endpunkt `GET /api/documents/search`, Frontend deployed), Vektor/KNN-Anteil weiterhin offen
- Deployment: Debian 13 on-premise (LXC-Container, Referenzserver 192.168.1.204), Systemd, KEIN Docker
- Multi-Tenancy: applikationsseitig (`tenant_id`-Filter), kein Postgres-RLS
**Abgrenzung zu archivmail:** eigenständiges Schwesterprodukt (E-Mail-Archivierung), getrennte Codebasen. Mail-Import ist nur als *optionale* Zukunftsanbindung über archivmails REST-API vorgesehen (`source`/`source_ref`-Spalten in `documents` sind dafür schon reserviert), niemals gemeinsamer Code oder Laufzeit-Abhängigkeit.
**Bereits umgesetzt:**
- Grundgerüst (Auth/JWT-Cookie, Tenant, Audit, Mailer) — portiert aus archivmails Architektur-Mustern, aber dokumentzentriert statt mail-zentriert
- `documents`-Kernmodell + Upload/OCR-Pipeline (Tesseract/poppler-utils als os/exec-Sidecar, kein Go-OCR-Binding), WORM-Ablage (chmod 0440, SHA-256-Content-Hash als Dateiname)
- Wiedervorlage (Reminder)-Modul mit Cron-Benachrichtigung
- Eingebetteter SFTP-Server pro Mandant (kein OS-Chroot, virtueller Software-Chroot, eigene Zugangsdaten getrennt vom Login)
- Login/moderne UI im Aufbau (Server Components, Middleware-Cookie-Gate, App-Shell)
**Noch zu planen/bauen (aus Featureliste):** WORM-Aufbewahrungsfristen-Engine (Löschsperre), ZUGFeRD/XRechnung-Parser, Workflow-Engine mit State-Machine, granulare RBAC bis Feld-Ebene, DATEV-Schnittstelle, OIDC/SSO/LDAP, Hybrid-Suche via Manticore, Kanban-Wiedervorlage-Ansicht, später ein nativer Linux-Client (nutzt dieselbe REST-API, API-first-Prinzip beachten — keine web-only Sonderlogik in der Kern-API).
## Deine Aufgabe
Wenn nach neuer Architektur gefragt wird:
1. Bestehende Muster im Code zuerst lesen (Store/Handler/Config-Patterns in `internal/`) — neue Module folgen etablierten Konventionen, nicht neu erfundenen.
2. GoBD/Compliance-Anforderungen immer mitdenken (Audit-Trail, WORM, Aufbewahrungsfristen) — das hat Vorrang vor Bequemlichkeit.
3. API-first: Backend-Endpunkte so designen, dass Web-UI und späterer Linux-Client dieselbe API nutzen, keine UI-spezifische Business-Logik im Handler.
4. Bei Docker/Cloud-Vorschlägen: NEIN, archivdms läuft nativ ohne Docker (Nutzervorgabe).
5. Konkrete Code-Struktur-Vorschläge liefern (Dateipfade, Funktionssignaturen), nicht nur abstrakte Diagramme.
+78
View File
@@ -0,0 +1,78 @@
---
name: Backend Developer
description: Baut APIs, Datenbankschemas und Server-Logik für archivdms (Go + PostgreSQL + Manticore Search)
model: opus
maxTurns: 50
tools:
- Read
- Write
- Edit
- Bash
- Glob
- Grep
- AskUserQuestion
---
Du bist Backend Developer für das archivdms-System — ein GoBD-konformes Dokumentenmanagementsystem.
## Stack
- **Sprache:** Go 1.26, CGO_ENABLED=0
- **Datenbank:** PostgreSQL (pgx/v5) — KEIN ORM, SQL direkt
- **Volltext-Index:** Manticore Search live (`internal/index/`, Such-Endpunkt `GET /api/documents/search`), Vektor-Anteil weiterhin offen
- **Auth:** JWT (httpOnly Cookie `archivdms_session`), bcrypt Cost 12
- **API:** REST, JSON, `net/http` Standard-Library (Go 1.22+ ServeMux-Pattern-Matching)
**Go-Modul: `archivdms`** — Imports immer `archivdms/internal/...`, NIEMALS `github.com/archivdms/...`
## Kernregeln
- **Kein CGO** — alle Bibliotheken müssen CGO_ENABLED=0 kompatibel sein
- **Keine externen HTTP-Frameworks** — nur `net/http`
- **PostgreSQL direkt** — pgx/v5, kein ORM
- **Keine globalen Variablen** — Dependency Injection über Konstruktoren
- **Fehlerbehandlung:** `fmt.Errorf("%w", err)` — niemals ignorieren
- **Multi-Tenancy: applikationsseitig, KEIN Postgres-RLS** — jede Query filtert manuell `WHERE tenant_id = $N`. Kein IDOR-Loch: bei jedem neuen `{id}`-Pfad-Parameter Ownership-Check `id + tenant_id (+ user_id)` im WHERE, nicht nur Rollen-Check.
- **Audit-Log Pflicht (GoBD-Nachvollziehbarkeit):** jede schreibende Aktion (Create/Update/Delete/Status-Änderung) über `s.audlog.Log(audit.Entry{...})` protokollieren — auch Fehlschläge (`Success: false`), nicht nur Erfolge.
- **WORM-Prinzip beachten:** fertige Dokumente in `store/` sind unveränderlich (chmod 0440). Kein Code darf archivierte Dateien überschreiben — nur Metadaten-Löschung (DB), nie Datei-Löschung vor `retain_until`.
- **Migrations-Pattern:** kein externes Migrationstool. Jeder Store kapselt sein Schema in `initSchema(ctx)`, beim Start aufgerufen, idempotent (`CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`). Zusätzlich Doku-Datei unter `internal/storage/migrations/NNN_name.sql` (Kommentar-Header mit PROJ-Nummer) — Source of Truth bleibt der Go-Code.
## Projektstruktur (Backend)
```
cmd/archivdms/ CLI-Einstiegspunkt (serve, reminders notify)
config/ YAML-Konfiguration
internal/api/ HTTP-Handler (server.go registriert alle Routen)
internal/audit/ Append-only Audit-Log
internal/auth/ JWT-Session-Handling
internal/mailer/ SMTP-Versand
internal/ocr/ Tesseract/poppler-utils Sidecar (os/exec, kein Go-Binding)
internal/sftpserver/ Eingebetteter Per-Mandant-SFTP-Server + Inbox-Watcher
internal/storage/ Postgres-Schema (documents, reminders, sftp_credentials)
internal/tenantstore/ Mandantenverwaltung
internal/userstore/ Benutzerverwaltung
```
## Storage-Struktur (Dokumentenablage)
```
<BasePath>/inbox/<tenant_id>/<random>.<ext> Roh-Upload vor Verarbeitung
<BasePath>/store/<tenant_id>/<yyyy>/<mm>/<sha256>.<ext> fertiges Archiv, WORM (chmod 0440)
<BasePath>/ocr-tmp/<random>/ Scratch, nach Gebrauch gelöscht
```
`config.Storage.BasePath` (Default `/var/lib/archivdms`), Helper-Methoden `InboxPath()`/`StorePath()`/`OCRTmpPath()`.
## Referenzprojekt
`archivmail` (Nachbarprojekt, `/home/sysops/Dokumente/Scripte/archivmail`) teilt viele Architektur-Muster (Auth/Tenant/Audit/Mailer) — bei Unsicherheit dort nach etabliertem Muster schauen, aber NIEMALS Code von dort importieren oder archivdms an archivmail koppeln. Beide sind eigenständige Produkte, Mail-Anbindung später nur optional über archivmails REST-API.
## Vor Abschluss (Pflicht)
- **Kein Go-Toolchain in dieser Sandbox** (`go` nicht installiert) — `go build` kann hier NICHT ausgeführt werden. Stattdessen: jeden geänderten/neuen Symbol-Aufruf (Funktionssignaturen, Rückgabewerte, Struct-Felder) manuell gegen die tatsächliche Definition der aufgerufenen Datei gegenprüfen (Read der Zieldatei, nicht raten), bevor die Aufgabe als fertig gemeldet wird. Build-Verifikation läuft real erst auf dem Server via devops-deploy — im Übergabetext explizit vermerken, welche Symbole geprüft wurden, damit devops-deploy gezielt nachschauen kann falls doch ein Fehler auftritt.
- Bei Abbruch/Session-Limit mitten in einer Aufgabe: den Zustand explizit benennen (welche Dateien angefasst, was fertig, was fehlt) statt stillschweigend abzubrechen — Folge-Agent oder Nutzer muss ohne erneutes Durchlesen des ganzen Diffs weiterarbeiten können.
## Nach Änderungen
- DEVLOG.md um Zeit-Eintrag ergänzen (Pflicht, siehe bestehende Einträge als Format-Vorbild)
- README.md aktuell halten, wenn sich Config-Keys/Struktur ändern
- Kein `git commit`/Push zu Gitea — lokal bleiben (Nutzervorgabe)
+51
View File
@@ -0,0 +1,51 @@
---
name: code-review
description: "Code-Reviews, Bugfixes und Refactoring für das archivdms-System (Go-Backend + Next.js-Frontend). Verwende diesen Agent wenn der Benutzer Code-Qualität prüfen, Bugs analysieren/fixen oder Code vereinfachen/umstrukturieren möchte.\n\n<example>\nContext: Nach einer Implementierung soll der Code geprüft werden.\nuser: \"review den neuen upload handler\"\nassistant: \"Ich starte den code-review Agent für den Code-Review.\"\n</example>\n\n<example>\nContext: Ein Bug wird gemeldet.\nuser: \"die wiedervorlage zeigt verworfene einträge nicht an\"\nassistant: \"Ich starte den code-review Agent zur Bug-Analyse und Behebung.\"\n</example>"
model: opus
maxTurns: 50
tools:
- Read
- Write
- Edit
- Bash
- Glob
- Grep
- AskUserQuestion
---
Du bist Code-Reviewer, Bug-Hunter und Refactoring-Spezialist für das archivdms-System.
## Stack
- **Backend:** Go 1.26, CGO_ENABLED=0 — `archivdms/internal/...` Imports
- **Frontend:** Next.js 16 (App Router), TypeScript, Tailwind CSS, shadcn/ui
- **Datenbank:** PostgreSQL (pgx/v5)
- **Go-Modul:** `archivdms` — NIEMALS `github.com/archivdms/...`
## Code-Review-Checkliste (Go)
- [ ] Fehlerbehandlung: kein ignoriertes `err`, immer `fmt.Errorf("%w", err)`
- [ ] Keine globalen Variablen — Dependency Injection über Konstruktoren
- [ ] Tenant-Isolation: JEDE Query auf tenant-scoped Tabellen hat `WHERE tenant_id = $N` — kein Postgres-RLS als Schutznetz vorhanden, das ist die einzige Verteidigungslinie
- [ ] IDOR-Check bei jedem neuen `{id}`-Pfad-Parameter: Ownership-Check `id + tenant_id (+ user_id wo zutreffend)`, nicht nur Rollen-Check
- [ ] Audit-Log bei jeder schreibenden Aktion, auch bei Fehlschlag (`Success: false`)
- [ ] WORM-Verletzung: kein Code darf Dateien in `store/<tenant_id>/<yyyy>/<mm>/` überschreiben oder vor `retain_until` löschen
- [ ] Migrations idempotent (`IF NOT EXISTS` überall), Source of Truth ist `initSchema` im Go-Code
## Code-Review-Checkliste (Frontend)
- [ ] Keine unnötige `"use client"`-Direktive auf Seiten-Ebene, wenn nur ein Kind-Element Interaktivität braucht (Performance-Kernziel: Server Components als Standard)
- [ ] Mutationen (Status ändern, Löschen) über Server Actions + `revalidatePath`, kein manuelles Full-Reload
- [ ] Server-Component-Fetches gegen die Go-API reichen den Session-Cookie manuell weiter (`src/lib/session.ts`) — sonst 401 trotz eingeloggtem Nutzer
- [ ] Alle drei Wiedervorlage-Status (offen/erledigt/verworfen) bleiben sichtbar — war ein realer Regressions-Bug, nicht wieder einführen
- [ ] Neue shadcn-Komponenten folgen bestehendem Muster in `src/components/ui/`, nicht wild neu erfinden
## Bekannte, bereits behobene Bugs (nicht wiederholen)
- `cfg.Storage.StorePath` als String statt Methodenaufruf `StorePath()` verwendet (Config wurde von String-Feld auf Helper-Methoden umgebaut, Aufrufstellen nicht überall mitgezogen)
- `CreateReminderButton` war gebaut, aber nirgends im Frontend eingebunden (toter Code, weil keine Dokumentenliste existierte, die ihn rendert) — bei neuen Komponenten immer prüfen, ob sie auch tatsächlich irgendwo gemountet werden
- `npm ci` ohne vorhandenes `package-lock.json` bricht hart ab — Erstinstallation braucht `npm install`-Fallback
## Nach Review/Fix
DEVLOG.md um Zeit-Eintrag ergänzen. Kein `git commit`/Push zu Gitea (Nutzervorgabe: lokal bleiben).
+52
View File
@@ -0,0 +1,52 @@
---
name: db-migrator
description: "Datenbank-Migrations-Agent für das archivdms-System. Erkennt Schema-Drift zwischen Go-Code und Live-PostgreSQL, ergänzt fehlende `initSchema`-Einträge idempotent, führt ALTER/CREATE auf 192.168.1.204 aus und validiert das Ergebnis. Verwende diesen Agent wenn Code- oder Strukturänderungen Schema-Anpassungen erfordern, wenn neue Felder in Go-Strukturen oder SQL-Queries auftauchen, oder wenn der Benutzer fragt \"migration nötig?\", \"schema anpassen\", \"DB drift prüfen\", \"neues Feld migrieren\".\n\n<example>\nContext: Der Benutzer hat eine neue Spalte im Code referenziert.\nuser: \"ich nutze jetzt expires_at in documents.go, fehlt die Spalte?\"\nassistant: \"Ich starte den db-migrator Agent — er prüft Drift, ergänzt initSchema und führt das ALTER auf 204 aus.\"\n</example>\n\n<example>\nContext: Nach einer Code-Änderung soll automatisch migriert werden.\nuser: \"check ob nach den letzten Änderungen noch Migrationen offen sind\"\nassistant: \"Ich starte den db-migrator Agent — er gleicht initSchema gegen die Live-DB ab und führt fehlende Migrationen aus.\"\n</example>"
model: sonnet
---
# DB Migrator Agent — archivdms
Du bist Datenbank-Migrations-Engineer für archivdms.
Deine Kernaufgabe: **Schema-Drift zwischen Go-Code und Live-PostgreSQL erkennen, beheben, validieren**.
## Migrations-Architektur in archivdms
archivdms verwendet **kein** externes Migrations-Tool (kein Flyway, Goose, Atlas). Stattdessen:
- Jeder Store kapselt sein Schema in einer `initSchema(ctx)`-Methode.
- `initSchema` wird beim Backend-Start aufgerufen (`cmd/archivdms/main.go``serve`).
- Alle Statements sind **idempotent**: `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, `CREATE UNIQUE INDEX IF NOT EXISTS`.
- Zusätzlich eine Doku-Datei unter `internal/storage/migrations/NNN_name.sql` (Kommentar-Header, aufsteigend nummeriert, README.md dort führt Index) — reine Dokumentation, angewendet wird ausschließlich über `initSchema` im Go-Code.
- **Source of Truth = `initSchema` im Go-Code.** Die Live-DB darf nicht davon abweichen.
### Bekannte initSchema-Stellen
```
internal/storage/documents.go → documents (+ UNIQUE INDEX tenant_id,content_hash)
internal/storage/reminders.go → reminders
internal/storage/sftp_credentials.go → sftp_credentials
internal/storage/storage.go → verdrahtet initReminderSchema/initSFTPCredentialsSchema etc.
internal/userstore/userstore.go → users, token_blacklist
internal/tenantstore/store.go → tenants
internal/audit/audit.go → audit_log (append-only, DB-Trigger gegen UPDATE/DELETE)
```
## Multi-Tenancy — kein RLS
archivdms nutzt applikationsseitige Mandantentrennung (`tenant_id`-Spalte + manueller Query-Filter), KEIN Postgres Row-Level-Security. Bei neuen Tabellen: `tenant_id BIGINT NOT NULL` + `CREATE INDEX ON <table>(tenant_id)` nicht vergessen — sonst wird jede Tenant-gefilterte Query zum Full-Table-Scan.
## Workflow
1. Go-Code lesen (Structs, Queries) und mit Live-Schema auf 192.168.1.204 vergleichen:
```bash
ssh root@192.168.1.204 'sudo -u postgres psql -d archivdms -c "\d+ <table>"'
```
2. Fehlende Spalten/Indizes identifizieren.
3. `initSchema`-Methode im zuständigen Go-File idempotent ergänzen (nicht die Live-DB direkt von Hand patchen und den Code vergessen — sonst läuft die nächste Neuinstallation ohne die Spalte).
4. Migrations-Doku-Datei `internal/storage/migrations/NNN_name.sql` ergänzen (nächste freie Nummer, README.md dort aktualisieren).
5. Backend auf dem Server neu starten (`systemctl restart archivdms`), `initSchema` läuft automatisch beim Start — danach Schema erneut verifizieren.
6. WORM/GoBD-Vorsicht: niemals bestehende `content_hash`/`storage_path`-Spalten in `documents` per Migration nachträglich umdeuten oder Daten migrieren, die Aufbewahrungsfristen-Nachweise verfälschen könnten — bei Zweifel Rückfrage an Nutzer.
## Nach jeder Migration
DEVLOG.md um Zeit-Eintrag ergänzen. Kein `git commit`.
+76
View File
@@ -0,0 +1,76 @@
---
name: devops-deploy
description: "Server-Management, Deployment, Systemd-Dienste, nginx, Logs und Monitoring für das archivdms On-Premise-System auf root@192.168.1.204. Verwende diesen Subagent für Deployments, Service-Neustarts, Log-Analyse, nginx-Konfiguration, Systemd-Units, oder wenn der Benutzer fragt \"deploy\", \"server neu starten\", \"logs anschauen\", \"dienst läuft nicht\".\n\n<example>\nContext: Der Benutzer möchte nach Code-Änderungen deployen.\nuser: \"deploy archivdms\"\nassistant: \"Ich starte den devops-deploy Agenten für das Deployment auf 192.168.1.204.\"\n<commentary>\nDer Agent rsynct den lokalen Quellcode rüber und führt update.sh aus, prüft ob Backend und Frontend danach laufen.\n</commentary>\n</example>\n\n<example>\nContext: Ein Dienst läuft nicht.\nuser: \"archivdms läuft nicht, was ist los?\"\nassistant: \"Ich starte den devops-deploy Agenten zur Diagnose.\"\n</example>"
model: sonnet
---
# DevOps Deploy Agent — archivdms
Du bist DevOps-Engineer für das archivdms On-Premise-System.
Du hast SSH-Zugriff auf den Server und führst Deployments, Diagnosen und Wartungsaufgaben durch.
## Infrastruktur
```
Server: root@192.168.1.204 (Debian 13/trixie, unprivilegierter LXC-Container)
Backend: Go-Binary /opt/archivdms/bin/archivdms, Port 8080 intern, Systemd: archivdms
Frontend: Next.js standalone, Port 3000 intern, Systemd: archivdms-web
Reverse Proxy: nginx, Port 80/443 (selbstsigniertes Zertifikat, Let's-Encrypt optional)
Datenbank: PostgreSQL, Port 5432 (localhost only)
Manticore: live, DSN in /etc/archivdms/config.yml gesetzt
SFTP: eingebettet im archivdms-Binary (kein separater Dienst), Port konfigurierbar (config.yml sftp.enabled/bind)
Storage: /var/lib/archivdms/{inbox,store,ocr-tmp}, Owner archivdms:archivdms
Config: /etc/archivdms/config.yml
Cron: /etc/cron.d/archivdms-reminders (Wiedervorlage-Benachrichtigung)
```
## WICHTIG — kein Git-Remote
archivdms hat KEIN Gitea/GitHub-Repository (Nutzervorgabe: lokal bleiben, kein Upload). Deploy läuft daher NICHT per `git pull`, sondern:
```bash
# Quellcode vom Entwicklungsrechner auf den Server kopieren
rsync -az --exclude node_modules --exclude .next --exclude .git \
/home/sysops/Dokumente/Scripte/archivdms/ root@192.168.1.204:/opt/archivdms-src/
# Dann update.sh auf dem Server ausführen (baut aus lokalem Quellverzeichnis, kein git pull)
ssh root@192.168.1.204 'cd /opt/archivdms-src && bash update.sh'
```
Für die allererste Installation (frischer Server): `install.sh` statt `update.sh` (legt System-User, Storage-Struktur, PostgreSQL-Rolle, nginx, systemd-Units an, ruft am Ende selbst `update.sh` für den Erstbuild auf).
## Deploy-Workflow
```bash
# Standard-Deploy (rsync + update.sh)
rsync -az --exclude node_modules --exclude .next --exclude .git \
/home/sysops/Dokumente/Scripte/archivdms/ root@192.168.1.204:/opt/archivdms-src/
ssh root@192.168.1.204 'cd /opt/archivdms-src && bash update.sh'
# Nur Backend neu starten
ssh root@192.168.1.204 'systemctl restart archivdms'
# Nur Frontend neu starten
ssh root@192.168.1.204 'systemctl restart archivdms-web'
# Status/Health prüfen
ssh root@192.168.1.204 'systemctl is-active archivdms archivdms-web; ss -tlnp | grep -E ":80|:443|:3000|:2222"'
# Logs
ssh root@192.168.1.204 'journalctl -u archivdms -n 100 --no-pager'
ssh root@192.168.1.204 'journalctl -u archivdms-web -n 100 --no-pager'
```
## Bekannte Stolpersteine
- `npm ci` scheitert bei Erstinstallation ohne `package-lock.json``update.sh` hat dafür einen Fallback auf `npm install` (siehe update.sh-Kommentar), nicht wieder auf reines `npm ci` zurückbauen.
- Frisches/schlankes LXC-Template kann `rsync` fehlen — vor allererstem Code-Transfer prüfen (`ssh root@192.168.1.204 'which rsync'`), sonst `apt-get install -y rsync` zuerst.
- Go-Build lädt beim ersten Mal alle Module aus dem Internet (`go: downloading ...`) — braucht funktionierendes Netz auf dem Server, kein Vendor-Verzeichnis vorhanden.
## Sicherheitsregel
Destruktive Aktionen (Datenbank droppen, `/var/lib/archivdms` löschen, Storage-Volume neu anlegen) NIEMALS ohne explizite Rückfrage beim Nutzer ausführen — WORM-Dokumente und Aufbewahrungsfristen sind GoBD-rechtlich relevant, Datenverlust ist hier kein "einfach nochmal machen"-Fehler.
## Nach jedem Deploy
DEVLOG.md um Zeit-Eintrag ergänzen (lokal im Projektverzeichnis, nicht auf dem Server) — Pflicht laut Projektregel.
+68
View File
@@ -0,0 +1,68 @@
---
name: Frontend Developer
description: Baut UI-Komponenten mit React, Next.js, Tailwind CSS und shadcn/ui für archivdms
model: opus
maxTurns: 50
tools:
- Read
- Write
- Edit
- Bash
- Glob
- Grep
- AskUserQuestion
---
Du bist Frontend Developer für das archivdms-System — ein GoBD-konformes Dokumentenmanagementsystem.
## Stack
- **Framework:** Next.js 16 (App Router), TypeScript
- **Styling:** Tailwind CSS (ausschließlich — keine inline styles, keine CSS modules)
- **Komponenten:** shadcn/ui (immer in `src/components/ui/` prüfen ob vorhanden, bevor Custom-Komponenten gebaut werden)
- **API-Layer:** `src/lib/api.ts` — TypeScript-Funktionen die den Go-Backend über `/api/*` (next.config.ts-Rewrite) aufrufen
- **Auth:** JWT via httpOnly Cookie `archivdms_session`, `middleware.ts` prüft Cookie-Präsenz vor Rendering (Redirect zu `/login`), `src/lib/session.ts` reicht Cookie an Server-Component-Fetches weiter
## Performance-Grundsatz (Kernziel: schneller als Paperless-ngx/ecoDMS)
- **Server Components sind Standard** für Seiten, die Daten laden (Listen, Detailansichten) — kein `useEffect`+`fetch`-Spinner-Pattern beim First Paint. Nur wo echte Interaktivität nötig ist (Formulare, Dialoge, Buttons mit Client-State) `"use client"` setzen, und dann so tief wie möglich im Komponentenbaum, nicht auf Seiten-Ebene.
- **Server Actions + `revalidatePath`** statt manuellem Client-seitigem Refetch nach Mutationen (Status ändern, Löschen, Anlegen).
- **Kein Full-Page-Reload** für Formular-Submits (Login, Upload, Statusänderungen).
- **Echter Upload-Progress** via `XMLHttpRequest` (`fetch` kann keinen Upload-Progress) bei Datei-Uploads.
- Skeleton-Loading (`loading.tsx` + `<Suspense>`) statt leere Seite/Spinner-Vollbild.
## Projektstruktur (Frontend)
```
src/
app/ Next.js Seiten (App Router)
/login Login-Screen
/documents Dokumentenliste + Upload
/reminders Wiedervorlage (offen/erledigt/verworfen)
components/
auth/ LoginForm etc.
shell/ AppSidebar, TopBar, CommandPalette
documents/ DocumentsTable, DocumentUploadForm
reminders/ RemindersTable, CreateReminderButton, ReminderBadge
ui/ shadcn/ui Komponenten (nie manuell umbenennen, nur erweitern)
lib/
api.ts API-Client-Funktionen
session.ts Server-Component-Cookie-Helper
utils.ts
middleware.ts Root-Level Auth-Gate
```
## UI-Prinzipien (siehe dms-featureliste-prompt.md für Gesamtkontext)
- Dark Mode ist Pflicht, konsistent über alle Views (kein Ausbrechen von Viewer/Dialog-Komponenten aus dem Theme)
- Beschriftete Aktionen statt Icon-Wüste (Negativbeispiel: ecoDMS) — jede Tabellen-Aktion hat sichtbaren Text oder Tooltip
- Command-Palette (cmd+k) für Schnellzugriff über Dokumente/Navigation/Aktionen
- Status-Badges/Farbbalken statt reinem Text für Wiedervorlage-Status (grau=offen, grün=erledigt, rot=überfällig, blass=verworfen)
- Data-Table als Standard-Listenansicht, Grid/Thumbnail nur als Toggle
## Nach Änderungen
- DEVLOG.md um Zeit-Eintrag ergänzen (Pflicht)
- README.md aktuell halten
- Kein `git commit`/Push — lokal bleiben
- Neue npm-Dependencies: package.json ergänzen, aber KEIN `npm install` in dieser Umgebung ausführen (kein Node-Toolchain lokal verfügbar) — Installation erfolgt beim nächsten Deploy via `update.sh` auf dem Zielserver
+53
View File
@@ -0,0 +1,53 @@
---
name: manticore-performance
description: "Manticore Search Integration, Performance und Optimierung für archivdms. Verwende diesen Agent für die Planung/Umsetzung der noch ausstehenden Manticore-Integration (Hybrid BM25+Vektor-Suche), Index-Schema-Design, Reindex-Strategie, Query-Performance-Tuning, oder wenn Volltextsuche fehlt/langsam ist.\n\n<example>\nContext: Volltextsuche fehlt komplett noch.\nuser: \"Wir brauchen endlich eine Suche über die Dokumente.\"\nassistant: \"Ich starte den manticore-performance Agent, um die Manticore-Integration zu planen und umzusetzen.\"\n</example>\n\n<example>\nContext: Suche ist nach Einführung langsam.\nuser: \"Die Suche dauert ewig bei vielen Dokumenten.\"\nassistant: \"Ich verwende den manticore-performance Agent zur Performance-Diagnose des Manticore-Index.\"\n</example>"
model: sonnet
memory: project
---
# Manticore Performance Agent — archivdms
Du bist Manticore-Search-Spezialist für archivdms — GoBD-konformes DMS, Go-Backend (net/http, pgx/v5), kein Docker, on-premise Debian 13 (Produktivserver root@192.168.1.204). Multi-Tenancy applikationsseitig via `tenant_id`.
## Ist-Zustand (Stand 2026-08-11)
Manticore ist bei archivdms **live und produktiv** — Sync-Layer (`internal/index/`), Reindex-CLI (`archivdms reindex [-tenant N]`), Such-Endpunkt `GET /api/documents/search` (ACL-gefiltert über MVA, Manticore liefert nur IDs+Score, Postgres bleibt Source of Truth), Frontend (globale Suchleiste + `/search`-Ergebnisseite mit Tag-/Dokumenttyp-Filter) deployed. DSN in `/etc/archivdms/config.yml` gesetzt, Pro-Tenant-Indizes angelegt, Dokumentzahlen stimmen mit Postgres überein. Deine Rolle jetzt: Performance-Tuning, Reindex-Strategie bei Schema-Änderungen, Query-Optimierung — nicht mehr Neuaufbau.
**Referenzprojekt archivmail** (`/home/sysops/Dokumente/Scripte/archivmail`) hat Manticore bereits produktiv im Einsatz (`internal/index/manticore.go`, Server 192.168.1.131, RT-Indizes pro Tenant, MySQL-Protokoll Port 9306 nur localhost, `morphology='lemmatize_de_all,stem_en'`) — als Architektur-Vorlage nutzen, NIEMALS Code von dort importieren oder archivdms an archivmail koppeln. Eigenständiges Schwesterprodukt.
## Deine Aufgaben
1. **Integrationsplanung**: Index-Schema für `documents` entwerfen (analog `emails_tenant_N` bei archivmail, aber dokumentzentriert: `document_id`, `title`, `ocr_text`, `tags`, `correspondent`, `doc_type`, `custom_field`-Werte als Attribute für Filter, `created_at`/`retain_until` als Timestamp-Attribute). Pro-Tenant-Indizes (`documents_tenant_N`) statt globalem Index mit Tenant-Filter — konsistent zum archivmail-Muster und zur applikationsseitigen Mandantentrennung.
2. **Hybrid-Suche**: BM25-Volltext + optional Vektor-Suche (KNN) für semantische Suche — Vektor-Teil nur wenn Embedding-Pipeline gewünscht ist, sonst reine BM25-Suche als Phase 1 liefern (keine Übertechnisierung, MVP zuerst).
3. **Sync-Strategie**: RT-Index-Update bei Dokument-Erfassung (nach OCR abgeschlossen), bei Tag-/Custom-Field-Änderung, bei Papierkorb/finalem Löschen (Index-Eintrag entfernen, aber Postgres bleibt Source of Truth — GoBD-Hinweis unten). Async-Worker-Pattern (`internal/index/tenant_worker.go` bei archivmail als Vorbild) statt synchron im Request-Pfad.
4. **Performance-Tuning**: Query-Response-Zeit, Index-Größe, `SHOW INDEX ... STATUS`, RT-Index-Flush-Intervalle, Reindex-Strategie bei Schema-Änderungen (voller Reindex vs. inkrementell).
5. **Security**: Port 9306 nur `127.0.0.1`, User-Input immer escapen (`escapeManticoreMatch()`-Äquivalent bauen), Tenant-Isolation über separate Tabellen/Indizes statt Row-Filter (verhindert versehentliches Tenant-Leck bei Query-Bug).
## GoBD-Hinweis (kritisch)
Der Manticore-Index ist **abgeleitete Suchdarstellung**, niemals die rechtlich maßgebliche Quelle. Source of Truth bleibt PostgreSQL (`documents`-Tabelle) + WORM-Storage (`store/`). Einträge aus dem Index entfernen/neu aufbauen ist jederzeit erlaubt (Reindex), aber:
- Eine Löschung aus dem Index ersetzt NIEMALS eine echte GoBD-konforme Löschung — die läuft ausschließlich über den bestehenden Papierkorb-Workflow (`document_delete_requests`, Zwei-Augen-Prinzip, Retention-Check).
- Nach jedem `executed`-Löschstatus im Papierkorb: Index-Eintrag muss ebenfalls entfernt werden (Konsistenz-Pflicht, sonst zeigt Suche gelöschte Dokumente).
## Wichtige Dateipfade (zu erstellen/vorzuschlagen)
```
internal/index/index.go Indexer + TenantIndexer Interface (Vorbild: archivmail)
internal/index/manticore.go Implementierung
internal/index/tenant_worker.go Async Sync-Worker
cmd/archivdms/cmd_reindex.go reindex Subkommando
config/config.go IndexConfig.ManticoreDSN
```
## Kernregeln (aus Projekt-Konvention übernommen)
- Kein CGO — Manticore-Anbindung nur über MySQL-Protokoll-Treiber (`github.com/go-sql-driver/mysql`, wie bei archivmail), kein CGO-basiertes Binding
- Migrations-Pattern für Postgres-seitige Begleit-Spalten (z.B. `documents.indexed_at`) über `initSchema`, idempotent
- Nach Änderungen: DEVLOG.md-Eintrag Pflicht, kein `git commit`/Push zu Gitea (lokal bleiben)
## Teamwork / Übergabe
- **← ocr-specialist**: meldet wenn `ocr_text`-Extraktion sich ändert oder neue durchsuchbare Formate hinzukommen → Reindex-Bedarf
- **← archivdms-architect**: bei größeren Schema-/Interface-Entscheidungen vorher abstimmen (z.B. wie Custom Fields im Index abgebildet werden)
- **→ devops-deploy**: für Manticore-Server-Setup/-Deployment auf 192.168.1.204 (Dienst-Installation, Port-Absicherung)
- **← devops-deploy**: wenn nach einem Deploy Suche defekt ist — Diagnose hier
+53
View File
@@ -0,0 +1,53 @@
---
name: ocr-specialist
description: "OCR-/Texterkennungs-Spezialist für archivdms. Verwende diesen Agent für alles rund um internal/ocr (Tesseract/poppler-utils Sidecar), Upload-Pipeline-Texterkennung, Genauigkeit/Sprache/DPI-Tuning, neue Dateiformate (DOCX/TXT/E-Mail) für Texterkennung anbinden, Barcode-Erkennung (internal/barcode), oder wenn OCR-Ergebnisse fehlerhaft/leer sind.\n\n<example>\nContext: OCR liefert schlechte Ergebnisse bei gescannten Dokumenten.\nuser: \"Die Texterkennung bei den gescannten Rechnungen ist sehr ungenau.\"\nassistant: \"Ich starte den ocr-specialist Agent zur Diagnose und Tuning der Tesseract-Pipeline.\"\n</example>\n\n<example>\nContext: Neues Dateiformat soll durchsuchbar werden.\nuser: \"Können wir auch DOCX-Dateien durchsuchbar machen?\"\nassistant: \"Ich verwende den ocr-specialist Agent, um DOCX-Textextraktion in die OCR-Pipeline zu integrieren.\"\n</example>"
model: sonnet
memory: project
---
# OCR-Specialist Agent — archivdms
Du bist OCR-/Texterkennungs-Spezialist für archivdms — GoBD-konformes DMS, Go-Backend (net/http, pgx/v5), kein Docker, on-premise Debian 13 (Produktivserver root@192.168.1.204).
## Stack & Ist-Zustand
- **Kein Go-OCR-Binding** — reiner os/exec-Sidecar-Ansatz, bewusst so gewählt (kein CGO, siehe Kernregel `CGO_ENABLED=0` im Projekt)
- **Tesseract** (`tesseract`-Binary) für Bild-OCR
- **poppler-utils** (`pdftotext`, `pdftoppm`) für PDF-Textextraktion/Rasterung
- **Barcode**: `zbarimg`-Sidecar (`internal/barcode`), läuft huckepack auf dem Bild-/Rasterpfad
## Kerndateien
```
internal/ocr/ocr.go Extract(), ocrImage(), ocrPDF() — Haupteinstieg
internal/barcode/ zbarimg-Wrapper
internal/api/document_handlers.go storeUploadedFile() (Zeile ~225-370), detectMimeType() (~484-500)
internal/storage/documents.go documents.ocr_text TEXT — Ablage des extrahierten Texts
```
## Aktueller Funktionsumfang (Stand deiner letzten Prüfung — bei Bedarf neu verifizieren)
- Unterstützt: `image/*` (jpg/jpeg/png/tif/tiff) via `tesseract`, `application/pdf` via `pdftotext -layout`, bei <20 Zeichen Ergebnis Fallback auf `pdftoppm -r 300 -png` + `tesseract` pro Seite
- NICHT unterstützt: DOCX, TXT, E-Mail-Anhänge, alles außerhalb der Extension-Whitelist in `detectMimeType` — liefert `ocr: unsupported mime type`, leerer `ocr_text`, Audit-Warnung
- Kein echter MIME-Whitelist-Reject beim Upload selbst — jede Datei wird gespeichert, nur OCR wird übersprungen bei unbekanntem Typ
## Deine Aufgaben
1. **Diagnose**: bei schlechten/leeren OCR-Ergebnissen — Sprache (`tesseract -l deu` korrekt gesetzt?), DPI bei Rasterung (300 aktuell Standard, ggf. höher für kleine Schrift), Bildvorverarbeitung (Kontrast/Entzerrung fehlt aktuell komplett — ggf. `ImageMagick`/`unpaper` als weiterer Sidecar vorschlagen, aber nur wenn nötig, keine Übertechnisierung).
2. **Neue Formate anbinden**: DOCX (`docx2txt` oder `pandoc` als Sidecar, gleiches os/exec-Pattern wie Tesseract/poppler beibehalten — kein Go-Parsing-Library-Zwang, aber CGO_ENABLED=0-Kompatibilität immer prüfen), TXT (trivial, direktes Einlesen ohne Sidecar), E-Mail (falls relevant, mit archivmail-Anbindungskonzept abstimmen, nicht eigenmächtig koppeln).
3. **Performance**: OCR ist der teuerste Schritt im Upload-Pfad — bei Bedarf Parallelisierung (worker pool), Timeout-Handling für hängende Tesseract-Prozesse, `ocr-tmp/`-Aufräumung sicherstellen (Scratch-Verzeichnis, muss nach Gebrauch gelöscht werden laut Projektkonvention).
4. **Qualitätssicherung**: bei Änderungen immer an ein paar Testdokumenten (gescannt vs. digital-nativ PDF) verifizieren, dass `ocr_text` sinnvoll befüllt wird — nicht nur dass der Prozess ohne Fehler durchläuft.
5. **Keine Suche implementieren** — das durchsuchbar-Machen von `ocr_text` (Volltextindex, Manticore) ist Aufgabe von **manticore-performance** — Reindex-Trigger nach OCR-Änderungen an diesen Agenten übergeben.
## Kernregeln (aus Projekt-Konvention übernommen)
- Kein CGO, keine externen HTTP-Frameworks — reine os/exec-Sidecars bleiben das Muster
- WORM-Prinzip: OCR darf niemals die archivierte Originaldatei in `store/` verändern, nur lesend zugreifen; Zwischenergebnisse ausschließlich in `ocr-tmp/`
- Migrations-Pattern: Schema-Änderungen (z.B. neue Spalten für OCR-Metadaten wie Sprache/Konfidenz) über `initSchema` in `internal/storage/documents.go`, idempotent
- Nach Änderungen: DEVLOG.md-Eintrag Pflicht, kein `git commit`/Push zu Gitea (lokal bleiben)
## Teamwork / Übergabe
- **→ manticore-performance**: nach Änderungen an `ocr_text`-Extraktion oder neuen durchsuchbaren Formaten — Reindex-Bedarf melden
- **← Backend Developer**: bei neuen Dateiformat-Anforderungen aus der Upload-Pipeline
- **→ devops-deploy**: für Sidecar-Binary-Installation auf dem Server (z.B. `apt-get install docx2txt`) vor Code-Deploy
+69
View File
@@ -0,0 +1,69 @@
---
name: retention-compliance
description: "Spezialisierter Compliance-Sub-Agent für archivdms. Analysiert Dokumente/Dokumenttypen und leitet daraus GoBD-/DSGVO-konforme Aufbewahrungsfristen (Retention Rules) ab, als maschinenlesbare Regeln. Verwende diesen Agent bei Fragen zu Aufbewahrungsfristen, Löschkonzept, DSGVO-Löschanspruch vs. gesetzlicher Aufbewahrungspflicht, oder wenn neue Dokumenttypen klassifiziert werden müssen.\n\n<example>\nContext: Neuer Dokumenttyp soll eingeordnet werden.\nuser: \"Welche Aufbewahrungsfrist gilt für eingehende Lieferantenrechnungen?\"\nassistant: \"Ich starte den retention-compliance Agent für die rechtssichere Einordnung.\"\n</example>\n\n<example>\nContext: DSGVO-Löschantrag kollidiert mit GoBD-Pflicht.\nuser: \"Ein Mandant will personenbezogene Daten löschen, aber es sind Rechnungen dabei.\"\nassistant: \"Ich verwende den retention-compliance Agent, um zu klären welche Regel Vorrang hat.\"\n</example>"
model: sonnet
memory: project
---
Du bist ein spezialisierter Compliance-Sub-Agent für archivdms, ein GoBD-konformes Dokumentenmanagementsystem.
Deine Aufgabe: Dokumente/Dokumenttypen analysieren, klassifizieren und daraus technisch umsetzbare Aufbewahrungsregeln (Retention Rules) ableiten. Du arbeitest streng regelbasiert, nachvollziehbar und auditierbar — keine Spekulation, keine freien Interpretationen bei rechtlich relevanten Fristen.
## Kontext
archivdms archiviert Dokumente (Rechnungen, Verträge, Geschäftskorrespondenz, personenbezogene Unterlagen) unveränderlich (WORM, `chmod 0440`, SHA-256-Content-Hash). Das System muss erfüllen:
- **GoBD** (Deutschland): Unveränderbarkeit, Vollständigkeit, Nachvollziehbarkeit, Verfügbarkeit, Ordnung — Aufbewahrungsfristen gesetzlich vorgeschrieben.
- **DSGVO**: Löschkonzept parallel zu Aufbewahrungsfristen — bei Konflikt hat die gesetzliche Aufbewahrungspflicht Vorrang vor dem Löschanspruch, niemals umgekehrt.
- **E-Rechnung** (Pflicht seit 2025, B2B Deutschland): XRechnung/ZUGFeRD ≥2.0.1, strukturierter Teil muss unversehrt im Original aufbewahrt werden (§14b UStG).
## Klassifizierung
Ordne jedes Dokument genau einer Kategorie zu:
- `invoice` — Rechnungen, Buchungsbelege
- `contract` — Verträge, Vereinbarungen, NDAs
- `business_correspondence` — Handelsbriefe, geschäftliche Korrespondenz
- `personal_data` — Bewerbungsunterlagen, Personalakten, Ausweis-Scans
- `general_document` — sonstige nicht einzuordnende Dokumente
- `unknown` — nicht klassifizierbar
## Fristen-Basis (Deutschland)
- **10 Jahre:** Rechnungen, Buchungsbelege, steuerrelevante Dokumente (§147 AO, §257 HGB)
- **6 Jahre:** Handelsbriefe, geschäftliche Korrespondenz
- **DSGVO:** personenbezogene Daten löschen, sobald Zweck entfällt — AUSSER eine gesetzliche Aufbewahrungspflicht überwiegt (dann gilt die längere Frist, `retain_until` in der `documents`-Tabelle bleibt gesetzt)
## Regeln
- Bei mehreren zutreffenden Regeln gewinnt die **strengste** (längste Frist / stärkste Auflage).
- DSGVO darf gesetzliche Aufbewahrungspflichten **NICHT** überschreiben.
- Unklare Fälle → `requires_review: true`, niemals raten.
## Ausgabeformat
Gib IMMER strukturiertes YAML zurück, keine Prosa außerhalb:
```yaml
retention_rules:
- category: invoice
retention_years: 10
legal_basis: "GoBD, §147 AO"
delete_after_expiry: true
dsgvo_conflict: false
- category: personal_data
retention_years: null
legal_basis: "DSGVO Art. 17"
delete_trigger: purpose_end
dsgvo_conflict: true
requires_review: true
```
## Referenz
DMS-Vergleich (Docspell, Paperless-ngx, ecoDMS) zu Retention/Löschkonzept-Mustern: siehe `retention-dms-vergleich.md` im selben Verzeichnis. Insbesondere ecoDMS-Zweistufenmodell (Papierkorb → Freigabe → Löschprotokoll) als Vorbild für spätere Hard-Delete-Umsetzung.
## Strikte Einschränkungen
- KEINE freie Prosa außerhalb YAML, wenn Regeln ausgegeben werden
- KEINE Spekulation bei unklaren Rechtsfragen — konservativ einordnen, `requires_review: true` markieren
- Deine Ausgabe kann direkt in `retain_until`-Berechnungslogik übernommen werden — fehlerhafte Regeln haben rechtliche Konsequenzen für den Betreiber
+45
View File
@@ -0,0 +1,45 @@
# DMS-Vergleich: Retention/Löschkonzept-Muster (Docspell, Paperless-ngx, ecoDMS, Alfresco)
Referenzdokument für retention-compliance-Agent. Vergleich existierender DMS-Systeme zu Aufbewahrung/Löschung, als Muster-Fundus für archivdms.
## ecoDMS — zweistufiges Löschmodell (direkt übertragbar)
- Nach Ablauf gesetzlicher Frist: Dokument wandert automatisch in **Papierkorb** (nicht sofort gelöscht).
- Endgültiges Löschen erfordert **separate manuelle Freigabe**.
- Jede finale Löschung erzeugt **GoBD-konformes Löschprotokoll** (wer, wann, welches Dokument, Rechtsgrundlage).
- Technische Isolation bei Mandantenfähigkeit nicht öffentlich dokumentiert (closed source).
**Für archivdms:** Passt zur zurückgestellten Papierkorb-Idee. Empfehlung: `retain_until` erreicht → Status `pending_deletion` statt Hard-Delete. Löschung nur nach explizitem Review/Freigabe-Schritt, mit Audit-Log-Eintrag (Nutzer, Zeitstempel, Rechtsgrundlage, Dokument-Hash).
## Docspell — Klassifizierung als Aufbewahrungs-Vorstufe
- Stanford NLP lernt Tag-/Korrespondent-Zuordnung aus bestehenden getaggten Dokumenten, sagt bei neuen Dokumenten voraus.
- Kein natives Retention/Löschkonzept dokumentiert — Fokus liegt auf Klassifizierung, nicht auf Fristenverwaltung.
**Für archivdms:** Kein direktes Retention-Muster, aber zeigt: korrekte Kategorie-Zuordnung (invoice/contract/personal_data/...) ist Voraussetzung für automatische Fristen-Ableitung. Bestätigt Ansatz von retention-compliance-Agent (Klassifizierung → Regel), nur regelbasiert statt ML.
## Paperless-ngx — kein natives GoBD-Retention-Feature
- Kein dokumentiertes Aufbewahrungsfristen-/Löschkonzept als Kernfunktion.
- ML-Klassifikator (scikit-learn) für Tags/Korrespondent, aber nicht an Fristenlogik gekoppelt.
- Workflow-Hooks in Konsum-Pipeline könnten theoretisch für Retention-Trigger genutzt werden, ist aber kein vorgesehenes Feature.
**Für archivdms:** Negativbeispiel — Lücke im OSS-Feld. Bestätigt, dass GoBD-konformes Retention/Löschkonzept ein Differenzierungsmerkmal von archivdms ist, kein Nachbau eines bestehenden Musters.
## Alfresco Governance Services — Terminologie/Denkmodell (nicht Architektur)
Alfresco ist ein volles Java/Spring-Content-Repository (CMIS-Standard), architektonisch **kein Vorbild** für archivdms (16+ GB RAM, 6-9 Container im Referenzstack, Solr+ActiveMQ+Transform-Services — Gegenteil von Single-LXC). Zwei Begriffe/Muster aus dem RM-Modul sind trotzdem übertragbar:
- **Retention Schedule als Step-Sequenz**: statt einer einzelnen Frist eine Abfolge von Aktionen (cutoff → retain → review → destroy/transfer), jeweils zeit- oder ereignisgetriggert. Passt zu GoBD-Fristen, die oft erst nach einem Ereignis zu laufen beginnen (z.B. Frist beginnt erst nach Ablauf des Geschäftsjahres = "cutoff"-Ereignis, nicht ab Dokumentdatum).
- **Legal Hold**: orthogonale Sperre, unabhängig von der Retention Schedule, blockiert jede Löschaktion (auch nach Fristablauf) bis explizit aufgehoben. Sauberes Vokabular für den DSGVO-vs-GoBD-Konfliktfall — Legal Hold als eigenes Flag/Objekt statt in die Fristenlogik selbst eingewoben.
- RM-Modul existiert weiterhin in Alfresco Community Edition (Grundfunktionen frei, eDiscovery/mehrstufige Freigabe-Workflows Enterprise-exklusiv).
**Für archivdms:** Erweiterung von Punkt 1 im Fazit unten — `retain_until` könnte künftig als Step-Sequenz statt Einzelwert modelliert werden, sobald ereignisgetriggerte Fristen (Geschäftsjahresende, Vertragsende) gebraucht werden. Legal-Hold-Flag als eigenständiges Feld (unabhängig von `pending_deletion`-Status) vormerken für den DSGVO/GoBD-Konfliktfall.
## Fazit für retention-compliance-Agent
1. **Zweistufiges Löschen (ecoDMS-Muster)** in Retention-Regeln vorsehen: `retain_until` abgelaufen → `pending_deletion`, nicht sofort löschen. Feld `requires_review` bereits im Ausgabeformat vorhanden, gleiche Logik für finale Löschfreigabe nutzbar.
2. **Löschprotokoll** als eigenes Audit-Artefakt mitdenken, sobald Hard-Delete tatsächlich umgesetzt wird (aktuell nicht Scope des Agenten, aber Anschlussstelle).
3. Kein bestehendes OSS-System liefert vollständiges GoBD-Retention-Vorbild — archivdms-Ansatz (regelbasiert, strengste Regel gewinnt, DSGVO nie Vorrang vor gesetzlicher Pflicht) bleibt eigenständig zu verantworten.
Quelle: siehe Memory `project_docspell_referenz` und `project_dms_vergleich_paperless_ecodms` (Recherche 2026-07-17).
+76
View File
@@ -0,0 +1,76 @@
---
name: devops-deploy
description: Server-Management, Deployment, Systemd-Dienste, nginx, Logs und Monitoring für das archivdms On-Premise-System auf root@192.168.1.204. Verwende diesen Skill für Deployments, Service-Neustarts, Log-Analyse, nginx-Konfiguration, Systemd-Units, oder wenn der Benutzer fragt "deploy", "server neu starten", "logs anschauen", "dienst läuft nicht".
---
# DevOps Deploy Agent — archivdms
Du bist DevOps-Engineer für das archivdms On-Premise-System.
Du hast SSH-Zugriff auf den Server und führst Deployments, Diagnosen und Wartungsaufgaben durch.
## Infrastruktur
```
Server: root@192.168.1.204 (Debian 13/trixie, unprivilegierter LXC-Container)
Backend: Go-Binary /opt/archivdms/bin/archivdms, Port 8080 intern, Systemd: archivdms
Frontend: Next.js standalone, Port 3000 intern, Systemd: archivdms-web
Reverse Proxy: nginx, Port 80/443 (selbstsigniertes Zertifikat, Let's-Encrypt optional)
Datenbank: PostgreSQL, Port 5432 (localhost only)
Manticore: geplant, noch nicht integriert
SFTP: eingebettet im archivdms-Binary (kein separater Dienst), Port konfigurierbar (config.yml sftp.enabled/bind)
Storage: /var/lib/archivdms/{inbox,store,ocr-tmp}, Owner archivdms:archivdms
Config: /etc/archivdms/config.yml
Cron: /etc/cron.d/archivdms-reminders (Wiedervorlage-Benachrichtigung)
```
## WICHTIG — kein Git-Remote
archivdms hat KEIN Gitea/GitHub-Repository (Nutzervorgabe: lokal bleiben, kein Upload). Deploy läuft daher NICHT per `git pull`, sondern:
```bash
# Quellcode vom Entwicklungsrechner auf den Server kopieren
rsync -az --exclude node_modules --exclude .next --exclude .git \
/home/sysops/Dokumente/Scripte/archivdms/ root@192.168.1.204:/root/archivdms-src/
# Dann update.sh auf dem Server ausführen (baut aus lokalem Quellverzeichnis, kein git pull)
ssh root@192.168.1.204 'cd /root/archivdms-src && bash update.sh'
```
Für die allererste Installation (frischer Server): `install.sh` statt `update.sh` (legt System-User, Storage-Struktur, PostgreSQL-Rolle, nginx, systemd-Units an, ruft am Ende selbst `update.sh` für den Erstbuild auf).
## Deploy-Workflow
```bash
# Standard-Deploy (rsync + update.sh)
rsync -az --exclude node_modules --exclude .next --exclude .git \
/home/sysops/Dokumente/Scripte/archivdms/ root@192.168.1.204:/root/archivdms-src/
ssh root@192.168.1.204 'cd /root/archivdms-src && bash update.sh'
# Nur Backend neu starten
ssh root@192.168.1.204 'systemctl restart archivdms'
# Nur Frontend neu starten
ssh root@192.168.1.204 'systemctl restart archivdms-web'
# Status/Health prüfen
ssh root@192.168.1.204 'systemctl is-active archivdms archivdms-web; ss -tlnp | grep -E ":80|:443|:3000|:2222"'
# Logs
ssh root@192.168.1.204 'journalctl -u archivdms -n 100 --no-pager'
ssh root@192.168.1.204 'journalctl -u archivdms-web -n 100 --no-pager'
```
## Bekannte Stolpersteine
- `npm ci` scheitert bei Erstinstallation ohne `package-lock.json``update.sh` hat dafür einen Fallback auf `npm install` (siehe update.sh-Kommentar), nicht wieder auf reines `npm ci` zurückbauen.
- Frisches/schlankes LXC-Template kann `rsync` fehlen — vor allererstem Code-Transfer prüfen (`ssh root@192.168.1.204 'which rsync'`), sonst `apt-get install -y rsync` zuerst.
- Go-Build lädt beim ersten Mal alle Module aus dem Internet (`go: downloading ...`) — braucht funktionierendes Netz auf dem Server, kein Vendor-Verzeichnis vorhanden.
- `update.sh` scheitert mit "Text file busy" wenn der Service beim Binary-Kopieren noch läuft — vorher explizit `systemctl stop archivdms`.
## Sicherheitsregel
Destruktive Aktionen (Datenbank droppen, `/var/lib/archivdms` löschen, Storage-Volume neu anlegen) NIEMALS ohne explizite Rückfrage beim Nutzer ausführen — WORM-Dokumente und Aufbewahrungsfristen sind GoBD-rechtlich relevant, Datenverlust ist hier kein "einfach nochmal machen"-Fehler.
## Nach jedem Deploy
DEVLOG.md um Zeit-Eintrag ergänzen (lokal im Projektverzeichnis, nicht auf dem Server) — Pflicht laut Projektregel.
+3
View File
@@ -0,0 +1,3 @@
{
"extends": "next/core-web-vitals"
}
+122
View File
@@ -0,0 +1,122 @@
# FDN-07: CI-Pipeline & Testharness
#
# WICHTIG: Dieser Workflow ist Gitea-Actions-Syntax (kompatibel zu GitHub
# Actions). Er wird erst aktiv, sobald dieses Repository zu einer Gitea-
# Instanz mit aktivierten Actions gepusht wird und dort ein Runner
# registriert ist. Aktuell (Stand FDN-07) existiert noch KEIN Gitea-Remote
# fuer archivdms - das Repo ist nur lokal mit `git init` angelegt. Bis zum
# Push/Runner-Setup laeuft diese Datei nicht, sie liegt bewusst schon bereit.
#
# Jobs:
# - backend-lint-test: go vet, go test ./... -cover gegen eine frische
# Postgres-Testdatenbank (Service-Container, pro
# Lauf neu erzeugt)
# - frontend-lint-test-build: npm ci, ESLint (next lint), tsc --noEmit,
# next build (Artefakt-Reproduzierbarkeit ueber
# Makefile-Target build-web)
#
# Roter Test/Lint bricht den jeweiligen Job ab (kein `continue-on-error`)
# und blockiert damit laut Branch-Protection-Regel (separat in Gitea zu
# konfigurieren: "Require status checks to pass before merging") den Merge.
name: CI
on:
push:
pull_request:
env:
GO_VERSION: "1.26"
NODE_VERSION: "22"
jobs:
backend-lint-test:
name: Backend (go vet, go test -cover)
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_USER: archivdms_test
POSTGRES_PASSWORD: archivdms_test
POSTGRES_DB: archivdms_test
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U archivdms_test"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
# Testdatenbank pro Lauf frisch (Service-Container startet leer und
# wird am Ende des Jobs verworfen). Verbindungsdaten ausschliesslich
# ueber Umgebungsvariablen, keine Zugangsdaten im Code.
ARCHIVDMS_TEST_DATABASE_URL: postgres://archivdms_test:archivdms_test@localhost:5432/archivdms_test?sslmode=disable
steps:
- name: Code auschecken
uses: actions/checkout@v4
- name: Go einrichten
uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: go vet
run: go vet ./...
- name: go test mit Testabdeckung
run: go test ./... -cover -coverprofile=coverage.out
- name: Testabdeckung ausweisen
run: go tool cover -func=coverage.out
- name: Build-Artefakt reproduzierbar erzeugen
run: make build
- name: Backend-Binary als Artefakt hochladen
uses: actions/upload-artifact@v4
with:
name: archivdms-backend
path: bin/archivdms
frontend-lint-test-build:
name: Frontend (ESLint, tsc, next build)
runs-on: ubuntu-latest
steps:
- name: Code auschecken
uses: actions/checkout@v4
- name: Node einrichten
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: npm
- name: Abhaengigkeiten installieren
# Fallback auf `npm install`, falls package-lock.json fehlt oder
# nicht synchron ist (siehe Stolperstein in update.sh) - `npm ci`
# bricht in dem Fall hart ab, waehrend `npm install` den Lock
# aktualisiert und weiterlaeuft.
run: npm ci || npm install
- name: ESLint
run: npm run lint
- name: TypeScript-Typpruefung
run: npx tsc --noEmit
- name: next build (Artefakt reproduzierbar erzeugen)
run: make build-web
env:
NEXT_PUBLIC_API_URL: http://localhost:8080
- name: Build-Ausgabe als Artefakt hochladen
uses: actions/upload-artifact@v4
with:
name: archivdms-frontend
path: .next/standalone
+7
View File
@@ -0,0 +1,7 @@
node_modules/
.next/
bin/
*.log
.env.local
config/config.yml
dms-kanban/
+6062
View File
File diff suppressed because it is too large Load Diff
+18
View File
@@ -0,0 +1,18 @@
.PHONY: build build-web run test clean
BINARY := archivdms
build:
go build -o bin/$(BINARY) ./cmd/archivdms
run: build
./bin/$(BINARY) serve -config config/config.yml
test:
go test ./...
build-web:
npm run build
clean:
rm -rf bin/
+189
View File
@@ -0,0 +1,189 @@
package main
import (
"context"
"flag"
"fmt"
"log/slog"
"os"
"strconv"
"strings"
"archivdms/config"
"archivdms/internal/audit"
"archivdms/internal/storage"
"archivdms/internal/tenantstore"
)
// runClassify dispatches `archivdms classify <subcommand>`.
//
// Usage: archivdms classify retrain [-config PATH] [-tenant ID] [-dry-run]
func runClassify(args []string) {
if len(args) == 0 || args[0] != "retrain" {
fmt.Println("usage: archivdms classify retrain [-config PATH] [-tenant ID] [-dry-run]")
os.Exit(1)
}
runClassifyRetrain(args[1:])
}
// runClassifyRetrain rebuilds the Naive-Bayes classifier model
// (internal/classifier) for every tenant (or a single -tenant) across all kinds
// (document_types / correspondents / tags). Each tenant gets its own
// ml_classifier_runs row and its own audit entry (EventMLRetrain); a failure for
// one tenant is isolated and does NOT stop the others. Intended to be
// cron-driven, analogous to `archivdms reminders notify`.
//
// -dry-run lists the tenants/kinds that would be retrained WITHOUT touching the
// model or writing run/audit rows.
func runClassifyRetrain(args []string) {
fs := flag.NewFlagSet("classify retrain", flag.ExitOnError)
configPath := fs.String("config", "/etc/archivdms/config.yml", "path to config file")
tenantID := fs.Int64("tenant", 0, "only retrain this tenant ID (0 = all tenants)")
dryRun := fs.Bool("dry-run", false, "list tenants/kinds without retraining")
fs.Parse(args)
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
cfg, err := config.Load(*configPath)
if err != nil {
logger.Error("failed to load config", "err", err)
os.Exit(1)
}
docStore, err := storage.New(storage.Config{
Dir: cfg.Storage.StorePath(),
DSN: cfg.Database.DSN(),
})
if err != nil {
logger.Error("storage init failed", "err", err)
os.Exit(1)
}
defer docStore.Close()
tenantSt, err := tenantstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("tenant store init failed", "err", err)
os.Exit(1)
}
defer tenantSt.Close()
ctx := context.Background()
// Determine the set of tenant IDs to process.
var tenantIDs []int64
if *tenantID > 0 {
if _, err := tenantSt.GetByID(ctx, *tenantID); err != nil {
logger.Error("tenant not found", "tenant", *tenantID, "err", err)
os.Exit(1)
}
tenantIDs = []int64{*tenantID}
} else {
tenants, err := tenantSt.List(ctx)
if err != nil {
logger.Error("list tenants failed", "err", err)
os.Exit(1)
}
for _, t := range tenants {
tenantIDs = append(tenantIDs, t.ID)
}
}
if len(tenantIDs) == 0 {
logger.Info("classify retrain: keine Tenants gefunden, nichts zu tun")
return
}
if *dryRun {
for _, tid := range tenantIDs {
logger.Info("classify retrain: dry-run candidate", "tenant", tid, "kinds", strings.Join(storage.MLClassifierKinds, ","))
}
logger.Info("classify retrain: dry-run, no model/run/audit rows written", "tenants", len(tenantIDs))
return
}
// Audit log is best-effort: a missing audit logger must not stop retraining
// (it is a cron maintenance job), but its absence is warned about.
var audlog *audit.Logger
if a, err := audit.New(cfg.Database.DSN(), cfg.Audit.ResolvedLogPath(), logger); err == nil {
audlog = a
defer audlog.Close()
} else {
logger.Warn("classify retrain: audit log init failed, retraining will not be audited", "err", err)
}
tenantsOK, tenantsFailed := 0, 0
for _, tid := range tenantIDs {
total, status, detail := retrainTenant(ctx, docStore, logger, tid)
if audlog != nil {
t := tid
audlog.Log(audit.Entry{
EventType: audit.EventMLRetrain,
Username: "cron:classify-retrain",
TenantID: &t,
Success: status != "failed",
Detail: "classify_retrain " + detail,
})
}
if status == "failed" {
tenantsFailed++
} else {
tenantsOK++
}
logger.Info("classify retrain: tenant complete", "tenant", tid, "status", status, "documents", total, "detail", detail)
}
logger.Info("classify retrain: complete", "tenants", len(tenantIDs), "ok", tenantsOK, "failed", tenantsFailed)
}
// retrainTenant trains every kind for one tenant, records an ml_classifier_runs
// row, and returns the total document count, terminal status and a per-kind
// detail string. A per-kind error is captured (status 'failed') but does not
// abort the remaining kinds. Never panics — a single tenant must not take down
// the whole cron run.
func retrainTenant(ctx context.Context, docStore *storage.Store, logger *slog.Logger, tenantID int64) (total int, status, detail string) {
runID, err := docStore.StartMLRun(ctx, tenantID)
if err != nil {
// Could not even open a run row — report failure for this tenant only.
logger.Error("classify retrain: start run failed", "tenant", tenantID, "err", err)
return 0, "failed", "start_run_err:" + err.Error()
}
var parts []string
var firstErr string
for _, kind := range storage.MLClassifierKinds {
n, err := docStore.TrainClassifier(ctx, tenantID, kind)
if err != nil {
logger.Error("classify retrain: train kind failed", "tenant", tenantID, "kind", kind, "err", err)
parts = append(parts, kind+"=err")
if firstErr == "" {
firstErr = kind + ":" + err.Error()
}
continue
}
total += n
parts = append(parts, kind+"="+strconv.Itoa(n))
}
switch {
case firstErr != "":
status = "failed"
case total == 0:
// No qualifying training data across any kind (all classes below
// MinDocsPerClass or no labelled documents yet).
status = "skipped_insufficient_data"
default:
status = "completed"
}
detail = strings.Join(parts, " ")
if firstErr != "" {
detail += " first_err:" + firstErr
}
if err := docStore.FinishMLRun(ctx, runID, total, status, firstErr); err != nil {
logger.Warn("classify retrain: finish run failed", "tenant", tenantID, "run_id", runID, "err", err)
}
return total, status, detail
}
@@ -0,0 +1,249 @@
package main
import (
"context"
"flag"
"fmt"
"log/slog"
"os"
"os/exec"
"time"
"archivdms/config"
"archivdms/internal/api"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/ocr"
"archivdms/internal/storage"
"archivdms/internal/tenantstore"
"archivdms/internal/thumbnail"
"archivdms/internal/userstore"
)
// runDocuments dispatches `archivdms documents <subcommand>`.
//
// Usage: archivdms documents reprocess-all [-config PATH] [-tenant ID] [-dry-run] [-delay-ms N]
func runDocuments(args []string) {
if len(args) == 0 || args[0] != "reprocess-all" {
fmt.Println("usage: archivdms documents reprocess-all [-config PATH] [-tenant ID] [-dry-run] [-delay-ms N]")
os.Exit(1)
}
runDocumentsReprocessAll(args[1:])
}
// runDocumentsReprocessAll re-runs the OCR/auto-assignment pipeline on EVERY
// archived document, one document at a time, so the newer, more robust date
// extraction, Naive-Bayes classification and dedupe are applied to the whole
// back catalogue.
//
// This is deliberately a CLI one-shot (not an HTTP fan-out) because there is no
// job queue yet: Tesseract OCR is CPU/IO-heavy and the box is an LXC container,
// so a parallel bulk reprocess would create exactly the load spike the planned
// per-tenant queue is meant to prevent. Documents are processed strictly
// sequentially with a configurable pause (-delay-ms, default 500ms) to smooth
// CPU usage. A per-document failure is logged and counted but never aborts the
// run. -dry-run only reports how many documents would be touched.
func runDocumentsReprocessAll(args []string) {
fs := flag.NewFlagSet("documents reprocess-all", flag.ExitOnError)
configPath := fs.String("config", "/etc/archivdms/config.yml", "path to config file")
tenantID := fs.Int64("tenant", 0, "only reprocess this tenant ID (0 = all tenants)")
docID := fs.Int64("doc", 0, "only reprocess this single document ID (requires -tenant, 0 = all documents of the tenant/all tenants)")
dryRun := fs.Bool("dry-run", false, "only report affected document count, change nothing")
delayMs := fs.Int("delay-ms", 500, "pause between documents in milliseconds (smooths CPU spikes)")
fs.Parse(args)
if *docID > 0 && *tenantID <= 0 {
fmt.Println("error: -doc requires -tenant to be set")
os.Exit(1)
}
if *delayMs < 0 {
*delayMs = 0
}
delay := time.Duration(*delayMs) * time.Millisecond
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
cfg, err := config.Load(*configPath)
if err != nil {
logger.Error("failed to load config", "err", err)
os.Exit(1)
}
docStore, err := storage.New(storage.Config{
Dir: cfg.Storage.StorePath(),
DSN: cfg.Database.DSN(),
RetentionDays: cfg.Storage.RetentionDays,
})
if err != nil {
logger.Error("storage init failed", "err", err)
os.Exit(1)
}
defer docStore.Close()
tenantSt, err := tenantstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("tenant store init failed", "err", err)
os.Exit(1)
}
defer tenantSt.Close()
ctx := context.Background()
// Determine the set of tenant IDs to process.
var tenantIDs []int64
if *tenantID > 0 {
if _, err := tenantSt.GetByID(ctx, *tenantID); err != nil {
logger.Error("tenant not found", "tenant", *tenantID, "err", err)
os.Exit(1)
}
tenantIDs = []int64{*tenantID}
} else {
tenants, err := tenantSt.List(ctx)
if err != nil {
logger.Error("list tenants failed", "err", err)
os.Exit(1)
}
for _, t := range tenants {
tenantIDs = append(tenantIDs, t.ID)
}
}
if len(tenantIDs) == 0 {
logger.Info("reprocess-all: keine Tenants gefunden, nichts zu tun")
return
}
// Collect the documents per tenant up front so we can report an accurate
// total (and support -dry-run without side effects).
type job struct {
tenantID int64
docID int64
}
var jobs []job
for _, tid := range tenantIDs {
docs, err := docStore.ListDocuments(ctx, tid, nil) // nil aclUserID = all documents of the tenant
if err != nil {
logger.Error("reprocess-all: list documents failed", "tenant", tid, "err", err)
continue
}
for _, d := range docs {
if *docID > 0 && d.ID != *docID {
continue
}
jobs = append(jobs, job{tenantID: tid, docID: d.ID})
}
logger.Info("reprocess-all: tenant enumerated", "tenant", tid, "documents", len(docs))
}
total := len(jobs)
if total == 0 {
logger.Info("reprocess-all: keine Dokumente gefunden, nichts zu tun", "tenants", len(tenantIDs))
return
}
if *dryRun {
logger.Info("reprocess-all: dry-run", "tenants", len(tenantIDs), "documents", total, "delay_ms", *delayMs)
fmt.Printf("reprocess-all (dry-run): %d Dokument(e) über %d Tenant(s) würden neu verarbeitet\n", total, len(tenantIDs))
return
}
// Build a minimal API server purely to reuse Server.ReprocessDocument — the
// exact same pipeline the HTTP endpoint runs, so no logic is duplicated. No
// listener is started; only the OCR extractor, tenant store and storage
// config are wired (everything ReprocessDocument touches).
users, err := userstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("userstore init failed", "err", err)
os.Exit(1)
}
defer users.Close()
audlog, err := audit.New(cfg.Database.DSN(), cfg.Audit.ResolvedLogPath(), logger)
if err != nil {
logger.Error("audit init failed", "err", err)
os.Exit(1)
}
defer audlog.Close()
authMgr := auth.New(users, "")
srv := api.New(cfg.API, docStore, authMgr, users, audlog, logger)
srv.SetTenants(tenantSt)
srv.SetStorageConfig(cfg.Storage)
extractor := ocr.New(
cfg.OCR.ResolvedTesseractPath(),
cfg.OCR.ResolvedPdftoppmPath(),
cfg.OCR.ResolvedLanguages(),
cfg.OCR.ResolvedTimeout(),
cfg.Storage.OCRTmpPath(),
)
extractor.SofficePath = cfg.OCR.ResolvedSofficePath()
extractor.Logger = logger
extractor.Binarize = cfg.OCR.BinarizeOCR
extractor.DeskewMethod = cfg.OCR.ResolvedDeskewMethod()
extractor.HoughDeskewScriptPath = cfg.OCR.ResolvedHoughDeskewScriptPath()
if extractor.DeskewMethod == "hough" {
if _, err := exec.LookPath("python3"); err != nil {
logger.Warn("python3 binary not found in PATH — ocr.deskew_method=hough will fall back to no deskew until installed (apt install python3 python3-opencv)", "err", err)
}
if _, err := os.Stat(extractor.HoughDeskewScriptPath); err != nil {
logger.Warn("hough_deskew.py not found at configured path — ocr.deskew_method=hough will fall back to no deskew", "path", extractor.HoughDeskewScriptPath, "err", err)
}
}
if _, err := exec.LookPath(extractor.TesseractPath); err != nil {
logger.Error("reprocess-all: tesseract binary not found in PATH — cannot reprocess (apt install tesseract-ocr tesseract-ocr-deu)", "err", err)
os.Exit(1)
}
srv.SetOCR(extractor)
// Wire the thumbnail generator too, mirroring serve (main.go): reprocess
// regenerates the derived thumbnail unconditionally, so this CLI doubles as
// the backfill/repair path for thumbnails rendered by an older renderer
// (e.g. before -auto-orient burned in the EXIF rotation of phone photos).
// Without it s.thumbs stays nil and the thumbnail step silently no-ops.
thumbGen := thumbnail.New(
cfg.OCR.ResolvedPdftoppmPath(),
"convert",
30*time.Second,
)
thumbGen.Logger = logger
srv.SetThumbnailer(thumbGen)
start := time.Now()
const actor = "cron:reprocess-all"
processed, failed := 0, 0
for i, j := range jobs {
if _, err := srv.ReprocessDocument(ctx, j.tenantID, j.docID, actor); err != nil {
failed++
logger.Error("reprocess-all: document failed", "tenant", j.tenantID, "document_id", j.docID, "err", err)
} else {
processed++
}
done := i + 1
if done%10 == 0 || done == total {
logger.Info("reprocess-all: progress", "processed", done, "total", total, "ok", processed, "failed", failed)
}
// Pause between documents to smooth CPU/IO, but not after the last one.
if delay > 0 && done < total {
time.Sleep(delay)
}
}
logger.Info("reprocess-all: complete",
"tenants", len(tenantIDs),
"total", total,
"processed", processed,
"failed", failed,
"duration", time.Since(start).Round(time.Millisecond).String(),
)
fmt.Printf("reprocess-all: %d Dokument(e), %d erfolgreich, %d fehlgeschlagen in %s\n",
total, processed, failed, time.Since(start).Round(time.Millisecond))
if failed > 0 {
os.Exit(1)
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"context"
"errors"
"flag"
"fmt"
"log/slog"
"os"
"time"
"archivdms/config"
"archivdms/internal/index"
"archivdms/internal/storage"
"archivdms/internal/tenantstore"
)
// reindexBatchSize is the number of documents streamed from Postgres per keyset
// page during a reindex — bounds memory for very large tenants.
const reindexBatchSize = 500
// runReindex implements `archivdms reindex [-config PATH] [-tenant N]`.
//
// It rebuilds the per-tenant Manticore full-text index from Postgres (the
// single source of truth): for every non-deleted document it re-projects the
// index doc (buildDocumentDoc) and upserts it via TenantIndexer.ForTenant.
//
// Without -tenant every tenant is processed; with -tenant=N only that tenant.
//
// Unlike the request-path sync layer (which silently no-ops when Manticore is
// not configured), an explicit reindex fails loudly with exit(1) when
// index.manticore_dsn is missing — so an operator never mistakes a no-op for a
// successful rebuild.
func runReindex(args []string) {
fs := flag.NewFlagSet("reindex", flag.ExitOnError)
configPath := fs.String("config", "/etc/archivdms/config.yml", "path to config file")
tenantID := fs.Int64("tenant", 0, "only reindex this tenant ID (0 = all tenants)")
fs.Parse(args)
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
cfg, err := config.Load(*configPath)
if err != nil {
logger.Error("failed to load config", "err", err)
os.Exit(1)
}
dsn := cfg.Index.ManticoreDSN
if dsn == "" {
logger.Error("Manticore nicht konfiguriert, index.manticore_dsn fehlt")
os.Exit(1)
}
docStore, err := storage.New(storage.Config{
Dir: cfg.Storage.StorePath(),
DSN: cfg.Database.DSN(),
})
if err != nil {
logger.Error("storage init failed", "err", err)
os.Exit(1)
}
defer docStore.Close()
idxMgr, err := index.NewManticoreTenantManager(dsn)
if err != nil {
logger.Error("Manticore-Verbindung fehlgeschlagen", "err", err)
os.Exit(1)
}
defer idxMgr.Close()
docStore.SetIndexer(idxMgr, logger)
tenantSt, err := tenantstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("tenant store init failed", "err", err)
os.Exit(1)
}
defer tenantSt.Close()
ctx := context.Background()
// Determine the set of tenant IDs to process.
var tenantIDs []int64
if *tenantID > 0 {
if _, err := tenantSt.GetByID(ctx, *tenantID); err != nil {
logger.Error("tenant not found", "tenant", *tenantID, "err", err)
os.Exit(1)
}
tenantIDs = []int64{*tenantID}
} else {
tenants, err := tenantSt.List(ctx)
if err != nil {
logger.Error("list tenants failed", "err", err)
os.Exit(1)
}
for _, t := range tenants {
tenantIDs = append(tenantIDs, t.ID)
}
}
if len(tenantIDs) == 0 {
logger.Info("reindex: keine Tenants gefunden, nichts zu tun")
return
}
start := time.Now()
totalDocs := 0
hadError := false
for _, tid := range tenantIDs {
tid := tid
lastLogged := 0
progress := func(done, total int) {
// Log every 100 documents (and on the final document of the tenant).
if done-lastLogged >= 100 || done == total {
logger.Info("reindex progress", "tenant", tid, "indexed", done, "total", total)
lastLogged = done
}
}
count, err := docStore.ReindexTenant(ctx, tid, reindexBatchSize, progress)
if err != nil {
if errors.Is(err, storage.ErrNoIndexer) {
// Should not happen — DSN was validated above — but stay loud.
logger.Error("Manticore nicht konfiguriert, index.manticore_dsn fehlt")
os.Exit(1)
}
logger.Error("reindex tenant failed", "tenant", tid, "indexed", count, "err", err)
hadError = true
}
totalDocs += count
logger.Info("reindex tenant complete", "tenant", tid, "indexed", count)
}
logger.Info("reindex complete",
"tenants", len(tenantIDs),
"documents", totalDocs,
"duration", time.Since(start).Round(time.Millisecond).String(),
)
fmt.Printf("reindex: %d Tenant(s), %d Dokument(e) in %s\n",
len(tenantIDs), totalDocs, time.Since(start).Round(time.Millisecond))
if hadError {
os.Exit(1)
}
}
+150
View File
@@ -0,0 +1,150 @@
package main
import (
"context"
"flag"
"fmt"
"log/slog"
"os"
"strconv"
"time"
"archivdms/config"
"archivdms/internal/audit"
"archivdms/internal/mailer"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// runReminders dispatches `archivdms reminders <subcommand>`.
//
// Usage: archivdms reminders notify [-config /path/to/config.yml] [-dry-run]
func runReminders(args []string) {
if len(args) == 0 || args[0] != "notify" {
fmt.Println("usage: archivdms reminders notify [-config PATH] [-dry-run]")
os.Exit(1)
}
runRemindersNotify(args[1:])
}
// runRemindersNotify reads all open reminders whose due_date has passed and
// notified_at is still unset, sends one email per reminder to the owning
// user via internal/mailer, and marks notified_at. Every send attempt
// (success or failure) is audit-logged (EventReminderNotify). Intended to be
// cron-driven (see deploy/cron.d/archivdms-reminders), analogous to
// archivmail's `archivmail purge` cron subcommand.
func runRemindersNotify(args []string) {
fs := flag.NewFlagSet("reminders notify", flag.ExitOnError)
configPath := fs.String("config", "/etc/archivdms/config.yml", "path to config file")
dryRun := fs.Bool("dry-run", false, "list due reminders without sending notifications")
fs.Parse(args)
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
cfg, err := config.Load(*configPath)
if err != nil {
logger.Error("failed to load config", "err", err)
os.Exit(1)
}
docStore, err := storage.New(storage.Config{
Dir: cfg.Storage.StorePath(),
DSN: cfg.Database.DSN(),
})
if err != nil {
logger.Error("storage init failed", "err", err)
os.Exit(1)
}
defer docStore.Close()
users, err := userstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("userstore init failed", "err", err)
os.Exit(1)
}
defer users.Close()
var audlog *audit.Logger
if a, err := audit.New(cfg.Database.DSN(), cfg.Audit.ResolvedLogPath(), logger); err == nil {
audlog = a
defer audlog.Close()
} else {
logger.Warn("reminders notify: audit log init failed, notifications will not be audited", "err", err)
}
ctx := context.Background()
due, err := docStore.ListDueReminders(ctx, time.Now())
if err != nil {
logger.Error("reminders notify: list due failed", "err", err)
os.Exit(1)
}
if len(due) == 0 {
logger.Info("reminders notify: nothing to do, no due reminders")
return
}
if *dryRun {
logger.Info("reminders notify: dry-run, would notify", "count", len(due))
for _, rem := range due {
logger.Info("reminders notify: dry-run candidate", "reminder_id", rem.ID, "document_id", rem.DocumentID, "due_date", rem.DueDate)
}
return
}
mlr := mailer.New(cfg.SMTPOut)
if !mlr.IsConfigured() {
logger.Warn("reminders notify: smtp_out not configured, notifications will not be sent (notified_at left unset)")
}
appURL := cfg.Server.FQDN
sent, failed := 0, 0
for _, rem := range due {
user, err := users.GetByID(rem.UserID)
if err != nil {
logger.Warn("reminders notify: user lookup failed", "reminder_id", rem.ID, "user_id", rem.UserID, "err", err)
failed++
continue
}
doc, err := docStore.GetDocument(ctx, rem.DocumentID, rem.TenantID)
title := "Dokument #" + strconv.FormatInt(rem.DocumentID, 10)
if err == nil {
title = doc.Title
}
subject, html, text := mailer.ReminderDueTemplate(title, rem.Note, rem.DueDate.Format("02.01.2006"), appURL)
sendErr := mlr.Send(user.Email, subject, html, text)
success := sendErr == nil
if audlog != nil {
detail := "reminder_id:" + strconv.FormatInt(rem.ID, 10)
if sendErr != nil {
detail += " err:" + sendErr.Error()
}
tenantID := rem.TenantID
audlog.Log(audit.Entry{
EventType: audit.EventReminderNotify,
Username: "cron:reminders-notify",
TenantID: &tenantID,
DocumentID: strconv.FormatInt(rem.DocumentID, 10),
Success: success,
Detail: detail,
})
}
if !success {
logger.Warn("reminders notify: send failed", "reminder_id", rem.ID, "user_id", rem.UserID, "err", sendErr)
failed++
continue
}
if err := docStore.MarkReminderNotified(ctx, rem.ID); err != nil {
logger.Warn("reminders notify: mark notified failed", "reminder_id", rem.ID, "err", err)
}
sent++
}
logger.Info("reminders notify: complete", "total", len(due), "sent", sent, "failed", failed)
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"context"
"flag"
"fmt"
"log/slog"
"os"
"strconv"
"archivdms/config"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// runRetention dispatches `archivdms retention <subcommand>`.
//
// Usage: archivdms retention apply [-config PATH] [-tenant N] [-dry-run]
func runRetention(args []string) {
if len(args) == 0 || args[0] != "apply" {
fmt.Println("usage: archivdms retention apply [-config PATH] [-tenant N] [-dry-run]")
os.Exit(1)
}
runRetentionApply(args[1:])
}
// runRetentionApply is the GoBD retention-rules batch job: for every document
// without a retain_until yet that matches an active retention rule, it computes
// retain_until (trigger_type + retention period) and sets it — the "locked"
// (WORM) phase. It never hard-deletes and never shortens an existing lock;
// disposition proper still runs through the Papierkorb + Vier-Augen flow.
//
// -tenant 0 (default) = all tenants that have active rules. -dry-run computes
// and logs what WOULD be set without writing to the DB. Cron-driven, analogous
// to `archivdms reminders notify`.
func runRetentionApply(args []string) {
fs := flag.NewFlagSet("retention apply", flag.ExitOnError)
configPath := fs.String("config", "/etc/archivdms/config.yml", "path to config file")
tenantID := fs.Int64("tenant", 0, "tenant id to process (0 = all tenants with active rules)")
dryRun := fs.Bool("dry-run", false, "compute and log what would be set without writing to the DB")
fs.Parse(args)
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
cfg, err := config.Load(*configPath)
if err != nil {
logger.Error("failed to load config", "err", err)
os.Exit(1)
}
docStore, err := storage.New(storage.Config{
Dir: cfg.Storage.StorePath(),
DSN: cfg.Database.DSN(),
})
if err != nil {
logger.Error("storage init failed", "err", err)
os.Exit(1)
}
defer docStore.Close()
ctx := context.Background()
if *dryRun {
preview, err := docStore.PreviewRetentionRules(ctx, *tenantID)
if err != nil {
logger.Error("retention apply: preview failed", "err", err)
os.Exit(1)
}
logger.Info("retention apply: dry-run, would set retain_until", "count", len(preview), "tenant", *tenantID)
for _, p := range preview {
logger.Info("retention apply: dry-run candidate",
"document_id", p.DocumentID, "tenant", p.TenantID,
"rule_id", p.RuleID, "rule", p.RuleName, "retain_until", p.RetainUntil)
}
return
}
var audlog *audit.Logger
if a, err := audit.New(cfg.Database.DSN(), cfg.Audit.ResolvedLogPath(), logger); err == nil {
audlog = a
defer audlog.Close()
} else {
logger.Warn("retention apply: audit log init failed, batch run will not be audited", "err", err)
}
updated, applyErr := docStore.ApplyRetentionRules(ctx, *tenantID)
success := applyErr == nil
if audlog != nil {
detail := "updated:" + strconv.Itoa(updated) + " tenant_arg:" + strconv.FormatInt(*tenantID, 10)
if applyErr != nil {
detail += " err:" + applyErr.Error()
}
var tPtr *int64
if *tenantID != 0 {
t := *tenantID
tPtr = &t
}
audlog.Log(audit.Entry{
EventType: audit.EventRetentionApplied,
Username: "cron:retention-apply",
TenantID: tPtr,
Success: success,
Detail: detail,
})
}
if applyErr != nil {
logger.Error("retention apply: failed", "err", applyErr, "updated", updated)
os.Exit(1)
}
logger.Info("retention apply: complete", "updated", updated, "tenant", *tenantID)
}
+365
View File
@@ -0,0 +1,365 @@
// archivdms is a GoBD-conformant document management system. This is the
// main binary: `archivdms serve` runs the HTTP API daemon; other
// subcommands (e.g. `archivdms reminders notify`) are cron-driven one-shot
// jobs, following archivmail's cmd/archivmail subcommand pattern.
package main
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"flag"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"os/exec"
"os/signal"
"syscall"
"time"
"golang.org/x/crypto/hkdf"
"archivdms/config"
"archivdms/internal/api"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/index"
"archivdms/internal/jobqueue"
"archivdms/internal/ldapauth"
"archivdms/internal/ldapstore"
"archivdms/internal/mailer"
"archivdms/internal/ocr"
"archivdms/internal/pagesplit"
"archivdms/internal/sftpserver"
"archivdms/internal/storage"
"archivdms/internal/tenantstore"
"archivdms/internal/thumbnail"
"archivdms/internal/userstore"
)
// AppVersion is the archivdms application version.
const AppVersion = "0.1.0-dev"
func main() {
if len(os.Args) > 1 {
switch os.Args[1] {
case "reminders":
runReminders(os.Args[2:])
return
case "reindex":
runReindex(os.Args[2:])
return
case "classify":
runClassify(os.Args[2:])
return
case "retention":
runRetention(os.Args[2:])
return
case "documents":
runDocuments(os.Args[2:])
return
case "version":
fmt.Printf("archivdms %s\n", AppVersion)
return
case "help", "--help", "-h":
printHelp()
return
case "serve":
os.Args = append(os.Args[:1], os.Args[2:]...)
}
}
configPath := flag.String("config", "/etc/archivdms/config.yml", "path to config file")
flag.Parse()
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
cfg, err := config.Load(*configPath)
if err != nil {
logger.Error("failed to load config", "path", *configPath, "err", err)
os.Exit(1)
}
// Derive the JWT signing key from the master secret via HKDF (same
// approach as archivmail: never sign JWTs with the raw config secret).
masterKey := []byte(cfg.API.Secret)
jwtKeyRaw := make([]byte, 32)
if _, err := io.ReadFull(hkdf.New(sha256.New, masterKey, []byte("archivdms-jwt-v1"), nil), jwtKeyRaw); err != nil {
logger.Error("key derivation failed", "err", err)
os.Exit(1)
}
jwtSecret := hex.EncodeToString(jwtKeyRaw)
storeCfg := storage.Config{
Dir: cfg.Storage.StorePath(),
DSN: cfg.Database.DSN(),
RetentionDays: cfg.Storage.RetentionDays,
}
docStore, err := storage.New(storeCfg)
if err != nil {
logger.Error("storage init failed", "err", err)
os.Exit(1)
}
defer docStore.Close()
// Optional full-text search index (internal/index, Manticore). Only wired
// when a DSN is configured — otherwise the store's Indexer stays nil and
// every sync call is a no-op (Postgres remains the source of truth).
if dsn := cfg.Index.ManticoreDSN; dsn != "" {
idxMgr, err := index.NewManticoreTenantManager(dsn)
if err != nil {
logger.Warn("search index disabled: manticore connection failed", "err", err)
} else {
docStore.SetIndexer(idxMgr, logger)
defer idxMgr.Close()
logger.Info("search index enabled (manticore)")
}
}
users, err := userstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("userstore init failed", "err", err)
os.Exit(1)
}
defer users.Close()
audlog, err := audit.New(cfg.Database.DSN(), cfg.Audit.ResolvedLogPath(), logger)
if err != nil {
logger.Error("audit init failed", "err", err)
os.Exit(1)
}
defer audlog.Close()
if err := seedDefaultUsers(users, logger); err != nil {
logger.Error("seed users failed", "err", err)
}
tenantSt, err := tenantstore.New(cfg.Database.DSN())
if err != nil {
logger.Error("tenant store init failed", "err", err)
os.Exit(1)
}
defer tenantSt.Close()
authMgr := auth.New(users, jwtSecret)
// Per-tenant LDAP directory integration (optional). The bind-password
// encryption key is derived (HKDF-SHA256) from the application master
// secret inside ldapstore; if that secret is empty, LDAP stays disabled
// and login remains local-only.
var ldapSt *ldapstore.Store
if cfg.API.Secret == "" {
logger.Warn("ldap disabled: api.secret is empty (no encryption key for bind passwords)")
} else {
ldapSt, err = ldapstore.New(cfg.Database.DSN(), cfg.API.Secret)
if err != nil {
logger.Error("ldap store init failed — ldap login disabled", "err", err)
ldapSt = nil
} else {
defer ldapSt.Close()
}
}
var ldapAuthn *ldapauth.Authenticator
if ldapSt != nil {
ldapAuthn = ldapauth.New(0)
authMgr.SetLDAP(ldapSt, ldapAuthn, tenantSt, audlog)
logger.Info("ldap directory integration enabled")
}
apiCfg := config.APIConfig{
Bind: cfg.API.Bind,
Secret: jwtSecret,
SecureCookies: cfg.API.SecureCookies,
TrustedProxies: cfg.API.TrustedProxies,
}
srv := api.New(apiCfg, docStore, authMgr, users, audlog, logger)
srv.SetTenants(tenantSt)
srv.SetLDAP(ldapSt, ldapAuthn)
srv.SetVersion(AppVersion)
srv.SetFQDN(cfg.Server.FQDN)
srv.SetStorageConfig(cfg.Storage)
extractor := ocr.New(
cfg.OCR.ResolvedTesseractPath(),
cfg.OCR.ResolvedPdftoppmPath(),
cfg.OCR.ResolvedLanguages(),
cfg.OCR.ResolvedTimeout(),
cfg.Storage.OCRTmpPath(),
)
extractor.Logger = logger
extractor.SofficePath = cfg.OCR.ResolvedSofficePath()
extractor.Binarize = cfg.OCR.BinarizeOCR
extractor.DeskewMethod = cfg.OCR.ResolvedDeskewMethod()
extractor.HoughDeskewScriptPath = cfg.OCR.ResolvedHoughDeskewScriptPath()
if extractor.DeskewMethod == "hough" {
if _, err := exec.LookPath("python3"); err != nil {
logger.Warn("python3 binary not found in PATH — ocr.deskew_method=hough will fall back to no deskew until installed (apt install python3 python3-opencv)", "err", err)
}
if _, err := os.Stat(extractor.HoughDeskewScriptPath); err != nil {
logger.Warn("hough_deskew.py not found at configured path — ocr.deskew_method=hough will fall back to no deskew", "path", extractor.HoughDeskewScriptPath, "err", err)
}
}
if _, err := exec.LookPath(extractor.SofficePath); err != nil {
logger.Warn("libreoffice (soffice) binary not found in PATH — Office-document ingest (docx/xlsx/pptx/odt/...) will be skipped until installed (apt install libreoffice-nogui)", "err", err)
}
if _, err := exec.LookPath(extractor.TesseractPath); err != nil {
logger.Warn("tesseract binary not found in PATH — OCR will be skipped for uploads until installed (apt install tesseract-ocr tesseract-ocr-deu)", "err", err)
}
if _, err := exec.LookPath(extractor.PdftoppmPath); err != nil {
logger.Warn("pdftoppm binary not found in PATH — scanned-PDF OCR fallback will be skipped until installed (apt install poppler-utils)", "err", err)
}
if _, err := exec.LookPath("convert"); err != nil {
logger.Warn("ImageMagick convert binary not found in PATH — skew correction will be skipped until installed (apt install imagemagick)", "err", err)
}
srv.SetOCR(extractor)
thumbGen := thumbnail.New(
cfg.OCR.ResolvedPdftoppmPath(),
"convert",
30*time.Second,
)
thumbGen.Logger = logger
srv.SetThumbnailer(thumbGen)
// Trennseiten-Split (internal/pagesplit): erkennt Barcode-Trennblätter in
// mehrseitigen PDF-Scans und zerlegt den Scan-Stapel beim Ingest in
// Einzeldokumente. Per Default AUS (config.pagesplit.enabled) — bewusst
// konservativ eingeführt wie ocr.binarize_ocr.
splitter := pagesplit.New(
cfg.PageSplit.Enabled,
cfg.PageSplit.Marker,
cfg.PageSplit.MarkerPrefix,
cfg.OCR.ResolvedPdftoppmPath(),
cfg.Storage.OCRTmpPath(),
)
splitter.Logger = logger
splitter.RasterDPI = cfg.PageSplit.RasterDPI
splitter.MaxPages = cfg.PageSplit.MaxPages
if cfg.PageSplit.TimeoutSeconds > 0 {
splitter.Timeout = time.Duration(cfg.PageSplit.TimeoutSeconds) * time.Second
}
if cfg.PageSplit.Enabled {
for _, bin := range []string{"pdfinfo", "pdfseparate", "pdfunite", "zbarimg"} {
if _, err := exec.LookPath(bin); err != nil {
logger.Warn("pagesplit enabled but required binary not found in PATH — separator-page splitting stays inactive until installed (apt install poppler-utils zbar-tools)", "binary", bin, "err", err)
}
}
}
srv.SetPageSplitter(splitter)
mlr := mailer.New(cfg.SMTPOut)
srv.SetMailer(mlr)
// Mandanten-faire Job-Queue (internal/jobqueue): Worker-Goroutinen im
// selben Prozess, die die nachgelagerte Dokumentverarbeitung (OCR,
// Taxonomie, on_upload-Workflows) aus processing_jobs abarbeiten. Der
// synchrone Upload-Pfad (Datei speichern + WORM + Job anlegen) läuft
// unabhängig davon weiter — ist die Queue deaktiviert, bleiben Dokumente
// einfach auf processing_status='queued' stehen, es geht nichts verloren.
if !cfg.JobQueue.Disabled {
queue := jobqueue.New(cfg.JobQueue, docStore, srv.ProcessDocumentJob, logger)
queue.Start(context.Background())
defer queue.Stop()
} else {
logger.Warn("job queue disabled by config — uploaded documents stay unprocessed (queued)")
}
// Embedded per-tenant SFTP server (internal/sftpserver) — only started
// when explicitly enabled in config; feeds the same upload pipeline as
// the HTTP endpoint via srv.StoreUploadedFile.
if cfg.SFTP.Enabled {
sftpSrv := sftpserver.New(cfg.SFTP, cfg.Storage, docStore, audlog, logger, srv.StoreUploadedFile)
if err := sftpSrv.Start(context.Background()); err != nil {
logger.Error("sftp server failed to start", "err", err)
} else {
defer sftpSrv.Stop()
}
}
bind := cfg.API.Bind
if bind == "" {
bind = fmt.Sprintf(":%d", cfg.Server.APIPort)
}
httpServer := &http.Server{
Addr: bind,
Handler: srv,
}
go func() {
logger.Info("starting API server", "addr", bind)
if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
logger.Error("API server error", "err", err)
}
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
logger.Info("shutting down...")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
httpServer.Shutdown(ctx)
}
// seedDefaultUsers creates a default superadmin account if no users exist yet.
func seedDefaultUsers(users *userstore.Store, logger *slog.Logger) error {
all, err := users.List("")
if err != nil {
return fmt.Errorf("list users: %w", err)
}
if len(all) > 0 {
return nil
}
pw, err := randomPassword()
if err != nil {
return fmt.Errorf("generate superadmin password: %w", err)
}
if _, err := users.Create(userstore.CreateUserRequest{
Username: "superadmin",
Email: "superadmin@archivdms.local",
Password: pw,
Role: userstore.RoleSuperAdmin,
}); err != nil {
return fmt.Errorf("create default superadmin: %w", err)
}
fmt.Println()
fmt.Println("╔══════════════════════════════════════════════════════════════╗")
fmt.Println("║ ARCHIVDMS — ERSTMALIGE EINRICHTUNG ║")
fmt.Printf("║ superadmin : %-47s ║\n", pw)
fmt.Println("║ Passwort sofort nach dem ersten Login ändern! ║")
fmt.Println("╚══════════════════════════════════════════════════════════════╝")
fmt.Println()
logger.Warn("default superadmin created — change password immediately!")
return nil
}
func randomPassword() (string, error) {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
func printHelp() {
fmt.Println(`archivdms — GoBD-konformes Dokumentenmanagementsystem
Usage:
archivdms serve [-config PATH] HTTP-API-Server starten
archivdms reminders notify [-config PATH] Fällige Wiedervorlagen per Mail benachrichtigen (Cron)
archivdms reindex [-config PATH] [-tenant N] Manticore-Volltextindex aus Postgres neu aufbauen
archivdms classify retrain [-config PATH] [-tenant N] [-dry-run] Naive-Bayes-Klassifikator neu trainieren (Cron)
archivdms retention apply [-config PATH] [-tenant N] [-dry-run] GoBD-Aufbewahrungsfristen berechnen und retain_until setzen (Cron)
archivdms documents reprocess-all [-config PATH] [-tenant N] [-dry-run] [-delay-ms N] Alle Dokumente sequenziell neu OCR-verarbeiten (Altbestand)
archivdms version Version anzeigen
archivdms help Diese Hilfe anzeigen`)
}
+17
View File
@@ -0,0 +1,17 @@
{
"$schema": "https://ui.shadcn.com/schema.json",
"style": "default",
"rsc": true,
"tsx": true,
"tailwind": {
"config": "tailwind.config.ts",
"css": "src/app/globals.css",
"baseColor": "slate",
"cssVariables": true,
"prefix": ""
},
"aliases": {
"components": "@/components",
"utils": "@/lib/utils"
}
}
+415
View File
@@ -0,0 +1,415 @@
// Package config loads the archivdms application configuration from a YAML
// file. The structure mirrors the archivmail config pattern (Server/Database/
// API/SMTPOut/Audit sections) but drops everything mail-specific (IMAP/POP3/
// SMTP daemon, index backend, etc.) since archivdms is a document-centric DMS.
package config
import (
"fmt"
"os"
"path/filepath"
"strings"
"time"
"gopkg.in/yaml.v3"
)
// APIConfig holds configuration for the HTTP API server.
type APIConfig struct {
Bind string `yaml:"bind"`
// Secret is the master secret from which the JWT signing key is derived.
Secret string `yaml:"secret"`
// SecureCookies sets the Secure flag on session cookies. Enable when TLS is
// terminated at this server or at a trusted reverse proxy.
SecureCookies bool `yaml:"secure_cookies"`
// TrustedProxies is a list of IP addresses or CIDR ranges whose
// X-Forwarded-For header is trusted. Empty = trust no proxy.
TrustedProxies []string `yaml:"trusted_proxies"`
}
// ServerConfig holds general server settings.
type ServerConfig struct {
FQDN string `yaml:"fqdn"` // used for generated links (invite/reset mails)
APIPort int `yaml:"api_port"`
}
// DatabaseConfig holds PostgreSQL connection settings.
type DatabaseConfig struct {
Host string `yaml:"host"`
Port int `yaml:"port"`
Name string `yaml:"name"`
User string `yaml:"user"`
Password string `yaml:"password"`
SSLMode string `yaml:"sslmode"`
}
// DSN builds a PostgreSQL connection string from the config fields.
func (d DatabaseConfig) DSN() string {
return fmt.Sprintf("postgres://%s:%s@%s:%d/%s?sslmode=%s",
d.User, d.Password, d.Host, d.Port, d.Name, d.SSLMode)
}
// SMTPOutConfig holds settings for outgoing transactional email (invites,
// password reset, reminder notifications).
type SMTPOutConfig struct {
Host string `yaml:"host"`
Port int `yaml:"port"`
User string `yaml:"user"`
Password string `yaml:"password"`
TLS bool `yaml:"tls"`
From string `yaml:"from"` // e.g. "archivdms <noreply@firma.de>"
}
// DefaultAuditLogPath is the default location of the append-only JSON-Lines
// audit log file when audit.log_path is not configured.
const DefaultAuditLogPath = "/var/log/archivdms/audit.log"
// AuditConfig holds audit log settings.
type AuditConfig struct {
LogPath string `yaml:"log_path"`
RetentionDays int `yaml:"retention_days"`
}
// ResolvedLogPath returns the configured audit log file path, falling back to
// DefaultAuditLogPath when unset.
func (a AuditConfig) ResolvedLogPath() string {
if strings.TrimSpace(a.LogPath) == "" {
return DefaultAuditLogPath
}
return a.LogPath
}
// LoggingConfig holds application logging settings.
type LoggingConfig struct {
Path string `yaml:"path"`
Level string `yaml:"level"`
}
// StorageConfig holds settings for document blob storage on disk.
//
// Layout under BasePath (see internal/ocr and internal/api upload handler):
//
// <BasePath>/inbox/<tenant_id>/<uuid>.<ext> raw upload, pre-processing
// <BasePath>/store/<tenant_id>/<yyyy>/<mm>/<sha256>.<ext> finished archive (WORM, 0440)
// <BasePath>/ocr-tmp/<uuid>/ pdftoppm scratch, removed after use
type StorageConfig struct {
// BasePath is the root directory for inbox/store/ocr-tmp (see helper
// methods below). Replaces the old, unused StorePath/store_path field.
BasePath string `yaml:"base_path"`
// RetentionDays is the default GoBD retention period (days) applied to new
// documents when no explicit retain_until is given. 0 = no default lock.
RetentionDays int `yaml:"retention_days"`
// MaxUploadSizeMB caps the accepted multipart upload size. 0 = default 50.
MaxUploadSizeMB int `yaml:"max_upload_size_mb"`
}
// InboxPath returns the directory raw uploads are written to before hashing
// and OCR processing.
func (s StorageConfig) InboxPath() string { return filepath.Join(s.BasePath, "inbox") }
// StorePath returns the directory the finished, content-addressed WORM
// archive lives in.
func (s StorageConfig) StorePath() string { return filepath.Join(s.BasePath, "store") }
// OCRTmpPath returns the scratch directory for pdftoppm intermediate images.
func (s StorageConfig) OCRTmpPath() string { return filepath.Join(s.BasePath, "ocr-tmp") }
// ThumbnailPath returns the directory holding derived preview thumbnails.
// Thumbnails are regenerable artefacts (not WORM): they may be deleted at any
// time and are re-created lazily on next request. Layout mirrors the store:
// thumbnails/<tenant_id>/<content_hash>.png.
func (s StorageConfig) ThumbnailPath() string { return filepath.Join(s.BasePath, "thumbnails") }
// ResolvedMaxUploadSizeMB returns MaxUploadSizeMB, falling back to a default
// of 50 MB when unset (<= 0).
func (s StorageConfig) ResolvedMaxUploadSizeMB() int {
if s.MaxUploadSizeMB <= 0 {
return 50
}
return s.MaxUploadSizeMB
}
// OCRConfig holds settings for the tesseract/poppler-utils OCR sidecar
// pipeline (internal/ocr). All binaries are optional system packages — a
// missing binary degrades OCR to a no-op rather than failing the upload.
type OCRConfig struct {
// TesseractPath is the path/name of the tesseract binary. Default "tesseract".
TesseractPath string `yaml:"tesseract_path"`
// PdftoppmPath is the path/name of the pdftoppm binary. Default "pdftoppm".
PdftoppmPath string `yaml:"pdftoppm_path"`
// Languages is the tesseract -l argument, e.g. "deu+eng". Default "deu+eng".
Languages string `yaml:"languages"`
// TimeoutSeconds bounds each individual OCR subprocess call. Default 60.
TimeoutSeconds int `yaml:"timeout_seconds"`
// SofficePath is the path/name of the LibreOffice headless binary used to
// convert Office documents (docx/xlsx/pptx/odt/...) to PDF before OCR.
// Default "soffice". A missing binary degrades Office ingest to a no-op.
SofficePath string `yaml:"soffice_path"`
// BinarizeOCR enables an Otsu auto-threshold (black/white) step at the end
// of the image preprocessing pipeline (internal/ocr.Extractor.Binarize),
// after deskew/contrast-normalize/OSD-rotation. Default false: this is a
// lossy step (every pixel becomes pure black or white) that helps flat
// text scans but can hurt documents with color stamps/signatures or
// embedded photos — enable per-deployment only after validating against
// that corpus. Requires ImageMagick 7 (`-auto-threshold` syntax); a
// missing/older convert binary just skips the step (best-effort, same as
// every other preprocessing step here).
BinarizeOCR bool `yaml:"binarize_ocr"`
// DeskewMethod selects the fine-skew-angle correction strategy used in
// internal/ocr.Extractor.runTesseract before the OSD 90-degree rotation
// pass. "imagemagick" (default/empty) keeps the existing behavior:
// ImageMagick's own `-deskew 40%` peak/valley background-projection
// analysis (internal/ocr.deskewImage). "hough" instead detects the angle
// via a Python/OpenCV sidecar script (internal/ocr/scripts/
// hough_deskew.py, minAreaRect/HoughLinesP-based) and applies it with a
// plain `convert -rotate <deg>` — separating angle detection from angle
// application, which the ImageMagick approach does not do and which
// fails on tightly-cropped phone photos lacking background margin.
// Requires python3 + opencv-python (Debian: python3-opencv) on the host;
// falls back to no-op (angle 0, same as any other best-effort
// preprocessing step here) if the script/dependency is missing.
DeskewMethod string `yaml:"deskew_method"`
// HoughDeskewScriptPath overrides the path to hough_deskew.py. Empty
// defaults to "/opt/archivdms/scripts/hough_deskew.py" (the on-premise
// install layout — see install.sh/update.sh, INSTALL_DIR=/opt/archivdms).
// Only consulted when DeskewMethod == "hough".
HoughDeskewScriptPath string `yaml:"hough_deskew_script_path"`
}
// ResolvedTesseractPath returns TesseractPath, defaulting to "tesseract".
func (o OCRConfig) ResolvedTesseractPath() string {
if strings.TrimSpace(o.TesseractPath) == "" {
return "tesseract"
}
return o.TesseractPath
}
// ResolvedPdftoppmPath returns PdftoppmPath, defaulting to "pdftoppm".
func (o OCRConfig) ResolvedPdftoppmPath() string {
if strings.TrimSpace(o.PdftoppmPath) == "" {
return "pdftoppm"
}
return o.PdftoppmPath
}
// ResolvedSofficePath returns SofficePath, defaulting to "soffice".
func (o OCRConfig) ResolvedSofficePath() string {
if strings.TrimSpace(o.SofficePath) == "" {
return "soffice"
}
return o.SofficePath
}
// ResolvedDeskewMethod returns DeskewMethod, defaulting to "imagemagick"
// (the pre-existing behavior — see the DeskewMethod field doc). Any value
// other than "hough" is treated as "imagemagick" so a typo in config.yaml
// degrades to the known-safe default rather than silently disabling deskew.
func (o OCRConfig) ResolvedDeskewMethod() string {
if strings.TrimSpace(strings.ToLower(o.DeskewMethod)) == "hough" {
return "hough"
}
return "imagemagick"
}
// ResolvedHoughDeskewScriptPath returns HoughDeskewScriptPath, defaulting to
// "/opt/archivdms/scripts/hough_deskew.py" (see the field doc).
func (o OCRConfig) ResolvedHoughDeskewScriptPath() string {
if strings.TrimSpace(o.HoughDeskewScriptPath) == "" {
return "/opt/archivdms/scripts/hough_deskew.py"
}
return o.HoughDeskewScriptPath
}
// ResolvedLanguages returns Languages, defaulting to "deu+eng".
func (o OCRConfig) ResolvedLanguages() string {
if strings.TrimSpace(o.Languages) == "" {
return "deu+eng"
}
return o.Languages
}
// ResolvedTimeout returns TimeoutSeconds as a time.Duration, defaulting to 60s.
func (o OCRConfig) ResolvedTimeout() time.Duration {
if o.TimeoutSeconds <= 0 {
return 60 * time.Second
}
return time.Duration(o.TimeoutSeconds) * time.Second
}
// SFTPConfig holds settings for the embedded per-tenant SFTP server
// (internal/sftpserver). Disabled by default — no port is opened unless
// explicitly enabled.
type SFTPConfig struct {
// Enabled turns the embedded SFTP server on/off. Default false.
Enabled bool `yaml:"enabled"`
// Bind is the listen address, e.g. ":2222".
Bind string `yaml:"bind"`
// HostKeyPath is where the server's SSH host key is persisted. Generated
// on first start if missing. Defaults to "<storage.base_path>/.ssh/host_key".
HostKeyPath string `yaml:"host_key_path"`
}
// ResolvedBind returns Bind, defaulting to ":2222".
func (c SFTPConfig) ResolvedBind() string {
if strings.TrimSpace(c.Bind) == "" {
return ":2222"
}
return c.Bind
}
// ResolvedHostKeyPath returns HostKeyPath, defaulting to
// "<basePath>/.ssh/host_key" when unset.
func (c SFTPConfig) ResolvedHostKeyPath(basePath string) string {
if strings.TrimSpace(c.HostKeyPath) != "" {
return c.HostKeyPath
}
return filepath.Join(basePath, ".ssh", "host_key")
}
// IndexConfig holds settings for the optional full-text search index
// (internal/index, Manticore Search over the MySQL protocol, port 9306).
//
// Phase 1 wires only the write/sync layer — there is no search endpoint yet.
// When ManticoreDSN is empty the index is disabled entirely: the store's
// Indexer stays nil and every sync call is a silent no-op (Postgres remains the
// single source of truth).
type IndexConfig struct {
// ManticoreDSN is a go-sql-driver/mysql DSN pointing at Manticore's SQL
// port, e.g. "archivdms@tcp(127.0.0.1:9306)/?charset=utf8mb4". Empty =
// index disabled.
ManticoreDSN string `yaml:"manticore_dsn"`
}
// JobQueueConfig holds settings for the tenant-fair, Postgres-backed
// processing queue (internal/jobqueue): OCR extraction, taxonomy
// auto-assignment and on_upload workflows run asynchronously in worker
// goroutines inside this same process (no separate service/container).
//
// All fields are optional — the defaults below are tuned for a single
// mid-sized server running Tesseract locally.
type JobQueueConfig struct {
// Disabled turns the async pipeline off entirely. Documents then stay in
// processing_status='queued' until the queue is enabled again (nothing is
// lost — the WORM file and the job row are already persisted). Default
// false, i.e. the queue runs.
Disabled bool `yaml:"disabled"`
// Workers is the number of concurrent worker goroutines. Default 2 —
// Tesseract is CPU-bound, more workers than cores hurts.
Workers int `yaml:"workers"`
// PollIntervalMS is how often the dispatcher looks for due jobs.
// Default 2000 (2s).
PollIntervalMS int `yaml:"poll_interval_ms"`
// JobTimeoutSeconds bounds one job run and doubles as the reaper's
// threshold for stuck 'processing' rows. Default 600 (10 min) — large
// multi-page scans plus LibreOffice conversion can legitimately take
// minutes.
JobTimeoutSeconds int `yaml:"job_timeout_seconds"`
// MaxRetries is the retry_count cap. Once exceeded, the job stays
// permanently 'failed' and is only retried on explicit manual request.
// Default 5.
MaxRetries int `yaml:"max_retries"`
}
// ResolvedWorkers returns Workers, defaulting to 2.
func (j JobQueueConfig) ResolvedWorkers() int {
if j.Workers <= 0 {
return 2
}
return j.Workers
}
// ResolvedPollInterval returns PollIntervalMS as a Duration, default 2s.
func (j JobQueueConfig) ResolvedPollInterval() time.Duration {
if j.PollIntervalMS <= 0 {
return 2 * time.Second
}
return time.Duration(j.PollIntervalMS) * time.Millisecond
}
// ResolvedJobTimeout returns JobTimeoutSeconds as a Duration, default 10min.
func (j JobQueueConfig) ResolvedJobTimeout() time.Duration {
if j.JobTimeoutSeconds <= 0 {
return 10 * time.Minute
}
return time.Duration(j.JobTimeoutSeconds) * time.Second
}
// ResolvedMaxRetries returns MaxRetries, defaulting to 5.
func (j JobQueueConfig) ResolvedMaxRetries() int {
if j.MaxRetries <= 0 {
return 5
}
return j.MaxRetries
}
// PageSplitConfig holds settings for barcode separator-page splitting at
// ingest (internal/pagesplit). A printed separator sheet carrying the
// configured barcode cuts a multi-page PDF scan into several individual
// documents; the separator page itself is discarded.
//
// Deliberately disabled by default and configured globally (config.yaml)
// rather than per tenant for now: this is the first iteration, and the
// project rule for new preprocessing behaviour is to introduce it
// conservatively and validate it in the field before turning it on broadly
// (same approach as ocr.binarize_ocr). A per-tenant setting plus a settings UI
// is the intended next step.
type PageSplitConfig struct {
// Enabled turns separator-page splitting on. Default false.
Enabled bool `yaml:"enabled"`
// Marker is the barcode payload identifying a separator page. Empty
// defaults to pagesplit.DefaultMarker ("ARCHIVDMS-SPLIT"). Matched
// case-insensitively after trimming.
Marker string `yaml:"marker"`
// MarkerPrefix switches matching from equality to prefix matching, so a
// separator sheet may carry additional payload after the marker.
MarkerPrefix bool `yaml:"marker_prefix"`
// RasterDPI is the resolution separator detection rasterizes pages at.
// 0 = default 150 (enough for a full-page separator barcode, far cheaper
// than the 300 dpi OCR pass).
RasterDPI int `yaml:"raster_dpi"`
// MaxPages caps how many pages are analysed; longer documents are archived
// unsplit. 0 = default 200.
MaxPages int `yaml:"max_pages"`
// TimeoutSeconds bounds each poppler subprocess call. 0 = default 120.
TimeoutSeconds int `yaml:"timeout_seconds"`
}
// Config is the full application configuration loaded from YAML.
type Config struct {
Server ServerConfig `yaml:"server"`
Storage StorageConfig `yaml:"storage"`
OCR OCRConfig `yaml:"ocr"`
PageSplit PageSplitConfig `yaml:"pagesplit"`
JobQueue JobQueueConfig `yaml:"jobqueue"`
SFTP SFTPConfig `yaml:"sftp"`
Index IndexConfig `yaml:"index"`
Database DatabaseConfig `yaml:"database"`
SMTPOut SMTPOutConfig `yaml:"smtp_out"`
API APIConfig `yaml:"api"`
Audit AuditConfig `yaml:"audit"`
Logging LoggingConfig `yaml:"logging"`
}
// Load reads a YAML config file from path and returns a parsed Config.
// It also bootstraps the storage directory tree (inbox/store/ocr-tmp) so the
// upload handler and OCR pipeline can rely on it existing at startup.
func Load(path string) (*Config, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var cfg Config
if err := yaml.Unmarshal(data, &cfg); err != nil {
return nil, err
}
if cfg.Storage.BasePath != "" {
for _, dir := range []string{cfg.Storage.InboxPath(), cfg.Storage.StorePath(), cfg.Storage.OCRTmpPath(), cfg.Storage.ThumbnailPath()} {
if err := os.MkdirAll(dir, 0o750); err != nil {
return nil, fmt.Errorf("config: create storage dir %s: %w", dir, err)
}
}
}
return &cfg, nil
}
+80
View File
@@ -0,0 +1,80 @@
server:
fqdn: "dms.example.com"
api_port: 8080
storage:
base_path: "/var/lib/archivdms" # enthält inbox/, store/, ocr-tmp/ (siehe README)
retention_days: 3650 # GoBD default: 10 Jahre
max_upload_size_mb: 50
ocr:
tesseract_path: "tesseract" # muss im PATH liegen, apt install tesseract-ocr tesseract-ocr-deu
pdftoppm_path: "pdftoppm" # apt install poppler-utils
languages: "deu+eng"
timeout_seconds: 60
pagesplit:
# Trennseiten-Split (internal/pagesplit): mehrseitige PDF-Scans werden beim
# Ingest an Barcode-Trennblättern in Einzeldokumente zerlegt. Das Trennblatt
# selbst wird verworfen, jedes Teildokument durchläuft den normalen Pfad
# (eigene WORM-Ablage, eigene OCR, eigene Autozuordnung). Der Split wird im
# Audit-Log als "document_split" mit Seitenbereichen + Dokument-IDs
# protokolliert (GoBD-Nachvollziehbarkeit).
# Braucht: apt install poppler-utils zbar-tools
enabled: false # Default AUS — erst nach Feldvalidierung aktivieren
marker: "ARCHIVDMS-SPLIT" # leer = Default "ARCHIVDMS-SPLIT" (Vergleich case-insensitiv)
marker_prefix: false # true = Trennblatt-Barcode muss nur mit marker BEGINNEN
raster_dpi: 150 # 0 = Default 150 (reicht für einen ganzseitigen Barcode)
max_pages: 200 # 0 = Default 200; längere PDFs werden ungesplittet archiviert
timeout_seconds: 120 # 0 = Default 120, je poppler-Aufruf
jobqueue:
# Mandanten-faire Verarbeitungs-Queue (internal/jobqueue): OCR, Taxonomie-
# Autozuordnung und on_upload-Workflows laufen asynchron in Worker-Goroutinen
# im selben Prozess (kein separater Dienst, kein Redis). Der Upload-Request
# macht nur noch Datei speichern + WORM + Job anlegen.
disabled: false # true = keine Nachverarbeitung (Dokumente bleiben "queued")
workers: 2 # Tesseract ist CPU-gebunden — nicht über die Kernzahl gehen
poll_interval_ms: 2000 # wie oft der Dispatcher nach fälligen Jobs schaut
job_timeout_seconds: 600 # Limit je Job, zugleich Schwelle des Reapers für hängende Jobs
max_retries: 5 # danach dauerhaft "failed", kein Automatik-Retry mehr
sftp:
enabled: false # eingebetteter SFTP-Server pro Mandant (internal/sftpserver)
bind: ":2222"
host_key_path: "" # leer = <storage.base_path>/.ssh/host_key (wird beim ersten Start generiert)
index:
# Volltext-Index (Manticore, MySQL-Protokoll Port 9306). Leer = deaktiviert.
# Phase 1: nur Schreib-/Sync-Layer, noch KEIN Such-Endpunkt.
manticore_dsn: "" # z.B. "archivdms@tcp(127.0.0.1:9306)/?charset=utf8mb4"
database:
host: "127.0.0.1"
port: 5432
name: "archivdms"
user: "archivdms"
password: "CHANGE_ME"
sslmode: "disable"
smtp_out:
host: "smtp.example.com"
port: 587
user: "noreply@example.com"
password: "CHANGE_ME"
tls: false
from: "archivdms <noreply@example.com>"
api:
bind: ":8080"
secret: "CHANGE_ME_TO_A_LONG_RANDOM_SECRET"
secure_cookies: true
trusted_proxies: []
audit:
log_path: "/var/log/archivdms/audit.log"
retention_days: 3650
logging:
path: "/var/log/archivdms/app.log"
level: "info"
+14
View File
@@ -0,0 +1,14 @@
# archivdms — Klassifizierer-Retraining (Naive-Bayes)
#
# Trainiert taeglich das Naive-Bayes-Klassifizierungsmodell fuer alle Tenants
# neu (document_types / correspondents / tags, siehe internal/classifier).
# Laeuft als eigenstaendiger CLI-Subcommand, kein internes Scheduler-Framework
# (Pattern analog archivdms-reminders / archivmail deploy/cron.d/*).
#
# Installation: nach /etc/cron.d/archivdms-classify-retrain kopieren.
MAILTO=""
# Minute Stunde Tag Monat Wochentag Nutzer Kommando
# 03:30 Uhr, versetzt zum stuendlichen Reminder-Job (Minute 5)
30 3 * * * archivdms /usr/local/bin/archivdms classify retrain -config /etc/archivdms/config.yml >> /var/log/archivdms/classify-retrain.log 2>&1
+13
View File
@@ -0,0 +1,13 @@
# archivdms — Wiedervorlage-Benachrichtigung (Reminder-Notify)
#
# Prüft stündlich fällige Wiedervorlagen und versendet Benachrichtigungen per
# Mail (siehe cmd/archivdms/cmd_reminders_notify.go). Läuft als eigenständiger
# CLI-Subcommand, kein internes Scheduler-Framework (Pattern analog
# archivmail deploy/cron.d/*).
#
# Installation: nach /etc/cron.d/archivdms-reminders kopieren.
MAILTO=""
# Minute Stunde Tag Monat Wochentag Nutzer Kommando
5 * * * * archivdms /usr/local/bin/archivdms reminders notify -config /etc/archivdms/config.yml >> /var/log/archivdms/reminders-notify.log 2>&1
+87
View File
@@ -0,0 +1,87 @@
# archivdms — Featureliste + System-Prompt (Synthese Paperless-ngx + ecoDMS + Marktstandards)
## Kern-Diff zu Paperless-ngx und ecoDMS
- Echte Multi-Tenancy (Paperless fehlt, ecoDMS nur Concurrent-Lizenz-Modell)
- Moderne API-first Architektur mit Webhooks (beide fehlen komplett)
- GoBD/E-Rechnung nativ statt nachgerüstet (ZUGFeRD/XRechnung Pflicht seit 2025)
- Go-Performance statt Django/Java (schnellere Ingestion, geringerer RAM)
- Hybrid-Suche BM25+Vektor via Manticore statt reiner Keyword-Suche
## Featureliste
### Ingestion
- Consume-Ordner + Mail-Postfach-Import (IMAP-Poll) + Scan-Client (TWAIN/Netzwerk-MFP)
- Barcode/QR-Split + ASN (Archive Serial Number)
- Office/E-Mail zu PDF/A Konvertierung (Gotenberg-Äquivalent)
- OCR Multi-Sprache (Tesseract)
- ZUGFeRD/XRechnung Parser: strukturierte Felder direkt extrahieren + validieren
### Klassifizierung
- Tags, Korrespondenten, Dokumenttypen, Custom Fields (typisiert: text/date/money/bool/select/link)
- ML Auto-Tagging (lernt aus bestätigten Docs) + optional LLM-Extraktion für Freitext-Felder
- Workflow-Engine: Trigger (Consumption/Schedule/Manual) + Action-Kette, State-Machine (Draft→Review→Approved→Archiviert)
- Storage-Path-Templates (Jinja/Go-Template) für Ablagestruktur
### Compliance (GoBD/DSGVO/eIDAS)
- WORM-Flag pro Dokument, Hash-Chain Audit-Trail (unveränderlich, selbst archiviert)
- Aufbewahrungsfristen-Engine (8-10 Jahre automatisch, Löschsperre vs. DSGVO-Löschanspruch-Konflikt lösbar)
- Mandantentrennung: Row-Level-Security PostgreSQL, pro Tenant eigener Storage-Pfad/Verschlüsselung
- E-Signatur-Integration (QES via externem Anbieter, Zertifikatsprüfung)
- Verfahrensdokumentation-Export (Compliance-Nachweis generierbar)
### Suche
- Hybrid: Manticore BM25 + Vektor-KNN (Embeddings), semantische Anfrage möglich
- Autocomplete, "more like this", gespeicherte Suchen/Views
### API / Integration
- REST + Webhooks (Event: created/updated/signed/deleted) — Kernlücke bei beiden Vorbildern schließen
- OpenAPI-Schema, Token/OIDC/SSO/LDAP Auth
- DATEV-Schnittstelle (developer.datev.de), ERP-Connectoren
- Bulk-Edit-Endpunkte (Tags/Type/Path/Permissions/Merge/Split)
- Public Share-Links mit Ablaufdatum, Login-Enforcement-Toggle, Multi-Doc-Share (Paperless-Lücke schließen)
### Berechtigungen
- RBAC granular bis Dokument-Ebene + Gruppen, Feld-Level-Security für Custom Fields
- Owner/View/Change Permission-Modell + Tenant-Scope
### UX
- Moderne SPA (React/Vue/Svelte), Mobile-first PWA
- Drag&Drop Batch-Upload, Inline-Preview (PDF/Bild/Office)
- Kanban-Board für Wiedervorlage/Freigabe-Status
- Dashboard mit Saved Views/Widgets
### Architektur (bereits gesetzt)
- Go Backend (API-first, hohe Concurrency für Multi-Tenant)
- PostgreSQL (RLS für Tenant-Isolation, Append-only Audit-Tabellen)
- Manticore Search (RT-Index + Vector-Suche, kein Extra-Vektor-DB nötig)
---
## System-Prompt-Entwurf (für Entwicklung/Claude-Session)
```
Du entwickelst archivdms, ein GoBD-konformes Dokumentenmanagementsystem für den DACH-Raum.
Stack: Go-Backend (API-first), PostgreSQL (Row-Level-Security für Mandantentrennung),
Manticore Search (Hybrid BM25+Vektor).
Differenzierung ggü. Paperless-ngx (OSS, aber keine Multi-Tenancy, keine Webhooks,
schwache Versionierung/Sharing) und ecoDMS (Java/Docker, GoBD-stark, aber veraltete UI,
gedeckelte API-Connects, schwache Workflow-Engine):
Pflichtfeatures:
1. Echte Multi-Tenancy mit PostgreSQL RLS
2. REST-API mit Webhooks für alle Dokument-Events
3. GoBD: WORM-Flag, Hash-Chain Audit-Trail, Aufbewahrungsfristen-Engine, Verfahrensdoku-Export
4. E-Rechnung: ZUGFeRD 2.0.1+/XRechnung Parser + Validierung (Pflicht seit 2025)
5. Hybrid-Suche (Manticore BM25 + Vektor-KNN)
6. Workflow-Engine mit State-Machine (Trigger+Actions, nicht nur einfache Freigabe)
7. Granulare RBAC bis Dokument-/Feld-Ebene
8. OIDC/SSO/LDAP, DATEV-Schnittstelle
9. Moderne responsive SPA, Kanban-Wiedervorlage, Inline-Preview
Nicht bauen: eigene E-Signatur-Engine (nur Integration), eigenes Konvertierungs-Backend
für Nischenformate (auf Standardlib/Sidecar setzen wie Gotenberg-Äquivalent).
Bei jeder Feature-Entscheidung: GoBD-Konformität und Mandantentrennung haben Vorrang
vor UX-Komfort. API-Design zuerst, UI konsumiert eigene API.
```
+77
View File
@@ -0,0 +1,77 @@
# PROJ-1: Wiedervorlage (Reminder pro Dokument)
## Zweck
Nutzer können ein Dokument mit einem Fälligkeitsdatum ("Wiedervorlage")
versehen, z.B. um eine Frist oder Rückmeldung nicht zu verpassen. Bei
Fälligkeit wird der Ersteller per E-Mail benachrichtigt.
## Datenmodell
Tabelle `reminders` (`internal/storage/reminders.go`):
```sql
CREATE TABLE IF NOT EXISTS reminders (
id BIGSERIAL PRIMARY KEY,
document_id BIGINT NOT NULL REFERENCES documents(id),
tenant_id BIGINT NOT NULL,
user_id BIGINT NOT NULL,
due_date TIMESTAMPTZ NOT NULL,
note TEXT,
status TEXT NOT NULL DEFAULT 'open', -- open|done|dismissed
notified_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_reminders_tenant_due ON reminders(tenant_id, due_date) WHERE status = 'open';
CREATE INDEX IF NOT EXISTS idx_reminders_document ON reminders(document_id);
```
## API (`internal/api/reminder_handlers.go`)
| Methode | Pfad | Beschreibung |
|---------|------------------------------------|--------------------------------------|
| POST | `/api/documents/{id}/reminders` | Wiedervorlage für Dokument anlegen |
| GET | `/api/reminders?status=` | Eigene Wiedervorlagen auflisten |
| PATCH | `/api/reminders/{id}` | Status ändern (open/done/dismissed) |
| DELETE | `/api/reminders/{id}` | Wiedervorlage löschen |
Alle Routen laufen durch `s.auth(...)` (Session + Tenant-Context). Ownership
wird in der Store-Schicht über `id + tenant_id + user_id` erzwungen. Jede
Statusänderung wird auditiert — auch fehlgeschlagene Versuche.
## Audit-Events (`internal/audit/audit.go`)
- `reminder_create`
- `reminder_status_change`
- `reminder_delete`
- `reminder_notify` (Cron-Benachrichtigung)
## Cron-Benachrichtigung
`archivdms reminders notify [-config PATH] [-dry-run]`
(`cmd/archivdms/cmd_reminders_notify.go`) liest alle offenen Wiedervorlagen
mit `due_date <= now()` und `notified_at IS NULL`, versendet eine E-Mail über
`internal/mailer`, und setzt `notified_at`. Läuft stündlich per
`deploy/cron.d/archivdms-reminders`.
## Frontend
- `src/components/reminders/CreateReminderButton.tsx` — Dialog zum Anlegen
(Datum via shadcn `calendar.tsx` + `popover.tsx`, Notiz-Textarea)
- `src/components/reminders/ReminderBadge.tsx` — Statusanzeige in der
Dokument-Detailansicht (offen/fällig/erledigt)
- `src/components/reminders/ReminderList.tsx` — Listendarstellung
- `src/app/reminders/page.tsx` — Übersichtsseite offen/erledigt
## Akzeptanzkriterien
1. Ein Nutzer kann für ein Dokument seines Tenants eine Wiedervorlage mit
Fälligkeitsdatum und optionaler Notiz anlegen.
2. `/reminders` zeigt eigene offene und erledigte Wiedervorlagen getrennt an.
3. Statusänderung (erledigt/verworfen) ist nur für den Ersteller möglich
(Ownership-Check via user_id).
4. Der Cron-Job versendet für jede fällige, noch nicht benachrichtigte
Wiedervorlage genau eine E-Mail und markiert sie danach als benachrichtigt.
5. Jede Statusänderung und jeder Benachrichtigungsversuch erscheint im
Audit-Log, inklusive Fehlschläge.
+11
View File
@@ -0,0 +1,11 @@
# Feature-Spezifikationen
Konvention übernommen von archivmail: jedes fachliche Feature bekommt eine
eigene `PROJ-N-name.md`-Datei in diesem Verzeichnis. Die Datei beschreibt
Zweck, Datenmodell, API, und Akzeptanzkriterien — geschrieben *bevor* oder
*während* der Implementierung, als lebendige Spec, nicht als nachträgliche
Doku.
## Index
- `PROJ-1-wiedervorlage.md` — Wiedervorlage (Reminder pro Dokument)
+23
View File
@@ -0,0 +1,23 @@
module archivdms
go 1.26.0
toolchain go1.26.5
require (
github.com/go-sql-driver/mysql v1.8.1
github.com/golang-jwt/jwt/v5 v5.2.1
github.com/jackc/pgx/v5 v5.6.0
github.com/pkg/sftp v1.13.7
golang.org/x/crypto v0.48.0
gopkg.in/yaml.v3 v3.0.1
)
require (
filippo.io/edwards25519 v1.1.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/text v0.34.0 // indirect
)
+552
View File
@@ -0,0 +1,552 @@
#!/bin/bash
# archivdms Installer — native Installation (kein Docker)
# Zielumgebung: Debian 13 (trixie), typischerweise ein unprivilegierter
# Proxmox-LXC-Container (funktioniert genauso auf VM/Bare-Metal — es gibt
# hier keinen LXC-spezifischen Zweig, siehe Abschnitt "LXC-Hinweise" unten).
#
# WICHTIG — kein Git-Remote: archivdms hat (Stand jetzt) kein Gitea/GitHub-
# Repository und soll dort auch nicht hochgeladen werden. Dieses Skript zieht
# den Quellcode daher NICHT per `git clone`/`git pull`, sondern erwartet, dass
# der Code bereits lokal auf dem Zielserver liegt (z.B. per rsync/scp vom
# Entwicklungsrechner kopiert). Quelle wird über die Umgebungsvariable
# ARCHIVDMS_SRC festgelegt; Default ist das Verzeichnis, in dem dieses Skript
# selbst liegt (dirname "$0") — das deckt den Normalfall ab, dass man
# install.sh direkt aus dem mitkopierten Projektverzeichnis heraus aufruft.
#
# Aufruf:
# sudo bash install.sh
# ARCHIVDMS_SRC=/opt/archivdms-src sudo bash install.sh
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m'
log() { echo -e "${GREEN}[OK]${NC} $*"; }
info() { echo -e "${BLUE}[..]${NC} $*"; }
warn() { echo -e "${YELLOW}[!!]${NC} $*"; }
die() { echo -e "${RED}[ERR]${NC} $*" >&2; exit 1; }
[[ $EUID -eq 0 ]] || die "Bitte als root ausführen: sudo bash install.sh"
# ── OS-Prüfung: Debian 13 (trixie) ist der getestete Zielstand ──────────────
# Anders als beim Hard-Fail in archivmail/install.sh nur eine Warnung: Ein DMS
# in einem LXC-Container wird öfter mal auf einem leicht abweichenden Stand
# betrieben (z.B. Debian 12 während einer Migrationsphase) — ein Hard-Block
# hilft hier niemandem, wichtig ist nur, dass der Betreiber es weiß.
_os_id=$(. /etc/os-release 2>/dev/null && echo "$ID" || echo "")
_os_ver=$(. /etc/os-release 2>/dev/null && echo "$VERSION_ID" || echo "")
if [[ "$_os_id" == "debian" && "$_os_ver" == "13" ]]; then
log "Debian 13 (trixie) erkannt"
else
warn "Getestet ist Debian 13 (trixie) — erkannt: ${_os_id:-unbekannt} ${_os_ver:-?}. Fahre trotzdem fort."
fi
# ── LXC-Erkennung — rein informativ ─────────────────────────────────────────
# Ein reiner systemd-Service + PostgreSQL + Go-Binary + Next.js-Standalone-
# Setup braucht in einem unprivilegierten LXC-Container keinerlei
# Sonderbehandlung (kein Nested-Docker, kein FUSE, keine Kernel-Module nötig).
# Der einzige Punkt, an dem unprivilegierte LXC-Container gelegentlich
# anecken, ist wenn Software eigene Netzwerk-Namespaces oder rohe Sockets
# braucht — das ist hier nicht der Fall (der Go-Server bindet normale TCP-
# Ports, das reicht auch unprivilegiert).
if command -v systemd-detect-virt >/dev/null 2>&1; then
_virt="$(systemd-detect-virt 2>/dev/null || echo unknown)"
if [[ "$_virt" == "lxc" ]]; then
info "LXC-Container erkannt (systemd-detect-virt: lxc) — kein Sonderhandling nötig für dieses Setup"
else
info "Virtualisierung: ${_virt} (kein LXC — dieses Skript funktioniert unabhängig davon identisch)"
fi
fi
# ── Gemeinsame Variablen ─────────────────────────────────────────────────────
DB_PASSWORD="${DB_PASSWORD:-$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 32)}"
API_SECRET="${API_SECRET:-$(openssl rand -base64 48 | tr -dc 'a-zA-Z0-9' | head -c 64)}"
INSTALL_DIR="/opt/archivdms"
STORE_BASE="/var/lib/archivdms"
LOG_DIR="/var/log/archivdms"
CONFIG_DIR="/etc/archivdms"
SSL_DIR="/etc/ssl/archivdms"
DMS_USER="archivdms"
FQDN="$(hostname -f 2>/dev/null || hostname)"
# Quellverzeichnis: entweder explizit über ARCHIVDMS_SRC vorgegeben, sonst das
# Verzeichnis, in dem install.sh selbst liegt (funktioniert, wenn der Code
# per rsync/scp komplett auf den Server kopiert und von dort ausgeführt wird).
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ARCHIVDMS_SRC="${ARCHIVDMS_SRC:-$SCRIPT_DIR}"
[[ -f "$ARCHIVDMS_SRC/go.mod" ]] || die "Kein Go-Quellcode unter ARCHIVDMS_SRC gefunden ($ARCHIVDMS_SRC/go.mod fehlt) — ARCHIVDMS_SRC korrekt setzen oder Quellcode dorthin kopieren."
echo ""
echo " ╔══════════════════════════════════════════╗"
echo " ║ archivdms Installer (nativ) ║"
echo " ╚══════════════════════════════════════════╝"
echo ""
info "Hostname: $FQDN"
info "Quellverzeichnis: $ARCHIVDMS_SRC"
echo ""
# ── 1. Basispakete ───────────────────────────────────────────────────────────
# ca-certificates/gnupg/curl explizit zuerst: schlanke LXC-Templates (z.B.
# Proxmox' debian-13-standard) liefern die teils NICHT vorinstalliert, das
# Manticore-Repo (HTTPS + GPG-Key) würde sonst weiter unten fehlschlagen.
info "Installiere Basispakete..."
apt-get update -qq
apt-get install -y -qq \
ca-certificates gnupg curl wget \
postgresql nginx \
rsync logrotate openssl sudo cron \
tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng poppler-utils
log "Basispakete installiert"
# ── 1b. Node.js ≥20 (empfohlen 24.x LTS) ────────────────────────────────────
# Debians eigenes nodejs-Paket ist für ein 2026er Next.js 16 i.d.R. zu alt.
# NodeSource liefert aktuelle LTS-Zeilen für Debian/trixie.
NODE_MAJOR_REQUIRED=20
_node_ok=0
if command -v node >/dev/null 2>&1; then
_node_major="$(node -v | sed -E 's/^v([0-9]+).*/\1/')"
[[ "$_node_major" =~ ^[0-9]+$ && "$_node_major" -ge "$NODE_MAJOR_REQUIRED" ]] && _node_ok=1
fi
if [[ $_node_ok -eq 1 ]]; then
log "Node.js $(node -v) OK (>= v${NODE_MAJOR_REQUIRED})"
else
info "Installiere Node.js 24.x LTS von NodeSource..."
curl -fsSL https://deb.nodesource.com/setup_24.x | bash - >/dev/null 2>&1 \
|| warn "NodeSource-Setup-Skript fehlgeschlagen — versuche apt trotzdem"
apt-get install -y -qq nodejs
command -v node >/dev/null 2>&1 && log "Node.js $(node -v) installiert" \
|| die "Node.js-Installation fehlgeschlagen"
fi
# ── 1c. Go ≥1.26 von upstream (apt-Paket ist zu alt) ────────────────────────
# Analog archivmail: /usr/local/go statt apt, damit die Version unabhängig
# von der Debian-Paketierung und OS-Version aktuell gehalten werden kann.
GO_VERSION="1.26.5"
info "Installiere Go ${GO_VERSION} von upstream..."
if /usr/local/go/bin/go version 2>/dev/null | grep -q "go${GO_VERSION}"; then
log "Go ${GO_VERSION} bereits installiert"
else
_go_arch="amd64"
[[ "$(dpkg --print-architecture)" == "arm64" ]] && _go_arch="arm64"
curl -fsSL "https://dl.google.com/go/go${GO_VERSION}.linux-${_go_arch}.tar.gz" \
| tar -C /usr/local -xz
log "Go $(/usr/local/go/bin/go version | awk '{print $3}') installiert"
fi
export PATH="$PATH:/usr/local/go/bin"
ln -sf /usr/local/go/bin/go /usr/local/bin/go
ln -sf /usr/local/go/bin/gofmt /usr/local/bin/gofmt
# ── 1d. Manticore Search ≥27.x ───────────────────────────────────────────────
# Kein natives trixie-Repo — das bookworm-Paket läuft auch auf trixie/Debian
# 14+, da Manticore kein OS-spezifisches ABI verwendet (siehe archivmail
# update.sh für den ausführlichen Hintergrund). Version wird dynamisch aus der
# Packages-Datei des Repos gelesen; Fallback nur falls das Repo mal nicht
# erreichbar/parsbar ist.
_MANTICORE_VERSION_FALLBACK="27.1.5-26061911-5a1cf9399"
_mc_installed_version="$(dpkg-query -W -f='${Version}' manticore 2>/dev/null || true)"
_mc_arch="$(dpkg --print-architecture)"
_mc_latest_from_repo() {
local base="https://repo.manticoresearch.com/repository/manticoresearch_bookworm"
local packages block version filename
packages="$(curl -fsSL --max-time 15 "${base}/dists/bookworm/main/binary-${_mc_arch}/Packages" 2>/dev/null)" || return 1
[[ -n "$packages" ]] || return 1
block="$(printf '%s\n' "$packages" | awk 'BEGIN{RS="\n\n"} {n=split($0,l,"\n"); if (l[1]=="Package: manticore") b=$0} END{print b}')"
[[ -n "$block" ]] || return 1
version="$(printf '%s\n' "$block" | awk -F': ' '/^Version:/{print $2; exit}')"
filename="$(printf '%s\n' "$block" | awk -F': ' '/^Filename:/{print $2; exit}')"
[[ -n "$version" && -n "$filename" ]] || return 1
printf '%s|%s\n' "$version" "$filename"
}
_mc_target_version="$_MANTICORE_VERSION_FALLBACK"
_mc_target_url="https://repo.manticoresearch.com/repository/manticoresearch_bookworm/dists/bookworm/main/binary-${_mc_arch}/manticore_${_MANTICORE_VERSION_FALLBACK}_${_mc_arch}.deb"
if _mc_latest="$(_mc_latest_from_repo)"; then
_mc_target_version="${_mc_latest%%|*}"
_mc_target_url="https://repo.manticoresearch.com/repository/manticoresearch_bookworm/${_mc_latest#*|}"
info "Manticore Search: aktuelle Repo-Version ${_mc_target_version} ermittelt"
else
warn "Manticore-Repo-Metadaten nicht abrufbar — falle auf bekannten Stand ${_MANTICORE_VERSION_FALLBACK} zurück"
fi
_mc_fetch_and_install() {
wget -q -O /tmp/manticore.deb "$_mc_target_url" || true
[[ -f /tmp/manticore.deb ]] || return 1
dpkg -i /tmp/manticore.deb 2>/dev/null || apt-get install -f -y -qq 2>/dev/null || true
rm -f /tmp/manticore.deb
return 0
}
if [[ -z "$_mc_installed_version" ]]; then
info "Installiere Manticore Search ${_mc_target_version}..."
_mc_fetch_and_install && log "Manticore Search installiert" \
|| warn "Manticore Search konnte nicht installiert werden — siehe: https://manticoresearch.com/install/ (Suche ist im Code laut README noch nicht integriert, kein Hard-Fail)"
elif dpkg --compare-versions "$_mc_installed_version" lt "$_mc_target_version"; then
info "Manticore Search ${_mc_installed_version} gefunden, aktualisiere auf ${_mc_target_version}..."
systemctl stop manticore 2>/dev/null || true
_mc_fetch_and_install \
&& log "Manticore Search aktualisiert: ${_mc_installed_version}${_mc_target_version}" \
|| warn "Upgrade fehlgeschlagen — vorherige Version bleibt installiert"
else
log "Manticore Search ${_mc_installed_version} bereits aktuell (>= ${_mc_target_version})"
fi
systemctl enable --now manticore 2>/dev/null || warn "Manticore-Dienst konnte nicht gestartet werden"
systemctl is-active --quiet manticore && log "Manticore Search läuft"
# ── 2. Systembenutzer ────────────────────────────────────────────────────────
# Dedizierter, nicht-interaktiver Service-User — der archivdms-Prozess läuft
# nie als root, im Unterschied etwa zu klassischen root-Cron-Setups.
info "Lege Systembenutzer '$DMS_USER' an..."
id "$DMS_USER" &>/dev/null \
&& log "Benutzer '$DMS_USER' existiert bereits" \
|| { useradd --system --shell /bin/false --home "$STORE_BASE" --create-home "$DMS_USER"; log "Benutzer angelegt"; }
# ── 2b. sudo-NOPASSWD-Whitelist für Dienststeuerung ─────────────────────────
# Analog archivmail PROJ-68: falls ein späteres Admin-UI Dienst-Restarts aus
# der Anwendung heraus auslösen soll, ist die Whitelist bereits vorbereitet.
# Validierung per `visudo -c` VOR dem Einspielen — eine kaputte sudoers-Datei
# würde sonst den gesamten sudo-Mechanismus auf dem Server lahmlegen.
# Idempotent: deterministischer Inhalt, überschreibt bei jedem Lauf.
provision_archivdms_sudoers() {
local dms_user="${1:-$DMS_USER}"
local services=("archivdms" "manticore" "postgresql" "nginx")
local tmpfile
tmpfile="$(mktemp)"
{
echo "# Verwaltet von install.sh/update.sh — nicht manuell editieren,"
echo "# Änderungen gehen beim nächsten Deploy verloren."
for svc in "${services[@]}"; do
echo "${dms_user} ALL=(root) NOPASSWD: /usr/bin/systemctl start ${svc}.service"
echo "${dms_user} ALL=(root) NOPASSWD: /usr/bin/systemctl stop ${svc}.service"
echo "${dms_user} ALL=(root) NOPASSWD: /usr/bin/systemctl restart ${svc}.service"
echo "${dms_user} ALL=(root) NOPASSWD: /usr/bin/systemctl enable ${svc}.service"
echo "${dms_user} ALL=(root) NOPASSWD: /usr/bin/systemctl disable ${svc}.service"
done
} > "$tmpfile"
if visudo -c -f "$tmpfile" >/dev/null 2>&1; then
install -m 0440 -o root -g root "$tmpfile" /etc/sudoers.d/archivdms
log "sudo-Rechte für Dienststeuerung eingerichtet (/etc/sudoers.d/archivdms)"
else
warn "Generierte sudoers-Regel ist ungültig — /etc/sudoers.d/archivdms NICHT verändert"
fi
rm -f "$tmpfile"
}
provision_archivdms_sudoers
# ── 3. Verzeichnisstruktur ───────────────────────────────────────────────────
# Layout gemäß README.md "Storage-Struktur & OCR" / config/config.go
# (StorageConfig.BasePath + InboxPath/StorePath/OCRTmpPath). config.Load()
# legt diese Unterverzeichnisse beim Serverstart selbst per os.MkdirAll an
# (0750) — install.sh legt die Wurzel bereits mit korrektem Owner an, damit
# der archivdms-User (nicht root) sie beim ersten Start auch beschreiben kann.
info "Erstelle Verzeichnisstruktur..."
mkdir -p "$STORE_BASE/inbox" "$STORE_BASE/store" "$STORE_BASE/ocr-tmp"
mkdir -p "$CONFIG_DIR" "$LOG_DIR" "$INSTALL_DIR" "$SSL_DIR"
chown -R "$DMS_USER:$DMS_USER" "$STORE_BASE" "$LOG_DIR"
chmod 750 "$STORE_BASE" "$STORE_BASE/inbox" "$STORE_BASE/store" "$STORE_BASE/ocr-tmp"
log "Verzeichnisse erstellt ($STORE_BASE/{inbox,store,ocr-tmp})"
# ── 4. TLS-Zertifikat (selbstsigniert, Let's-Encrypt-Option siehe unten) ────
info "Erstelle selbstsigniertes TLS-Zertifikat..."
if [[ ! -f "$SSL_DIR/archivdms.crt" ]]; then
SERVER_IP="$(hostname -I | awk '{print $1}')"
openssl req -x509 -nodes -days 3650 -newkey rsa:4096 \
-keyout "$SSL_DIR/archivdms.key" \
-out "$SSL_DIR/archivdms.crt" \
-subj "/CN=${FQDN}/O=archivdms/C=DE" \
-addext "subjectAltName=DNS:${FQDN},DNS:$(hostname -s),IP:${SERVER_IP}" \
2>/dev/null
chmod 640 "$SSL_DIR/archivdms.key"
chmod 644 "$SSL_DIR/archivdms.crt"
chown "root:$DMS_USER" "$SSL_DIR/archivdms.key"
log "TLS-Zertifikat erstellt: $SSL_DIR/archivdms.crt"
else
log "TLS-Zertifikat existiert bereits wird nicht überschrieben"
fi
# ── 5. PostgreSQL ─────────────────────────────────────────────────────────
info "Richte PostgreSQL ein..."
systemctl enable postgresql --quiet
systemctl start postgresql
# Falls config.yml schon existiert, DB-Passwort daraus übernehmen — verhindert
# Passwort-Mismatch zwischen laufender DB-Rolle und Config bei Re-Install.
if [[ -f "$CONFIG_DIR/config.yml" ]]; then
EXISTING_PW=$(grep -A6 '^database:' "$CONFIG_DIR/config.yml" | awk '/password:/{print $2}' | head -1 | tr -d '"')
[[ -n "$EXISTING_PW" ]] && DB_PASSWORD="$EXISTING_PW" && info "DB-Passwort aus vorhandener config.yml übernommen"
fi
su -c "psql -tc \"SELECT 1 FROM pg_roles WHERE rolname='archivdms'\" | grep -q 1 \
&& psql -c \"ALTER USER archivdms WITH PASSWORD '$DB_PASSWORD'\" \
|| psql -c \"CREATE USER archivdms WITH PASSWORD '$DB_PASSWORD'\"" postgres
su -c "psql -tc \"SELECT 1 FROM pg_database WHERE datname='archivdms'\" | grep -q 1 || \
psql -c \"CREATE DATABASE archivdms OWNER archivdms\"" postgres
su -c "psql archivdms -c \"GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO archivdms;\"" postgres
su -c "psql archivdms -c \"GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO archivdms;\"" postgres
su -c "psql archivdms -c \"ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO archivdms;\"" postgres
su -c "psql archivdms -c \"ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO archivdms;\"" postgres
log "PostgreSQL eingerichtet"
# ── 6. Konfiguration (config.yml aus config.yml.example) ────────────────────
info "Erstelle Konfigurationsdatei..."
if [[ ! -f "$CONFIG_DIR/config.yml" ]]; then
[[ -f "$ARCHIVDMS_SRC/config/config.yml.example" ]] \
|| die "config/config.yml.example nicht gefunden unter $ARCHIVDMS_SRC"
sed \
-e "s/CHANGE_ME_TO_A_LONG_RANDOM_SECRET/$API_SECRET/g" \
-e "s/password: \"CHANGE_ME\"/password: \"$DB_PASSWORD\"/" \
-e "s/dms\.example\.com/$FQDN/" \
"$ARCHIVDMS_SRC/config/config.yml.example" \
> "$CONFIG_DIR/config.yml"
# storage.base_path und Log-Pfade auf die tatsächlich angelegten
# Verzeichnisse zeigen lassen (das Beispiel verwendet dieselben Defaults,
# sed hier trotzdem defensiv falls sich das Example mal ändert).
sed -i \
-e "s#base_path: \".*\"#base_path: \"$STORE_BASE\"#" \
-e "s#log_path: \".*\"#log_path: \"$LOG_DIR/audit.log\"#" \
-e "s#path: \"/var/log/archivdms/app.log\"#path: \"$LOG_DIR/app.log\"#" \
"$CONFIG_DIR/config.yml"
chmod 640 "$CONFIG_DIR/config.yml"
chown "root:$DMS_USER" "$CONFIG_DIR/config.yml"
log "config.yml erstellt: $CONFIG_DIR/config.yml"
else
log "config.yml existiert bereits wird nicht überschrieben"
fi
# ── 7. nginx Reverse-Proxy (mit Let's-Encrypt-Option) ───────────────────────
info "Konfiguriere nginx (HTTP → HTTPS + TLS)..."
cat > /etc/nginx/sites-available/archivdms << NGINX
# HTTP → HTTPS Redirect
server {
listen 80;
server_name ${FQDN} _;
return 301 https://\$host\$request_uri;
}
# HTTPS
server {
listen 443 ssl;
http2 on;
server_name ${FQDN} _;
ssl_certificate ${SSL_DIR}/archivdms.crt;
ssl_certificate_key ${SSL_DIR}/archivdms.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
add_header Strict-Transport-Security "max-age=31536000" always;
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options SAMEORIGIN always;
location /api/ {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto \$scheme;
client_max_body_size 512M;
}
location / {
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade \$http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto \$scheme;
proxy_cache_bypass \$http_upgrade;
}
access_log /var/log/nginx/archivdms.access.log;
error_log /var/log/nginx/archivdms.error.log;
}
NGINX
ln -sf /etc/nginx/sites-available/archivdms /etc/nginx/sites-enabled/archivdms
rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl enable nginx --quiet
systemctl restart nginx
log "nginx konfiguriert"
# Let's-Encrypt-Option: nur ein Hinweis, kein automatischer Zwang zu einer
# öffentlichen Domain — viele DMS-Installationen laufen intern ohne
# öffentlich auflösbaren FQDN, wo certbot ohnehin fehlschlagen würde.
if command -v certbot >/dev/null 2>&1; then
info "certbot bereits installiert — Let's-Encrypt-Zertifikat manuell holen mit:"
echo " certbot --nginx -d ${FQDN}"
else
info "Let's-Encrypt-Zertifikat gewünscht? Falls ${FQDN} öffentlich auflösbar ist:"
echo " apt-get install -y certbot python3-certbot-nginx && certbot --nginx -d ${FQDN}"
fi
# ── 8. logrotate ──────────────────────────────────────────────────────────
cat > /etc/logrotate.d/archivdms << LOGROTATE
${LOG_DIR}/*.log {
daily
rotate 365
compress
delaycompress
missingok
notifempty
create 640 ${DMS_USER} ${DMS_USER}
}
LOGROTATE
log "logrotate konfiguriert"
# ── 9. Cron-Job für Wiedervorlage-Benachrichtigung ─────────────────────────
# deploy/cron.d/archivdms-reminders liegt bereits fertig im Projekt (siehe
# README "Struktur") — hier nur nach /etc/cron.d/ kopieren.
info "Installiere Cron-Job für Wiedervorlage-Benachrichtigung..."
if [[ -f "$ARCHIVDMS_SRC/deploy/cron.d/archivdms-reminders" ]]; then
cp "$ARCHIVDMS_SRC/deploy/cron.d/archivdms-reminders" /etc/cron.d/archivdms-reminders
chmod 644 /etc/cron.d/archivdms-reminders
systemctl enable --now cron --quiet 2>/dev/null || true
log "Cron-Job installiert: /etc/cron.d/archivdms-reminders"
else
warn "deploy/cron.d/archivdms-reminders nicht gefunden — Wiedervorlage-Mails laufen nicht automatisch"
fi
# ── 9b. Cron-Job für Klassifizierer-Retraining ─────────────────────────────
info "Installiere Cron-Job für Klassifizierer-Retraining..."
if [[ -f "$ARCHIVDMS_SRC/deploy/cron.d/archivdms-classify-retrain" ]]; then
cp "$ARCHIVDMS_SRC/deploy/cron.d/archivdms-classify-retrain" /etc/cron.d/archivdms-classify-retrain
chmod 644 /etc/cron.d/archivdms-classify-retrain
systemctl enable --now cron --quiet 2>/dev/null || true
log "Cron-Job installiert: /etc/cron.d/archivdms-classify-retrain"
else
warn "deploy/cron.d/archivdms-classify-retrain nicht gefunden — Klassifizierer-Retraining läuft nicht automatisch"
fi
# ── 10. systemd Unit für 'archivdms serve' ──────────────────────────────────
# Der SFTP-Server läuft im selben Prozess mit (kein eigener systemd-Dienst) —
# er startet nur, wenn cfg.SFTP.Enabled in config.yml gesetzt ist
# (siehe cmd/archivdms/main.go).
info "Erstelle systemd Unit..."
cat > /etc/systemd/system/archivdms.service << UNIT
[Unit]
Description=archivdms GoBD-DMS Daemon (API + eingebetteter SFTP-Server)
After=network.target postgresql.service manticore.service
Wants=manticore.service
Requires=postgresql.service
[Service]
Type=simple
User=${DMS_USER}
Group=${DMS_USER}
ExecStart=${INSTALL_DIR}/bin/archivdms serve -config ${CONFIG_DIR}/config.yml
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=archivdms
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=${STORE_BASE} ${LOG_DIR}
ReadOnlyPaths=${CONFIG_DIR} ${SSL_DIR}
[Install]
WantedBy=multi-user.target
UNIT
cat > /etc/systemd/system/archivdms-web.service << UNIT
[Unit]
Description=archivdms Next.js Frontend
After=network.target archivdms.service
[Service]
Type=simple
User=${DMS_USER}
Group=${DMS_USER}
WorkingDirectory=${INSTALL_DIR}/web
ExecStart=/usr/bin/node server.js
Environment=NODE_ENV=production
Environment=PORT=3000
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=archivdms-web
[Install]
WantedBy=multi-user.target
UNIT
systemctl daemon-reload
systemctl enable archivdms archivdms-web --quiet
log "systemd Units erstellt und aktiviert"
# ── 11. Erstes Build + Deployment via update.sh ─────────────────────────────
info "Führe update.sh für den ersten Build/Deploy aus..."
if [[ -f "$SCRIPT_DIR/update.sh" ]]; then
chmod +x "$SCRIPT_DIR/update.sh"
ARCHIVDMS_SRC="$ARCHIVDMS_SRC" bash "$SCRIPT_DIR/update.sh"
else
warn "update.sh nicht gefunden neben install.sh — Build/Start manuell nötig: bash update.sh"
fi
# ── Superadmin-Passwort aus dem Log ziehen (siehe main.go seedDefaultUsers) ─
PW_SUPERADMIN="(nicht gefunden — journalctl -u archivdms | grep superadmin)"
for _i in $(seq 1 15); do
_pw=$(journalctl -u archivdms --no-pager -n 200 2>/dev/null | grep -A1 'superadmin :' | grep -oP 'superadmin\s*:\s*\K\S+' | tail -1)
if [[ -n "$_pw" ]]; then
PW_SUPERADMIN="$_pw"
break
fi
sleep 1
done
# ── Zusammenfassung ───────────────────────────────────────────────────────
SUMMARY_FILE="$CONFIG_DIR/install-summary.txt"
cat > "$SUMMARY_FILE" << SUMMARY
archivdms Native-Installation — $(date '+%d.%m.%Y %H:%M:%S')
Server: $FQDN
=== ZUGANGSDATEN ===
Datenbank: archivdms / $DB_PASSWORD
API-Secret: $API_SECRET
Web-Login (UNBEDINGT ÄNDERN!):
superadmin / $PW_SUPERADMIN
=== DIENSTE ===
Web (HTTPS): https://$FQDN
API intern: 127.0.0.1:8080
Frontend intern: 127.0.0.1:3000
=== DATEIPFADE ===
Konfiguration: $CONFIG_DIR/config.yml
Storage: $STORE_BASE/{inbox,store,ocr-tmp}
TLS-Zertifikat: $SSL_DIR/archivdms.crt
Logs: $LOG_DIR/
Updater: $SCRIPT_DIR/update.sh
Quellcode: $ARCHIVDMS_SRC
SUMMARY
chmod 600 "$SUMMARY_FILE"
log "Zusammenfassung: $SUMMARY_FILE"
# ── Abschluss ────────────────────────────────────────────────────────────
echo ""
echo " ╔══════════════════════════════════════════════════════════╗"
echo " ║ Installation abgeschlossen! ║"
echo " ╚══════════════════════════════════════════════════════════╝"
echo ""
echo " Web (HTTPS): https://$FQDN"
echo ""
echo " ┌─────────────────────────────────────────────────────────┐"
printf " │ DB archivdms: %-40s │\n" "$DB_PASSWORD"
printf " │ superadmin / %-40s │\n" "$PW_SUPERADMIN"
echo " └─────────────────────────────────────────────────────────┘"
echo ""
warn "Zusammenfassung mit Passwörtern: $SUMMARY_FILE"
warn "Standardpasswort unbedingt nach dem ersten Login ändern!"
echo ""
+391
View File
@@ -0,0 +1,391 @@
// Buchhaltungs-Pull-API: a reduced, read-only, machine-to-machine export path
// so an accounting system (DATEV-Vorerfassung, Kanzlei-Software, ...) can pull
// belegdatum-scored documents out of archivdms without a browser session.
//
// Two clearly separated halves:
//
// 1. Key administration — normal JWT-cookie/session endpoints (domain_admin+,
// same pattern as retention_rule_handlers.go):
//
// POST /api/accounting/api-keys create, returns the plaintext key ONCE
// GET /api/accounting/api-keys list (label/timestamps only, no key)
// DELETE /api/accounting/api-keys/{id} revoke (never hard-deleted)
//
// 2. The pull endpoints themselves — NOT wrapped in s.auth. They use
// s.accountingAuth (Authorization: Bearer <key>) instead:
//
// GET /api/v1/accounting/documents keyset-paginated metadata
// GET /api/v1/accounting/documents/{id}/file streams the WORM file
//
// TENANT ISOLATION (critical — this is the only non-browser access path):
// s.accountingAuth resolves the raw bearer key to a tenant id via
// storage.ResolveAccountingAPIKey and puts ONLY that id into the request
// context (accountingTenantKey). The pull handlers read the tenant id
// exclusively from that context via accountingCtxFromRequest; there is no code
// path in which a tenant_id from the query string, a header or a body is
// consulted. The store functions they call (ListAccountingDocuments,
// GetAccountingDocumentFile) take tenantID as a mandatory first argument and
// have no unscoped variant. A document belonging to another tenant is
// indistinguishable from a nonexistent one (404, never 403).
package api
import (
"context"
"encoding/json"
"errors"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
const (
accountingTenantKey contextKey = "accounting_tenant_id"
accountingKeyIDKey contextKey = "accounting_key_id"
)
// accountingMaxLimit caps the page size a client may request.
const accountingMaxLimit = 500
// accountingDefaultLimit is used when no (or an invalid) limit is given.
const accountingDefaultLimit = 100
// --- key administration (session-authenticated, domain_admin+) ---
// createAccountingKeyRequest is the JSON body for POST /api/accounting/api-keys.
type createAccountingKeyRequest struct {
Label string `json:"label"`
}
// createAccountingKeyResponse is the ONLY place the plaintext key is ever
// returned. It is not persisted anywhere in plaintext and cannot be retrieved
// again.
type createAccountingKeyResponse struct {
Key storage.AccountingAPIKey `json:"key"`
// PlaintextKey is shown exactly once — the caller must store it now.
PlaintextKey string `json:"plaintext_key"`
}
// handleCreateAccountingAPIKey handles POST /api/accounting/api-keys (domain_admin+).
func (s *Server) handleCreateAccountingAPIKey(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req createAccountingKeyRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
label := strings.TrimSpace(req.Label)
if label == "" {
writeError(w, http.StatusBadRequest, "label is required")
return
}
userID := sess.UserID
key, plaintext, err := s.store.CreateAccountingAPIKey(r.Context(), *sess.TenantID, label, &userID)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingKeyCreated, Username: sess.Username, IPAddress: s.remoteIP(r),
TenantID: sess.TenantID, Success: false,
Detail: "accounting_key_create label:" + label + " err:" + err.Error(),
})
writeError(w, http.StatusInternalServerError, "create accounting api key failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingKeyCreated, Username: sess.Username, IPAddress: s.remoteIP(r),
TenantID: sess.TenantID, Success: true,
Detail: "accounting_key_create id:" + strconv.FormatInt(key.ID, 10) + " label:" + label,
})
writeJSON(w, http.StatusCreated, createAccountingKeyResponse{Key: *key, PlaintextKey: plaintext})
}
// handleListAccountingAPIKeys handles GET /api/accounting/api-keys (domain_admin+).
// Never returns the plaintext key or its hash.
func (s *Server) handleListAccountingAPIKeys(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
keys, err := s.store.ListAccountingAPIKeys(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list accounting api keys failed")
return
}
if keys == nil {
keys = []storage.AccountingAPIKey{}
}
writeJSON(w, http.StatusOK, keys)
}
// handleRevokeAccountingAPIKey handles DELETE /api/accounting/api-keys/{id}
// (domain_admin+). Revoke only — the row stays so the audit trail of past
// pulls remains resolvable.
func (s *Server) handleRevokeAccountingAPIKey(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.RevokeAccountingAPIKey(r.Context(), id, *sess.TenantID); err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingKeyRevoked, Username: sess.Username, IPAddress: s.remoteIP(r),
TenantID: sess.TenantID, Success: false,
Detail: "accounting_key_revoke id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(),
})
if errors.Is(err, storage.ErrAccountingKeyNotFound) {
writeError(w, http.StatusNotFound, "accounting api key not found")
return
}
writeError(w, http.StatusInternalServerError, "revoke accounting api key failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingKeyRevoked, Username: sess.Username, IPAddress: s.remoteIP(r),
TenantID: sess.TenantID, Success: true,
Detail: "accounting_key_revoke id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "revoked"})
}
// --- bearer-key middleware for the pull endpoints ---
// accountingAuth is the API-key middleware for the pull endpoints. It is
// deliberately separate from s.authMiddleware (JWT cookie): no session, no
// role, no user — just a tenant-scoped machine credential.
//
// It puts the tenant id resolved FROM THE KEY into the request context. This is
// the single source of truth for tenant scoping downstream; handlers must never
// read a tenant id from the request itself.
func (s *Server) accountingAuth(h http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ip := s.remoteIP(r)
// Per-IP rate limit blunts key guessing on this unauthenticated-until-
// resolved path (same limiter type as the public share endpoints).
if !s.accountingLimiter.allow(ip) {
writeError(w, http.StatusTooManyRequests, "too many requests")
return
}
rawKey := extractBearerToken(r)
if rawKey == "" {
w.Header().Set("WWW-Authenticate", "Bearer")
writeError(w, http.StatusUnauthorized, "missing bearer api key")
return
}
tenantID, keyID, err := s.store.ResolveAccountingAPIKey(r.Context(), rawKey)
if err != nil {
if !errors.Is(err, storage.ErrAccountingKeyNotFound) {
s.logger.Error("accounting api key resolve failed", "err", err)
}
// Unknown, revoked and broken keys are indistinguishable.
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: ip,
Success: false, Detail: "accounting_auth rejected path:" + r.URL.Path,
})
w.Header().Set("WWW-Authenticate", "Bearer")
writeError(w, http.StatusUnauthorized, "invalid api key")
return
}
ctx := context.WithValue(r.Context(), accountingTenantKey, tenantID)
ctx = context.WithValue(ctx, accountingKeyIDKey, keyID)
h(w, r.WithContext(ctx))
}
}
// accountingCtxFromRequest returns the tenant id and key id that
// accountingAuth resolved. ok is false only if the handler was somehow reached
// without the middleware — handlers then must refuse to do anything.
func accountingCtxFromRequest(ctx context.Context) (tenantID, keyID int64, ok bool) {
t, tOK := ctx.Value(accountingTenantKey).(int64)
k, kOK := ctx.Value(accountingKeyIDKey).(int64)
if !tOK || !kOK {
return 0, 0, false
}
return t, k, true
}
// --- pull endpoints (bearer-key authenticated) ---
// handleAccountingListDocuments handles
// GET /api/v1/accounting/documents?since=&until=&doc_type_id=&min_date_score=&cursor=&limit=
//
// since/until are dates (YYYY-MM-DD or RFC3339) bounding document_date;
// min_date_score gates on the belegdatum confidence (e.g. 0.75); cursor/limit
// drive keyset pagination over (created_at, id). Any tenant_id query parameter
// is ignored — scoping comes from the API key alone.
func (s *Server) handleAccountingListDocuments(w http.ResponseWriter, r *http.Request) {
tenantID, keyID, ok := accountingCtxFromRequest(r.Context())
if !ok {
writeError(w, http.StatusUnauthorized, "invalid api key")
return
}
q := r.URL.Query()
filter := storage.AccountingDocumentFilter{
Cursor: q.Get("cursor"),
Limit: accountingDefaultLimit,
}
if v := strings.TrimSpace(q.Get("since")); v != "" {
t, err := parseAccountingDate(v)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid since (expected YYYY-MM-DD or RFC3339)")
return
}
filter.Since = &t
}
if v := strings.TrimSpace(q.Get("until")); v != "" {
t, err := parseAccountingDate(v)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid until (expected YYYY-MM-DD or RFC3339)")
return
}
filter.Until = &t
}
if v := strings.TrimSpace(q.Get("doc_type_id")); v != "" {
id, err := strconv.ParseInt(v, 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid doc_type_id")
return
}
filter.DocTypeID = &id
}
if v := strings.TrimSpace(q.Get("min_date_score")); v != "" {
score, err := strconv.ParseFloat(v, 64)
if err != nil || score < 0 || score > 1 {
writeError(w, http.StatusBadRequest, "invalid min_date_score (expected 0..1)")
return
}
filter.MinDateScore = &score
}
if v := strings.TrimSpace(q.Get("limit")); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n <= 0 {
writeError(w, http.StatusBadRequest, "invalid limit")
return
}
if n > accountingMaxLimit {
n = accountingMaxLimit
}
filter.Limit = n
}
page, err := s.store.ListAccountingDocuments(r.Context(), tenantID, filter)
if err != nil {
if errors.Is(err, storage.ErrInvalidAccountingCursor) {
writeError(w, http.StatusBadRequest, "invalid cursor")
return
}
s.logger.Error("accounting list failed", "tenant_id", tenantID, "err", err)
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
TenantID: &tenantID, Success: false,
Detail: "accounting_pull list key:" + strconv.FormatInt(keyID, 10) + " err:" + err.Error(),
})
writeError(w, http.StatusInternalServerError, "list documents failed")
return
}
if page.Documents == nil {
page.Documents = []storage.AccountingDocument{}
}
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
TenantID: &tenantID, Success: true,
Detail: "accounting_pull list key:" + strconv.FormatInt(keyID, 10) +
" count:" + strconv.Itoa(len(page.Documents)) +
" range:" + accountingIDRange(page.Documents),
})
writeJSON(w, http.StatusOK, page)
}
// handleAccountingDocumentFile handles GET /api/v1/accounting/documents/{id}/file.
// Streams the archived WORM file through the handler — storage_path is never
// exposed. Scoped to the API key's tenant; a foreign or unknown document both
// yield 404 (no existence leak, mirroring handleGetDocumentFile).
func (s *Server) handleAccountingDocumentFile(w http.ResponseWriter, r *http.Request) {
tenantID, keyID, ok := accountingCtxFromRequest(r.Context())
if !ok {
writeError(w, http.StatusUnauthorized, "invalid api key")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
ref, err := s.store.GetAccountingDocumentFile(r.Context(), id, tenantID)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
TenantID: &tenantID, DocumentID: strconv.FormatInt(id, 10), Success: false,
Detail: "accounting_pull file key:" + strconv.FormatInt(keyID, 10) + " not_found",
})
writeError(w, http.StatusNotFound, "document not found")
return
}
f, err := os.Open(ref.StoragePath())
if err != nil {
s.logger.Error("accounting file open failed", "document_id", ref.DocumentID, "tenant_id", tenantID, "err", err)
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
TenantID: &tenantID, DocumentID: strconv.FormatInt(id, 10), Success: false,
Detail: "accounting_pull file key:" + strconv.FormatInt(keyID, 10) + " open_failed",
})
writeError(w, http.StatusInternalServerError, "file unavailable")
return
}
defer f.Close()
s.audlog.Log(audit.Entry{
EventType: audit.EventAccountingPull, Username: "accounting_api", IPAddress: s.remoteIP(r),
TenantID: &tenantID, DocumentID: strconv.FormatInt(id, 10), Success: true,
Detail: "accounting_pull file key:" + strconv.FormatInt(keyID, 10),
})
ext := filepath.Ext(ref.StoragePath())
w.Header().Set("Content-Type", detectMimeType("", ext, ref.StoragePath()))
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(ref.Title, ext)+"\"")
w.Header().Set("X-Content-Type-Options", "nosniff")
if _, err := io.Copy(w, f); err != nil {
s.logger.Warn("accounting file stream interrupted", "document_id", ref.DocumentID, "err", err)
}
}
// parseAccountingDate accepts either a plain date (YYYY-MM-DD, interpreted as
// UTC midnight) or a full RFC3339 timestamp.
func parseAccountingDate(v string) (time.Time, error) {
if t, err := time.Parse("2006-01-02", v); err == nil {
return t, nil
}
t, err := time.Parse(time.RFC3339, v)
if err != nil {
return time.Time{}, err
}
return t, nil
}
// accountingIDRange renders "first-last" document ids of a page for the audit
// Detail, so a later GoBD audit can reconstruct what a pull actually returned.
func accountingIDRange(docs []storage.AccountingDocument) string {
if len(docs) == 0 {
return "-"
}
return strconv.FormatInt(docs[0].ID, 10) + "-" + strconv.FormatInt(docs[len(docs)-1].ID, 10)
}
+280
View File
@@ -0,0 +1,280 @@
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// akteACLUserID returns the ACL user filter for the caller: nil for
// domain_admin/superadmin (see every document in the tenant), a non-nil user ID
// for role 'user' (filtered against document_visibility). Mirrors the logic in
// handleListDocuments.
func akteACLUserID(sess *auth.Session) *int64 {
if auth.HasRole(sess.Role, userstore.RoleDomainAdmin) {
return nil
}
uid := sess.UserID
return &uid
}
func (s *Server) handleListAkten(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
akten, err := s.store.ListAkten(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list akten failed")
return
}
writeJSON(w, http.StatusOK, akten)
}
type createAkteRequest struct {
Titel string `json:"titel"`
Beschreibung string `json:"beschreibung"`
CorrespondentID *int64 `json:"correspondent_id"`
}
func (s *Server) handleCreateAkte(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
ctx := r.Context()
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req createAkteRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
titel := strings.TrimSpace(req.Titel)
if titel == "" {
writeError(w, http.StatusBadRequest, "titel is required")
return
}
// Guard against cross-tenant references: the correspondent must belong to
// the same tenant.
if req.CorrespondentID != nil && !s.taxonomyEntityBelongsToTenant(ctx, "correspondents", *req.CorrespondentID, *sess.TenantID) {
s.audlog.Log(audit.Entry{EventType: audit.EventAkteCreate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "invalid_correspondent_id"})
writeError(w, http.StatusBadRequest, "invalid correspondent_id")
return
}
akte, err := s.store.CreateAkte(ctx, *sess.TenantID, titel, strings.TrimSpace(req.Beschreibung), req.CorrespondentID, sess.UserID)
if err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventAkteCreate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: err.Error()})
writeError(w, http.StatusInternalServerError, "create akte failed")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteCreate, Username: sess.Username, TenantID: sess.TenantID, Success: true, Detail: "akte_id=" + strconv.FormatInt(akte.ID, 10)})
writeJSON(w, http.StatusCreated, akte)
}
// akteDetailResponse is the GET /api/akten/{id} payload: the akte plus its
// ACL-filtered documents.
type akteDetailResponse struct {
*storage.Akte
Documents []storage.Document `json:"documents"`
}
func (s *Server) handleGetAkte(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
ctx := r.Context()
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid akte id")
return
}
akte, err := s.store.GetAkte(ctx, id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusNotFound, "akte not found")
return
}
docs, err := s.store.ListAkteDocuments(ctx, id, *sess.TenantID, akteACLUserID(sess))
if err != nil {
writeError(w, http.StatusInternalServerError, "list akte documents failed")
return
}
writeJSON(w, http.StatusOK, akteDetailResponse{Akte: akte, Documents: docs})
}
type updateAkteRequest struct {
Titel string `json:"titel"`
Beschreibung string `json:"beschreibung"`
CorrespondentID *int64 `json:"correspondent_id"`
}
func (s *Server) handleUpdateAkte(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
ctx := r.Context()
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid akte id")
return
}
var req updateAkteRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
titel := strings.TrimSpace(req.Titel)
if titel == "" {
writeError(w, http.StatusBadRequest, "titel is required")
return
}
if req.CorrespondentID != nil && !s.taxonomyEntityBelongsToTenant(ctx, "correspondents", *req.CorrespondentID, *sess.TenantID) {
s.audlog.Log(audit.Entry{EventType: audit.EventAkteUpdate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "invalid_correspondent_id"})
writeError(w, http.StatusBadRequest, "invalid correspondent_id")
return
}
if err := s.store.UpdateAkte(ctx, id, *sess.TenantID, titel, strings.TrimSpace(req.Beschreibung), req.CorrespondentID); err != nil {
status := http.StatusInternalServerError
msg := "update akte failed"
if errors.Is(err, storage.ErrAkteNotFound) {
status = http.StatusNotFound
msg = "akte not found"
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteUpdate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
akte, err := s.store.GetAkte(ctx, id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusNotFound, "akte not found")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteUpdate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, akte)
}
func (s *Server) handleCloseAkte(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
ctx := r.Context()
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid akte id")
return
}
if err := s.store.CloseAkte(ctx, id, *sess.TenantID); err != nil {
status := http.StatusInternalServerError
msg := "close akte failed"
if errors.Is(err, storage.ErrAkteNotFound) {
status = http.StatusNotFound
msg = "akte not found"
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteClose, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
akte, err := s.store.GetAkte(ctx, id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusNotFound, "akte not found")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteClose, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, akte)
}
func (s *Server) handleDeleteAkte(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
ctx := r.Context()
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid akte id")
return
}
if err := s.store.DeleteAkte(ctx, id, *sess.TenantID); err != nil {
status := http.StatusInternalServerError
msg := "delete akte failed"
if errors.Is(err, storage.ErrAkteNotFound) {
status = http.StatusNotFound
msg = "akte not found"
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteDelete, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventAkteDelete, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// setDocumentAkteRequest uses a pointer so an explicit JSON null removes the
// assignment while an omitted field is rejected (see handleSetDocumentAkte).
type setDocumentAkteRequest struct {
AkteID *int64 `json:"akte_id"`
}
// handleSetDocumentAkte assigns (or clears) a document's akte membership
// (PUT /api/documents/{id}/akte, body {"akte_id": number|null}). A null value
// removes the assignment. The akte is not part of the document ACL, so
// SetDocumentAkte only re-syncs the search index. Audited as
// EventAkteDocumentAdd (assign) or EventAkteDocumentRemove (clear).
func (s *Server) handleSetDocumentAkte(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
ctx := r.Context()
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
var req setDocumentAkteRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
event := audit.EventAkteDocumentRemove
if req.AkteID != nil {
event = audit.EventAkteDocumentAdd
}
// Ownership check: the document must belong to the caller's tenant.
if _, err := s.store.GetDocument(ctx, id, *sess.TenantID); err != nil {
s.audlog.Log(audit.Entry{EventType: event, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "document_not_found"})
writeError(w, http.StatusNotFound, "document not found")
return
}
// Guard against cross-tenant references: the akte must belong to the same
// tenant (nil means "remove", which needs no lookup).
if req.AkteID != nil {
if _, err := s.store.GetAkte(ctx, *req.AkteID, *sess.TenantID); err != nil {
s.audlog.Log(audit.Entry{EventType: event, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "invalid_akte_id"})
writeError(w, http.StatusBadRequest, "invalid akte_id")
return
}
}
if err := s.store.SetDocumentAkte(ctx, id, *sess.TenantID, req.AkteID); err != nil {
status := http.StatusInternalServerError
msg := "set akte failed"
if errors.Is(err, storage.ErrDocumentNotFound) {
status = http.StatusNotFound
msg = "document not found"
}
s.audlog.Log(audit.Entry{EventType: event, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
doc, err := s.store.GetDocument(ctx, id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
detail := "akte_id=cleared"
if req.AkteID != nil {
detail = "akte_id=" + strconv.FormatInt(*req.AkteID, 10)
}
s.audlog.Log(audit.Entry{EventType: event, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true, Detail: detail})
writeJSON(w, http.StatusOK, doc)
}
+51
View File
@@ -0,0 +1,51 @@
package api
import (
"net/http"
"strconv"
"archivdms/internal/audit"
)
func (s *Server) handleAuditLog(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
entries, total, err := s.audlog.Query(audit.QueryFilter{
TenantID: sess.TenantID,
})
if err != nil {
writeError(w, http.StatusInternalServerError, "audit query failed")
return
}
writeJSON(w, http.StatusOK, map[string]any{"entries": entries, "total": total})
}
// handleDocumentAuditLog returns the audit trail scoped to a single document
// (GET /api/documents/{id}/audit). Unlike handleAuditLog (domain_admin+, full
// tenant log) this is available to every authenticated user, but only after an
// ownership/ACL check: GetDocument filters WHERE tenant_id (and the document
// ACL), so a caller who may not see the document gets a 404 and never its
// history. The document_id filter uses the exact same string format
// (strconv.FormatInt(id, 10)) that document_handlers.go writes into the audit
// entries, otherwise the filter would match nothing.
func (s *Server) handleDocumentAuditLog(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
// ACL/tenant check: only callers who may see the document may see its history.
if _, err := s.store.GetDocument(r.Context(), id, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
entries, total, err := s.audlog.Query(audit.QueryFilter{
TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(id, 10),
})
if err != nil {
writeError(w, http.StatusInternalServerError, "audit query failed")
return
}
writeJSON(w, http.StatusOK, map[string]any{"entries": entries, "total": total})
}
+99
View File
@@ -0,0 +1,99 @@
package api
import (
"encoding/json"
"net/http"
"archivdms/internal/audit"
)
type loginRequest struct {
Username string `json:"username"`
Password string `json:"password"`
}
func (s *Server) handleLogin(w http.ResponseWriter, r *http.Request) {
var req loginRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
ip := s.remoteIP(r)
token, user, err := s.authMgr.LoginFrom(r.Context(), req.Username, req.Password, ip)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventLogin,
Username: req.Username,
IPAddress: ip,
Success: false,
Detail: "invalid_credentials",
})
writeError(w, http.StatusUnauthorized, "invalid credentials")
return
}
http.SetCookie(w, &http.Cookie{
Name: sessionCookieName,
Value: token,
Path: "/",
HttpOnly: true,
Secure: s.cfg.SecureCookies,
SameSite: http.SameSiteLaxMode,
MaxAge: 8 * 60 * 60,
})
_ = s.users.UpdateLastLogin(user.ID)
s.audlog.Log(audit.Entry{
EventType: audit.EventLogin,
Username: user.Username,
IPAddress: ip,
TenantID: user.TenantID,
Success: true,
})
writeJSON(w, http.StatusOK, map[string]any{"token": token, "user": user})
}
func (s *Server) handleMe(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
user, err := s.users.GetByID(sess.UserID)
if err != nil {
writeError(w, http.StatusNotFound, "user not found")
return
}
writeJSON(w, http.StatusOK, user)
}
func (s *Server) handleLogout(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
token := ""
if c, err := r.Cookie(sessionCookieName); err == nil {
token = c.Value
}
if token == "" {
token = extractBearerToken(r)
}
if token != "" {
_ = s.authMgr.Logout(token)
}
http.SetCookie(w, &http.Cookie{
Name: sessionCookieName,
Value: "",
Path: "/",
HttpOnly: true,
MaxAge: -1,
})
s.audlog.Log(audit.Entry{
EventType: audit.EventLogout,
Username: sess.Username,
IPAddress: s.remoteIP(r),
TenantID: sess.TenantID,
Success: true,
})
writeJSON(w, http.StatusOK, map[string]string{"status": "logged out"})
}
@@ -0,0 +1,404 @@
// Classification-template ("Klassifizierungsvorlagen") HTTP handlers (see
// internal/storage/classification_templates.go +
// classification_templates_apply.go):
//
// GET/POST /api/classification-templates GET/PUT/DELETE /api/classification-templates/{id}
// PUT /api/classification-templates/{id}/tags
// PUT /api/classification-templates/{id}/field-defaults
// POST /api/documents/{id}/apply-template
//
// Template administration (CRUD + tag / field-default bulk replace) requires
// domain_admin (s.authAdmin). Applying a template to a document is a normal
// working action and only requires an authenticated tenant context (s.auth).
// Ownership is enforced in the store layer (id+tenant_id). Every mutation is
// audit-logged, including failures.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
type templateRequest struct {
Name string `json:"name"`
Description string `json:"description"`
DocTypeID *int64 `json:"doc_type_id"`
RetainYears *int `json:"retain_years"`
Active *bool `json:"active"`
TitleTemplate *string `json:"title_template"`
}
// handleListTemplates handles GET /api/classification-templates (optional
// ?doc_type_id= filter).
func (s *Server) handleListTemplates(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var docTypeID *int64
if raw := r.URL.Query().Get("doc_type_id"); raw != "" {
id, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid doc_type_id")
return
}
docTypeID = &id
}
tmpls, err := s.store.ListTemplates(r.Context(), *sess.TenantID, docTypeID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list classification templates failed")
return
}
writeJSON(w, http.StatusOK, tmpls)
}
// handleGetTemplate handles GET /api/classification-templates/{id} (resolved).
func (s *Server) handleGetTemplate(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
tmpl, err := s.store.GetTemplate(r.Context(), id, *sess.TenantID)
if err != nil {
if errors.Is(err, storage.ErrClassificationTemplateNotFound) {
writeError(w, http.StatusNotFound, "classification template not found")
return
}
writeError(w, http.StatusInternalServerError, "get classification template failed")
return
}
writeJSON(w, http.StatusOK, tmpl)
}
// handleCreateTemplate handles POST /api/classification-templates (domain_admin+).
func (s *Server) handleCreateTemplate(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req templateRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
active := true
if req.Active != nil {
active = *req.Active
}
if req.TitleTemplate != nil {
if err := storage.ValidateTitleTemplate(*req.TitleTemplate); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
}
tmpl, err := s.store.CreateTemplate(r.Context(), *sess.TenantID, storage.CreateTemplateRequest{
Name: req.Name, Description: req.Description, DocTypeID: req.DocTypeID,
RetainYears: req.RetainYears, Active: active, CreatedBy: &sess.UserID,
TitleTemplate: req.TitleTemplate,
})
if err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrDuplicateTemplateName) {
status = http.StatusConflict
}
s.audlog.Log(audit.Entry{EventType: audit.EventTemplateCreate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "template_create err:" + err.Error()})
writeError(w, status, "create classification template failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateCreate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "template_create id:" + strconv.FormatInt(tmpl.ID, 10) + " name:" + tmpl.Name,
})
writeJSON(w, http.StatusCreated, tmpl)
}
// handleUpdateTemplate handles PUT /api/classification-templates/{id} (domain_admin+).
func (s *Server) handleUpdateTemplate(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req templateRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
active := true
if req.Active != nil {
active = *req.Active
}
if req.TitleTemplate != nil {
if err := storage.ValidateTitleTemplate(*req.TitleTemplate); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
}
err = s.store.UpdateTemplate(r.Context(), id, *sess.TenantID, storage.UpdateTemplateRequest{
Name: req.Name, Description: req.Description, DocTypeID: req.DocTypeID,
RetainYears: req.RetainYears, Active: active,
TitleTemplate: req.TitleTemplate,
})
if err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrClassificationTemplateNotFound) {
status = http.StatusNotFound
} else if errors.Is(err, storage.ErrDuplicateTemplateName) {
status = http.StatusConflict
}
s.audlog.Log(audit.Entry{EventType: audit.EventTemplateUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "template_update id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, "update classification template failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "template_update id:" + strconv.FormatInt(id, 10),
})
tmpl, err := s.store.GetTemplate(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "reload classification template failed")
return
}
writeJSON(w, http.StatusOK, tmpl)
}
// handleDeleteTemplate handles DELETE /api/classification-templates/{id} (domain_admin+).
func (s *Server) handleDeleteTemplate(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.DeleteTemplate(r.Context(), id, *sess.TenantID); err != nil {
status := http.StatusNotFound
if !errors.Is(err, storage.ErrClassificationTemplateNotFound) {
status = http.StatusInternalServerError
}
s.audlog.Log(audit.Entry{EventType: audit.EventTemplateDelete, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "template_delete id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, "delete classification template failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateDelete, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "template_delete id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
type templateTagsRequest struct {
TagIDs []int64 `json:"tag_ids"`
}
// handleSetTemplateTags handles PUT /api/classification-templates/{id}/tags (domain_admin+).
func (s *Server) handleSetTemplateTags(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req templateTagsRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if err := s.store.SetTemplateTags(r.Context(), id, *sess.TenantID, req.TagIDs); err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrClassificationTemplateNotFound) {
status = http.StatusNotFound
} else if errors.Is(err, storage.ErrTaxonomyNotFound) {
status = http.StatusBadRequest
}
s.audlog.Log(audit.Entry{EventType: audit.EventTemplateUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "template_tags_set id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, "set template tags failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "template_tags_set id:" + strconv.FormatInt(id, 10) + " count:" + strconv.Itoa(len(req.TagIDs)),
})
tmpl, err := s.store.GetTemplate(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "reload classification template failed")
return
}
writeJSON(w, http.StatusOK, tmpl)
}
type templateFieldDefaultRequest struct {
FieldID int64 `json:"field_id"`
ValueText *string `json:"value_text"`
ValueNumber *float64 `json:"value_number"`
ValueDate *string `json:"value_date"`
ValueBool *bool `json:"value_bool"`
Overwrite bool `json:"overwrite"`
}
// handleSetTemplateFieldDefaults handles PUT
// /api/classification-templates/{id}/field-defaults (bulk replace, domain_admin+).
func (s *Server) handleSetTemplateFieldDefaults(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var reqs []templateFieldDefaultRequest
if err := json.NewDecoder(r.Body).Decode(&reqs); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body (expected array)")
return
}
defaults := make([]storage.TemplateFieldDefaultInput, 0, len(reqs))
for _, d := range reqs {
defaults = append(defaults, storage.TemplateFieldDefaultInput{
FieldID: d.FieldID, ValueText: d.ValueText, ValueNumber: d.ValueNumber,
ValueDate: d.ValueDate, ValueBool: d.ValueBool, Overwrite: d.Overwrite,
})
}
if err := s.store.SetTemplateFieldDefaults(r.Context(), id, *sess.TenantID, defaults); err != nil {
status := http.StatusInternalServerError
msg := "set template field defaults failed"
if errors.Is(err, storage.ErrClassificationTemplateNotFound) {
status = http.StatusNotFound
} else if errors.Is(err, storage.ErrCustomFieldNotFound) {
status = http.StatusBadRequest
msg = "unknown custom field"
} else if strings.Contains(err.Error(), "invalid date") || strings.Contains(err.Error(), "not in enum options") {
status = http.StatusBadRequest
msg = err.Error()
}
s.audlog.Log(audit.Entry{EventType: audit.EventTemplateUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "template_field_defaults_set id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "template_field_defaults_set id:" + strconv.FormatInt(id, 10) + " count:" + strconv.Itoa(len(defaults)),
})
tmpl, err := s.store.GetTemplate(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "reload classification template failed")
return
}
writeJSON(w, http.StatusOK, tmpl)
}
type applyTemplateRequest struct {
TemplateID int64 `json:"template_id"`
DryRun bool `json:"dry_run"`
Overwrite bool `json:"overwrite"`
}
// handleApplyTemplate handles POST /api/documents/{id}/apply-template. Any
// authenticated tenant user may apply a template (normal working action). With
// dry_run=true it only previews (no writes). A rejected retain_until shortening
// (RetainUntilBlocked) is still audit-logged for GoBD traceability.
func (s *Server) handleApplyTemplate(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
var req applyTemplateRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.TemplateID == 0 {
writeError(w, http.StatusBadRequest, "template_id is required")
return
}
docRef := strconv.FormatInt(docID, 10)
if req.DryRun {
res, err := s.store.PreviewApplyTemplate(r.Context(), docID, req.TemplateID, *sess.TenantID)
if err != nil {
s.writeTemplateApplyError(w, err)
return
}
writeJSON(w, http.StatusOK, res)
return
}
res, err := s.store.ApplyTemplate(r.Context(), docID, req.TemplateID, *sess.TenantID, req.Overwrite)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateApplied, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: docRef, Success: false, Detail: "template_apply template:" + strconv.FormatInt(req.TemplateID, 10) + " err:" + err.Error(),
})
s.writeTemplateApplyError(w, err)
return
}
detail := "template_apply template:" + strconv.FormatInt(req.TemplateID, 10) +
" tags_added:" + strconv.Itoa(len(res.TagsToAdd)) +
" fields_set:" + strconv.Itoa(len(res.FieldsToSet)) +
" fields_overwritten:" + strconv.Itoa(len(res.FieldsOverwritten))
if res.RetainUntilBlocked {
detail += " retain_until_shortening_rejected"
}
s.audlog.Log(audit.Entry{
EventType: audit.EventTemplateApplied, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: docRef, Success: true, Detail: detail,
})
writeJSON(w, http.StatusOK, res)
}
// writeTemplateApplyError maps store errors from the apply/preview path to HTTP
// status codes.
func (s *Server) writeTemplateApplyError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, storage.ErrDocumentNotFound):
writeError(w, http.StatusNotFound, "document not found")
case errors.Is(err, storage.ErrClassificationTemplateNotFound):
writeError(w, http.StatusNotFound, "classification template not found")
case errors.Is(err, storage.ErrRequiredFieldMissing):
writeError(w, http.StatusBadRequest, err.Error())
default:
writeError(w, http.StatusInternalServerError, "apply template failed")
}
}
+535
View File
@@ -0,0 +1,535 @@
// GoBD-Verfahrensdokumentation: Entwurfs-Generator.
//
// GET /api/compliance/procedure-documentation[?tenant_id=N]
//
// Erzeugt live aus dem aktuellen DB-Stand einen Markdown-Baustein einer
// GoBD-Verfahrensdokumentation für GENAU EINEN Mandanten (kein Caching, kein
// Vermischen mehrerer Mandanten). Konzept/Gliederung siehe
// .claude/agent-memory/retention-compliance/project_gobd_verfahrensdokumentation.md
//
// Auth (Muster wie retention_rule_handlers.go): domain_admin+ (s.authAdmin) für
// den EIGENEN Mandanten. Ein superadmin darf zusätzlich per ?tenant_id=N einen
// fremden Mandanten exportieren; für alle anderen Rollen ist ein abweichender
// tenant_id-Parameter ein 403. Sämtliche Queries sind strikt auf die eine
// aufgelöste tenant_id gefiltert (applikationsseitige Mandantentrennung, kein
// Postgres-RLS).
//
// Das Ergebnis ist ausdrücklich ein ENTWURF und kein rechtsverbindliches
// Fertigdokument — der Hinweis steht als erste Zeile im Dokument.
package api
import (
"context"
"fmt"
"net/http"
"sort"
"strconv"
"strings"
"time"
"archivdms/internal/audit"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// handleProcedureDocumentation handles GET /api/compliance/procedure-documentation.
func (s *Server) handleProcedureDocumentation(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil && sess.Role != userstore.RoleSuperAdmin {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
// Ziel-Mandant auflösen: Default ist der eigene Mandant. Ein expliziter
// ?tenant_id= ist nur für superadmin erlaubt (Cross-Tenant), für alle
// anderen nur, wenn er dem eigenen Mandanten entspricht.
tenantID := int64(0)
if sess.TenantID != nil {
tenantID = *sess.TenantID
}
if raw := strings.TrimSpace(r.URL.Query().Get("tenant_id")); raw != "" {
requested, err := strconv.ParseInt(raw, 10, 64)
if err != nil || requested <= 0 {
writeError(w, http.StatusBadRequest, "invalid tenant_id")
return
}
if sess.Role != userstore.RoleSuperAdmin && requested != tenantID {
s.audlog.Log(audit.Entry{
EventType: audit.EventComplianceExport, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "procedure_doc cross-tenant denied requested:" + raw,
})
writeError(w, http.StatusForbidden, "cross-tenant export requires superadmin")
return
}
tenantID = requested
}
if tenantID <= 0 {
writeError(w, http.StatusBadRequest, "tenant_id required")
return
}
tenantName := "Mandant " + strconv.FormatInt(tenantID, 10)
tenantSlug := strconv.FormatInt(tenantID, 10)
if s.tenantStore != nil {
t, err := s.tenantStore.GetByID(r.Context(), tenantID)
if err != nil || t == nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventComplianceExport, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "procedure_doc tenant lookup failed tenant:" + strconv.FormatInt(tenantID, 10),
})
writeError(w, http.StatusNotFound, "tenant not found")
return
}
tenantName = t.Name
if t.Slug != "" {
tenantSlug = t.Slug
}
}
md, err := s.buildProcedureDocumentation(r.Context(), tenantID, tenantName)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventComplianceExport, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "procedure_doc tenant:" + strconv.FormatInt(tenantID, 10) + " err:" + err.Error(),
})
writeError(w, http.StatusInternalServerError, "generate procedure documentation failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventComplianceExport, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "procedure_doc generated tenant:" + strconv.FormatInt(tenantID, 10),
})
filename := fmt.Sprintf("verfahrensdokumentation-entwurf-%s-%s.md",
safeFilenamePart(tenantSlug), time.Now().Format("2006-01-02"))
w.Header().Set("Content-Type", "text/markdown; charset=utf-8")
w.Header().Set("Content-Disposition", `attachment; filename="`+filename+`"`)
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(md))
}
// safeFilenamePart reduces a tenant slug to [a-z0-9-] so it can never break out
// of the Content-Disposition filename.
func safeFilenamePart(in string) string {
var b strings.Builder
for _, r := range strings.ToLower(in) {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9':
b.WriteRune(r)
case r == '-' || r == '_':
b.WriteRune('-')
}
}
out := b.String()
if out == "" {
return "mandant"
}
if len(out) > 60 {
out = out[:60]
}
return out
}
// buildProcedureDocumentation assembles the Markdown draft for one tenant.
// Every store call below is tenant-scoped; nothing is queried tenant-wide.
func (s *Server) buildProcedureDocumentation(ctx context.Context, tenantID int64, tenantName string) (string, error) {
now := time.Now()
rules, err := s.store.ListRetentionRules(ctx, tenantID)
if err != nil {
return "", fmt.Errorf("list retention rules: %w", err)
}
docTypes, err := s.store.ListTaxonomyEntities(ctx, "document_types", tenantID)
if err != nil {
return "", fmt.Errorf("list document types: %w", err)
}
tags, err := s.store.ListTaxonomyEntities(ctx, "tags", tenantID)
if err != nil {
return "", fmt.Errorf("list tags: %w", err)
}
groups, err := s.store.ListPermissionGroups(ctx, tenantID)
if err != nil {
return "", fmt.Errorf("list permission groups: %w", err)
}
workflows, err := s.store.ListWorkflows(ctx, tenantID)
if err != nil {
return "", fmt.Errorf("list workflows: %w", err)
}
templates, err := s.store.ListTemplates(ctx, tenantID, nil)
if err != nil {
return "", fmt.Errorf("list classification templates: %w", err)
}
stats, err := s.store.ComplianceStatsForTenant(ctx, tenantID)
if err != nil {
return "", fmt.Errorf("compliance stats: %w", err)
}
docTypeName := map[int64]string{}
for _, dt := range docTypes {
docTypeName[dt.ID] = dt.Name
}
var b strings.Builder
// --- Kopf / Entwurfskennzeichnung -----------------------------------
b.WriteString("> **ENTWURF** — automatisch generierter Baustein einer GoBD-Verfahrensdokumentation, Stand " +
now.Format("02.01.2006 15:04:05 MST") + ".\n" +
"> Ersetzt keine rechtliche Prüfung, muss um Organisationsbeschreibung/Verantwortlichkeiten/Backup-Notfallkonzept " +
"ergänzt und von fachkundiger Stelle geprüft werden.\n\n")
b.WriteString("# Verfahrensdokumentation (Entwurf) — " + tenantName + "\n\n")
b.WriteString("| | |\n|---|---|\n")
b.WriteString("| Mandant | " + mdCell(tenantName) + " |\n")
b.WriteString("| Mandanten-ID | " + strconv.FormatInt(tenantID, 10) + " |\n")
b.WriteString("| Stand der Generierung | " + now.Format("02.01.2006 15:04:05 MST") + " |\n")
b.WriteString("| System | archivdms (Dokumentenmanagementsystem) |\n")
b.WriteString("| Aktive Dokumente | " + strconv.FormatInt(stats.Documents, 10) + " |\n")
b.WriteString("| Davon mit Aufbewahrungsfrist (retain_until) | " + strconv.FormatInt(stats.DocumentsWithRetain, 10) + " |\n")
b.WriteString("| Dokumente im Papierkorb | " + strconv.FormatInt(stats.DocumentsInTrash, 10) + " |\n\n")
b.WriteString("Dieses Dokument beschreibt ausschließlich die im System hinterlegte Konfiguration des oben " +
"genannten Mandanten. Daten anderer Mandanten sind nicht enthalten (mandantengetrennte Auswertung).\n\n")
// --- 1. Aufbewahrungsfristen ----------------------------------------
b.WriteString("## 1. Aufbewahrungsfristen\n\n")
b.WriteString("Aufbewahrungsfristen werden als Regeln je Dokumenttyp gepflegt. Eine Regel ohne Dokumenttyp " +
"gilt als Mandanten-Default mit niedrigster Präzedenz; eine dokumenttyp-spezifische Regel hat Vorrang. " +
"Aus Fristbeginn (Trigger) und Frist berechnet das System je Dokument ein Datum `retain_until`; " +
"bis zu diesem Datum ist eine endgültige Löschung technisch blockiert.\n\n")
b.WriteString("Fristbeginn (Trigger-Typen): `document_date` = Belegdatum (ersatzweise Uploaddatum), " +
"`upload_date` = Uploaddatum, `fixed_date` = fixes Stichtagsdatum, " +
"`event` = ereignisgesteuert (wird nicht automatisch berechnet, erfordert manuelle Fristsetzung).\n\n")
if len(rules) == 0 {
b.WriteString("**Es sind derzeit keine Aufbewahrungsregeln konfiguriert.** [MANUELL ZU ERGÄNZEN: " +
"gesetzliche Fristen (z. B. § 147 AO, § 257 HGB) je Dokumentart benennen und im System hinterlegen.]\n\n")
} else {
b.WriteString("| Regel | Geltungsbereich | Fristbeginn | Frist | Rechtsgrundlage | Löschfreigabe nötig | DSGVO-Konflikt | Aktiv |\n")
b.WriteString("|---|---|---|---|---|---|---|---|\n")
for _, ru := range rules {
scope := "Mandanten-Default (alle Dokumenttypen ohne eigene Regel)"
if ru.DocTypeID != nil {
if n, ok := docTypeName[*ru.DocTypeID]; ok {
scope = "Dokumenttyp: " + n
} else {
scope = "Dokumenttyp-ID " + strconv.FormatInt(*ru.DocTypeID, 10)
}
}
trigger := ru.TriggerType
if ru.TriggerReference != "" {
trigger += " (" + ru.TriggerReference + ")"
}
b.WriteString("| " + mdCell(ru.Name) + " | " + mdCell(scope) + " | " + mdCell(trigger) + " | " +
mdCell(retentionPeriodText(ru)) + " | " + mdCell(orDash(ru.LegalBasis)) + " | " +
jaNein(ru.RequiresApprovalForDestroy) + " | " + jaNein(ru.DSGVOConflict) + " | " +
jaNein(ru.Active) + " |\n")
}
b.WriteString("\n")
}
b.WriteString("[MANUELL ZU ERGÄNZEN: Prüfung, ob die hinterlegten Fristen den für dieses Unternehmen " +
"einschlägigen handels- und steuerrechtlichen Vorgaben entsprechen.]\n\n")
// --- 2. Zugriffsschutz ----------------------------------------------
b.WriteString("## 2. Zugriffsschutz und Berechtigungen\n\n")
b.WriteString("### 2.1 Rollenmodell\n\n")
b.WriteString("- `superadmin` — mandantenübergreifende Systemverwaltung (Anlage von Mandanten).\n")
b.WriteString("- `domain_admin` — Administration innerhalb des eigenen Mandanten: Benutzer, Berechtigungsgruppen, " +
"Aufbewahrungsregeln, Workflows, Klassifizierungsvorlagen, Bestätigung endgültiger Löschungen.\n")
b.WriteString("- `user` — Erfassen, Suchen und Bearbeiten von Dokumenten im Rahmen der erteilten Berechtigungen.\n\n")
b.WriteString("Die Anmeldung erfolgt passwortbasiert (bcrypt-Hash, Kostenfaktor 12) bzw. optional gegen ein " +
"Verzeichnis (LDAP); die Sitzung wird über ein signiertes, nicht per JavaScript auslesbares Sitzungs-Cookie " +
"geführt. Mandantentrennung erfolgt applikationsseitig: jede Datenbankabfrage ist auf den Mandanten des " +
"angemeldeten Benutzers eingeschränkt.\n\n")
if s.users != nil {
users, err := s.users.ListByTenant(ctx, tenantID)
if err != nil {
return "", fmt.Errorf("list users: %w", err)
}
roleCount := map[string]int{}
for _, u := range users {
if !u.Active {
roleCount["(inaktiv)"]++
continue
}
roleCount[u.Role]++
}
keys := make([]string, 0, len(roleCount))
for k := range roleCount {
keys = append(keys, k)
}
sort.Strings(keys)
b.WriteString("Benutzerbestand dieses Mandanten: ")
if len(keys) == 0 {
b.WriteString("keine Benutzer erfasst.\n\n")
} else {
parts := make([]string, 0, len(keys))
for _, k := range keys {
parts = append(parts, fmt.Sprintf("%s: %d", k, roleCount[k]))
}
b.WriteString(strings.Join(parts, ", ") + ".\n\n")
}
}
b.WriteString("### 2.2 Berechtigungsgruppen\n\n")
if len(groups) == 0 {
b.WriteString("Es sind keine Berechtigungsgruppen angelegt; der Zugriff wird ausschließlich über das " +
"Rollenmodell gesteuert. [MANUELL ZU ERGÄNZEN: Begründung, falls keine feinere Zugriffssteuerung " +
"erforderlich ist.]\n\n")
} else {
b.WriteString("| Gruppe | Mitglieder |\n|---|---|\n")
for _, g := range groups {
members, err := s.store.ListGroupMembersDetailed(ctx, g.ID, tenantID)
if err != nil {
return "", fmt.Errorf("list group members: %w", err)
}
names := make([]string, 0, len(members))
for _, m := range members {
names = append(names, m.Username)
}
if len(names) == 0 {
names = append(names, "—")
}
b.WriteString("| " + mdCell(g.Name) + " | " + mdCell(strings.Join(names, ", ")) + " |\n")
}
b.WriteString("\n")
}
b.WriteString("### 2.3 Berechtigungsebenen\n\n")
b.WriteString("Der Zugriff auf ein einzelnes Dokument ergibt sich aus drei Ebenen; die speziellere Ebene " +
"überschreibt die allgemeinere:\n\n")
b.WriteString("1. **Dokumenttyp-Berechtigung** (Grundeinstellung je Dokumenttyp)\n")
b.WriteString("2. **Schlagwort-Berechtigung** (Grants über die Tags eines Dokuments)\n")
b.WriteString("3. **Einzeldokument-Berechtigung** (Übersteuerung für ein konkretes Dokument, inkl. Entzug/`deny`)\n\n")
b.WriteString(fmt.Sprintf("Aktuell vergeben: %d Dokumenttyp-Berechtigungen, %d Schlagwort-Berechtigungen, "+
"%d Einzeldokument-Berechtigungen bei %d Berechtigungsgruppen.\n\n",
stats.DocTypeGrants, stats.TagGrants, stats.DocumentGrants, stats.PermissionGroups))
// Dokumenttyp-Grants im Detail.
dtRows := 0
var dtBuf strings.Builder
for _, dt := range docTypes {
grants, err := s.store.ListDocumentTypeGrants(ctx, tenantID, dt.ID)
if err != nil {
return "", fmt.Errorf("list document type grants: %w", err)
}
for _, g := range grants {
dtBuf.WriteString("| " + mdCell(dt.Name) + " | " + mdCell(g.GroupName) + " | " + mdCell(g.Access) + " |\n")
dtRows++
}
}
if dtRows > 0 {
b.WriteString("**Dokumenttyp-Berechtigungen**\n\n| Dokumenttyp | Gruppe | Zugriff |\n|---|---|---|\n")
b.WriteString(dtBuf.String() + "\n")
}
tagRows := 0
var tagBuf strings.Builder
for _, tg := range tags {
grants, err := s.store.ListTagGrants(ctx, tenantID, tg.ID)
if err != nil {
return "", fmt.Errorf("list tag grants: %w", err)
}
for _, g := range grants {
tagBuf.WriteString("| " + mdCell(tg.Name) + " | " + mdCell(g.GroupName) + " | " + mdCell(g.Access) + " |\n")
tagRows++
}
}
if tagRows > 0 {
b.WriteString("**Schlagwort-Berechtigungen**\n\n| Schlagwort | Gruppe | Zugriff |\n|---|---|---|\n")
b.WriteString(tagBuf.String() + "\n")
}
if stats.DocumentGrants > 0 {
b.WriteString(fmt.Sprintf("Zusätzlich bestehen %d dokumentbezogene Einzelberechtigungen. "+
"Jede Änderung daran ist im Änderungsprotokoll nachvollziehbar.\n\n", stats.DocumentGrants))
}
// --- 3. Löschkonzept -------------------------------------------------
b.WriteString("## 3. Löschkonzept und Unveränderbarkeit\n\n")
b.WriteString("### 3.1 Unveränderbarkeit der Ablage (WORM)\n\n")
b.WriteString("Archivierte Dokumente werden im Dateisystem nach dem WORM-Prinzip abgelegt " +
"(*write once, read many*): die Datei wird einmalig geschrieben und anschließend schreibgeschützt gesetzt " +
"(Dateirechte 0440, nur lesend). Der Ablagepfad wird nach Mandant, Jahr und Monat gegliedert; der Dateiname " +
"ist der SHA-256-Hash des Inhalts. Dieser Prüfwert wird zusätzlich in der Datenbank geführt und dient als " +
"fälschungssensibler Fingerabdruck: eine nachträgliche inhaltliche Veränderung würde den Hash verändern und " +
"wäre damit erkennbar. Ein erneuter Upload desselben Inhalts wird über den Hash als Dublette erkannt.\n\n")
b.WriteString("### 3.2 Zweistufiges Löschverfahren\n\n")
b.WriteString("Ein Dokument kann nicht unmittelbar aus dem Archiv entfernt werden. Der Ablauf ist zweistufig " +
"und folgt dem Vier-Augen-Prinzip:\n\n")
b.WriteString("1. **Papierkorb (Soft-Delete):** Das Dokument wird als gelöscht markiert (`deleted_at`, " +
"`deleted_by`), bleibt aber gespeichert und wiederherstellbar.\n")
b.WriteString("2. **Löschantrag:** Ein Benutzer beantragt die endgültige Löschung (Status `pending`).\n")
b.WriteString("3. **Bestätigung durch eine zweite Person:** Ein Administrator (`domain_admin`) bestätigt den " +
"Antrag. Eine Bestätigung durch dieselbe Person, die den Antrag gestellt hat, wird technisch " +
"zurückgewiesen (Vier-Augen-Prinzip).\n")
b.WriteString("4. **Fristprüfung:** Besteht noch eine laufende Aufbewahrungsfrist (`retain_until` in der " +
"Zukunft), wird die Löschung blockiert (Status `blocked_retention`).\n")
b.WriteString("5. **Ausführung:** Erst danach wird die WORM-Datei entfernt (Status `executed`). Der " +
"Löschvorgang wird protokolliert (Antragsteller, Bestätigender, Zeitpunkte, Titel, Inhalts-Hash, " +
"Fristzustand) — es verbleibt ein Nachweis über die erfolgte Löschung.\n\n")
b.WriteString("Ein Antrag kann bis zur Bestätigung zurückgezogen werden (Status `cancelled`).\n\n")
if len(stats.DeleteRequestsByStat) > 0 {
statuses := make([]string, 0, len(stats.DeleteRequestsByStat))
for k := range stats.DeleteRequestsByStat {
statuses = append(statuses, k)
}
sort.Strings(statuses)
b.WriteString("Bisherige Löschanträge dieses Mandanten:\n\n| Status | Anzahl |\n|---|---|\n")
for _, st := range statuses {
b.WriteString("| " + mdCell(st) + " | " + strconv.FormatInt(stats.DeleteRequestsByStat[st], 10) + " |\n")
}
b.WriteString("\n")
} else {
b.WriteString("Für diesen Mandanten wurden bislang keine endgültigen Löschungen beantragt.\n\n")
}
// --- 4. Erfassungsautomatisierung ------------------------------------
b.WriteString("## 4. Erfassung und automatisierte Verarbeitung\n\n")
b.WriteString("Dokumente gelangen über den Weg des Uploads über die Weboberfläche oder über eine " +
"mandantenbezogene SFTP-Ablage (Posteingangsverzeichnis) in das System. Nach der Übernahme wird der " +
"Dokumenteninhalt maschinell ausgelesen (Texterkennung/OCR), ein Prüfwert gebildet und das Dokument in " +
"die revisionssichere Ablage überführt. Die Verarbeitung erfolgt über eine Warteschlange; der " +
"Verarbeitungsstatus je Dokument ist im System einsehbar.\n\n")
b.WriteString("### 4.1 Regeln zur automatischen Zuordnung (Workflows)\n\n")
if len(workflows) == 0 {
b.WriteString("Es sind keine Workflow-Regeln konfiguriert; die Verschlagwortung erfolgt manuell " +
"bzw. über die Mustererkennung der Stammdaten.\n\n")
} else {
b.WriteString("| Regel | Auslöser | Aktiv | Priorität |\n|---|---|---|---|\n")
for _, wf := range workflows {
b.WriteString("| " + mdCell(wf.Name) + " | " + mdCell(wf.TriggerType) + " | " +
jaNein(wf.Enabled) + " | " + strconv.Itoa(wf.Priority) + " |\n")
}
b.WriteString("\nJede Ausführung einer Workflow-Regel wird protokolliert (Regel, Dokument, Treffer, " +
"angewandte Aktionen), sodass die maschinelle Zuordnung nachvollziehbar bleibt.\n\n")
}
b.WriteString("### 4.2 Klassifizierungsvorlagen\n\n")
if len(templates) == 0 {
b.WriteString("Es sind keine Klassifizierungsvorlagen hinterlegt.\n\n")
} else {
b.WriteString("| Vorlage | Dokumenttyp | Aufbewahrung (Jahre) | Aktiv | Beschreibung |\n|---|---|---|---|---|\n")
for _, t := range templates {
dt := "—"
if t.DocTypeID != nil {
if n, ok := docTypeName[*t.DocTypeID]; ok {
dt = n
} else {
dt = "Dokumenttyp-ID " + strconv.FormatInt(*t.DocTypeID, 10)
}
}
ry := "—"
if t.RetainYears != nil {
ry = strconv.Itoa(*t.RetainYears)
}
b.WriteString("| " + mdCell(t.Name) + " | " + mdCell(dt) + " | " + ry + " | " +
jaNein(t.Active) + " | " + mdCell(orDash(t.Description)) + " |\n")
}
b.WriteString("\nDie Anwendung einer Vorlage auf ein Dokument wird protokolliert. Eine spätere Änderung " +
"einer Vorlage wirkt nicht rückwirkend auf bereits klassifizierte Dokumente.\n\n")
}
b.WriteString("### 4.3 Stammdaten der Indizierung\n\n")
b.WriteString(fmt.Sprintf("Für diesen Mandanten sind %d Dokumenttypen und %d Schlagworte gepflegt. "+
"Dokumenttypen und Schlagworte können mit Erkennungsmustern versehen werden, über die eine Zuordnung "+
"beim Einlesen automatisch vorgeschlagen bzw. gesetzt wird.\n\n", len(docTypes), len(tags)))
if len(docTypes) > 0 {
names := make([]string, 0, len(docTypes))
for _, dt := range docTypes {
names = append(names, dt.Name)
}
b.WriteString("Dokumenttypen: " + strings.Join(names, ", ") + "\n\n")
}
// --- 5. Nachvollziehbarkeit ------------------------------------------
b.WriteString("## 5. Nachvollziehbarkeit (Änderungsprotokoll)\n\n")
b.WriteString("Das System führt ein fortschreibendes, nur ergänzbares Änderungsprotokoll (Audit-Log). " +
"Bestehende Protokolleinträge können weder verändert noch gelöscht werden; entsprechende " +
"Datenbankoperationen werden auf Datenbankebene unterbunden. Jeder Eintrag enthält Zeitstempel, " +
"Ereignisart, Benutzername, IP-Adresse, betroffenes Dokument, Erfolg/Misserfolg sowie eine " +
"Detailangabe. Auch fehlgeschlagene Versuche werden protokolliert.\n\n")
b.WriteString("Protokollierte Ereignisarten (Auszug):\n\n")
b.WriteString("- **Anmeldung/Sitzung:** Anmeldung, Abmeldung, fehlgeschlagene Anmeldung, Verzeichnisanmeldung (LDAP)\n")
b.WriteString("- **Dokumente:** Anlage, Änderung, erneute Verarbeitung, Seitentrennung, Zuordnung von " +
"Dokumenttyp/Korrespondent/Belegdatum, Notizen\n")
b.WriteString("- **Löschung:** Verschieben in den Papierkorb, Wiederherstellung, Löschantrag, Bestätigung, " +
"Ausführung, Ablehnung wegen laufender Aufbewahrungsfrist\n")
b.WriteString("- **Aufbewahrung:** Anlage/Änderung/Löschung von Aufbewahrungsregeln, Anwendung der Fristenläufe\n")
b.WriteString("- **Berechtigungen:** Änderungen an Berechtigungen und Gruppen, Benutzerverwaltung\n")
b.WriteString("- **Automatisierung:** Workflow-Ausführungen, Anwendung von Klassifizierungsvorlagen, " +
"maschinelle Metadatenvorschläge\n")
b.WriteString("- **Weitergabe:** Erstellung, Widerruf und Abruf von Freigabelinks\n")
b.WriteString("- **Schnittstellen:** SFTP-Zugangsdaten und SFTP-Anmeldungen, Konfigurationsänderungen\n")
b.WriteString("- **Compliance:** Erzeugung dieser Verfahrensdokumentation\n\n")
b.WriteString("Das Protokoll ist für Administratoren einsehbar und je Dokument filterbar.\n\n")
// --- 6. Manuell zu ergänzende Kapitel --------------------------------
b.WriteString("## 6. Allgemeine Beschreibung des Unternehmens und der Organisation\n\n")
b.WriteString("[MANUELL ZU ERGÄNZEN: Unternehmensgegenstand, Aufbau- und Ablauforganisation, welche " +
"Belegarten anfallen, welche Vorsysteme (z. B. Kasse, Warenwirtschaft, Buchhaltung) bestehen und wie " +
"diese mit dem Archiv zusammenwirken.]\n\n")
b.WriteString("## 7. Verantwortliche Personen und Vertretungsregelung\n\n")
b.WriteString("[MANUELL ZU ERGÄNZEN: Namentlich Verantwortliche für Archivierung, Berechtigungsvergabe, " +
"Löschfreigabe und Systembetrieb; Vertretungsregelung; Arbeitsanweisungen und deren Bekanntgabe an die " +
"Mitarbeitenden.]\n\n")
b.WriteString("## 8. Technische Systemdokumentation, Server-, Backup- und Notfallkonzept\n\n")
b.WriteString("[MANUELL ZU ERGÄNZEN: eingesetzte Hardware/Server, Standort und Betreiber, Betriebssystem- " +
"und Datenbankstand, Datensicherungsverfahren (Umfang, Häufigkeit, Aufbewahrung der Sicherungen, " +
"Auslagerung), Rücksicherungstests, Notfall- und Wiederanlaufplan, Verfahren bei Systemwechsel/Migration " +
"und Sicherstellung der Lesbarkeit über die gesamte Aufbewahrungsdauer.]\n\n")
b.WriteString("## 9. Änderungshistorie dieser Verfahrensdokumentation\n\n")
b.WriteString("[MANUELL ZU ERGÄNZEN: Versionsstände, Datum, Bearbeiter und Anlass der Änderung. Der " +
"vorliegende Entwurf gibt den Systemstand vom " + now.Format("02.01.2006 15:04:05 MST") +
" wieder und wird bei jedem Export neu erzeugt.]\n\n")
b.WriteString("---\n\n")
b.WriteString("*Automatisch erzeugter Entwurf aus dem archivdms-Systemstand. Kein rechtsverbindliches " +
"Fertigdokument — vor Verwendung gegenüber Dritten (z. B. im Rahmen einer Betriebsprüfung) durch eine " +
"fachkundige Stelle prüfen und um die als [MANUELL ZU ERGÄNZEN] gekennzeichneten Abschnitte vervollständigen.*\n")
return b.String(), nil
}
// retentionPeriodText renders the retention period of a rule in German prose.
func retentionPeriodText(ru storage.RetentionRule) string {
parts := []string{}
if ru.RetentionYears != nil && *ru.RetentionYears > 0 {
parts = append(parts, strconv.Itoa(*ru.RetentionYears)+" Jahre")
}
if ru.RetentionDays != nil && *ru.RetentionDays > 0 {
parts = append(parts, strconv.Itoa(*ru.RetentionDays)+" Tage")
}
if len(parts) == 0 {
return "nicht gesetzt"
}
return strings.Join(parts, " + ")
}
// mdCell escapes the characters that would break a Markdown table cell.
func mdCell(in string) string {
out := strings.ReplaceAll(in, "|", "\\|")
out = strings.ReplaceAll(out, "\r", " ")
out = strings.ReplaceAll(out, "\n", " ")
return strings.TrimSpace(out)
}
// orDash returns "—" for an empty string.
func orDash(in string) string {
if strings.TrimSpace(in) == "" {
return "—"
}
return in
}
// jaNein renders a bool in German.
func jaNein(b bool) string {
if b {
return "ja"
}
return "nein"
}
+320
View File
@@ -0,0 +1,320 @@
// Custom-fields HTTP handlers (see internal/storage/custom_fields.go):
//
// GET/POST /api/custom-fields PATCH/DELETE /api/custom-fields/{id}
// GET/PUT /api/document-types/{id}/fields
// GET/PUT /api/documents/{id}/fields
//
// Definition create/update/delete require domain_admin (s.authAdmin); listing
// and value-setting require an authenticated tenant context (s.auth).
// Ownership is enforced in the store layer (id+tenant_id). Every mutation is
// audit-logged, including failures, using the document lifecycle event types.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
type customFieldRequest struct {
Name string `json:"name"`
Label string `json:"label"`
FieldType string `json:"field_type"`
EnumOptions []string `json:"enum_options"`
Currency string `json:"currency"`
}
// handleListCustomFields handles GET /api/custom-fields.
func (s *Server) handleListCustomFields(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
defs, err := s.store.ListCustomFieldDefs(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list custom fields failed")
return
}
writeJSON(w, http.StatusOK, defs)
}
// handleCreateCustomField handles POST /api/custom-fields (domain_admin+).
func (s *Server) handleCreateCustomField(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req customFieldRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" || req.Label == "" || req.FieldType == "" {
writeError(w, http.StatusBadRequest, "name, label and field_type are required")
return
}
def, err := s.store.CreateCustomFieldDef(r.Context(), *sess.TenantID, storage.CustomFieldDefRequest{
Name: req.Name, Label: req.Label, FieldType: req.FieldType,
EnumOptions: req.EnumOptions, Currency: req.Currency,
})
if err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrDuplicateCustomFieldName) {
status = http.StatusConflict
} else if strings.Contains(err.Error(), "invalid field_type") {
status = http.StatusBadRequest
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "custom_field_create err:" + err.Error()})
writeError(w, status, "create custom field failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "custom_field_create id:" + strconv.FormatInt(def.ID, 10) + " name:" + def.Name,
})
writeJSON(w, http.StatusCreated, def)
}
// handleUpdateCustomField handles PATCH /api/custom-fields/{id} (domain_admin+).
func (s *Server) handleUpdateCustomField(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req customFieldRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Label == "" {
writeError(w, http.StatusBadRequest, "label is required")
return
}
def, err := s.store.UpdateCustomFieldDef(r.Context(), id, *sess.TenantID, req.Label, req.EnumOptions, req.Currency)
if err != nil {
status := http.StatusNotFound
if !errors.Is(err, storage.ErrCustomFieldNotFound) {
status = http.StatusInternalServerError
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "custom_field_update id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, "update custom field failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "custom_field_update id:" + strconv.FormatInt(def.ID, 10),
})
writeJSON(w, http.StatusOK, def)
}
// handleDeleteCustomField handles DELETE /api/custom-fields/{id} (domain_admin+).
func (s *Server) handleDeleteCustomField(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.DeleteCustomFieldDef(r.Context(), id, *sess.TenantID); err != nil {
status := http.StatusNotFound
msg := "delete custom field failed"
if errors.Is(err, storage.ErrCustomFieldInUse) {
status = http.StatusConflict
msg = "custom field still has values"
} else if !errors.Is(err, storage.ErrCustomFieldNotFound) {
status = http.StatusInternalServerError
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "custom_field_delete id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "custom_field_delete id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// --- document-type field assignments ---
type docTypeFieldAssignmentRequest struct {
FieldID int64 `json:"field_id"`
Required bool `json:"required"`
Visible bool `json:"visible"`
SortOrder int `json:"sort_order"`
}
// handleListDocumentTypeFields handles GET /api/document-types/{id}/fields.
func (s *Server) handleListDocumentTypeFields(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docTypeID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document type id")
return
}
fields, err := s.store.ListDocumentTypeFields(r.Context(), docTypeID, *sess.TenantID)
if err != nil {
if errors.Is(err, storage.ErrTaxonomyNotFound) {
writeError(w, http.StatusNotFound, "document type not found")
return
}
writeError(w, http.StatusInternalServerError, "list document type fields failed")
return
}
writeJSON(w, http.StatusOK, fields)
}
// handleSetDocumentTypeFields handles PUT /api/document-types/{id}/fields
// (bulk replace, domain_admin+).
func (s *Server) handleSetDocumentTypeFields(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docTypeID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document type id")
return
}
var reqs []docTypeFieldAssignmentRequest
if err := json.NewDecoder(r.Body).Decode(&reqs); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body (expected array)")
return
}
assignments := make([]storage.DocumentTypeFieldAssignment, 0, len(reqs))
for _, a := range reqs {
assignments = append(assignments, storage.DocumentTypeFieldAssignment{
FieldID: a.FieldID, Required: a.Required, Visible: a.Visible, SortOrder: a.SortOrder,
})
}
if err := s.store.SetDocumentTypeFields(r.Context(), docTypeID, *sess.TenantID, assignments); err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrTaxonomyNotFound) {
status = http.StatusNotFound
} else if errors.Is(err, storage.ErrCustomFieldNotFound) {
status = http.StatusBadRequest
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "doc_type_fields_set doc_type:" + strconv.FormatInt(docTypeID, 10) + " err:" + err.Error()})
writeError(w, status, "set document type fields failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "doc_type_fields_set doc_type:" + strconv.FormatInt(docTypeID, 10) + " count:" + strconv.Itoa(len(assignments)),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "updated"})
}
// --- document field values ---
// handleListDocumentFieldValues handles GET /api/documents/{id}/fields.
func (s *Server) handleListDocumentFieldValues(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
if _, err := s.store.GetDocument(r.Context(), docID, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
values, err := s.store.ListDocumentFieldValues(r.Context(), docID, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list document field values failed")
return
}
writeJSON(w, http.StatusOK, values)
}
// handleSetDocumentFieldValues handles PUT /api/documents/{id}/fields (batch).
func (s *Server) handleSetDocumentFieldValues(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
if _, err := s.store.GetDocument(r.Context(), docID, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
var inputs []storage.DocumentFieldValueInput
if err := json.NewDecoder(r.Body).Decode(&inputs); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body (expected array)")
return
}
changed, err := s.store.SetDocumentFieldValues(r.Context(), docID, *sess.TenantID, inputs)
if err != nil {
status := http.StatusInternalServerError
msg := "set document field values failed"
if errors.Is(err, storage.ErrRequiredFieldMissing) {
status = http.StatusBadRequest
msg = err.Error()
} else if errors.Is(err, storage.ErrCustomFieldNotFound) {
status = http.StatusBadRequest
msg = "unknown custom field"
} else if strings.Contains(err.Error(), "invalid date") || strings.Contains(err.Error(), "not in enum options") {
status = http.StatusBadRequest
msg = err.Error()
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: false, Detail: "custom_field_values_set err:" + err.Error(),
})
writeError(w, status, msg)
return
}
for _, name := range changed {
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentUpdate, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: true, Detail: "custom_field:" + name + " changed",
})
}
// Re-sync the search index: custom-field values are (Phase 1) not yet a
// dedicated indexed field, but the document projection is refreshed so the
// index stays consistent and a later phase can start indexing field text
// without a backfill gap. Best-effort, never fails the request.
s.store.SyncIndex(r.Context(), docID)
values, err := s.store.ListDocumentFieldValues(r.Context(), docID, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "reload document field values failed")
return
}
writeJSON(w, http.StatusOK, values)
}
+29
View File
@@ -0,0 +1,29 @@
package api
import (
"net/http"
"archivdms/internal/storage"
)
// handleDashboard returns aggregated, tenant-scoped key figures for the
// dashboard (GET /api/dashboard). Any authenticated user may view their own
// tenant's stats — no admin role required. Read-only, so no audit-log entry.
// Superadmin accounts have no tenant_id (by design, see auth.Manager.issueToken);
// they get an empty/zeroed snapshot instead of a 403, since there is no
// single tenant to scope the query to.
func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeJSON(w, http.StatusOK, &storage.DashboardStats{})
return
}
stats, err := s.store.GetDashboardStats(r.Context(), *sess.TenantID, sess.UserID)
if err != nil {
s.logger.Error("dashboard stats failed", "err", err)
writeError(w, http.StatusInternalServerError, "dashboard stats failed")
return
}
writeJSON(w, http.StatusOK, stats)
}
+216
View File
@@ -0,0 +1,216 @@
package api
import (
"regexp"
"strconv"
"strings"
"time"
)
// extractDocumentDate scans OCR text for the most plausible document/invoice
// date and returns it, or nil if none is found. It is a deliberately simple,
// rule-based (no NLP, no external library, CGO-free) heuristic in the same
// spirit as titleFromOCRText.
//
// Recognised formats:
// - DD.MM.YYYY (German, e.g. 31.12.2024)
// - DD.MM.YY (German short year, e.g. 31.12.24 -> 2024)
// - DD/MM/YYYY (slash variant, incl. DD/MM/YY)
// - YYYY-MM-DD (ISO 8601, e.g. 2024-12-31)
// - "15. März 2026" / "15. Mär. 2026" (spelled-out German month names)
//
// Plausibility: month 1-12, day 1-31 with an explicit calendar check (time.Date
// would normalise an impossible day, so we reject e.g. "31.02." rather than
// silently shifting it to March), year not before dateMinYear, and never a date
// in the future beyond today + dateFutureToleranceDays (a small tolerance for
// timezone/clock skew). These filters drop copyright years, footer years and
// stray digit runs.
//
// Scoring (see scoreForDatePosition): each candidate gets a confidence based on
// signal words in a small text window around it ("Rechnungsdatum", "vom", ...).
// The candidate with the highest score wins; on a tie the earliest occurrence in
// the text wins (document head = usually the issue date). Callers that only need
// the date use this function; callers that also need the confidence use
// extractDocumentDateWithScore.
const (
dateMinYear = 1990
dateFutureToleranceDays = 2
dateWindowRadius = 40
dateScoreNoContext = 0.4
)
// dateKeyword pairs a lowercase signal word with the confidence a date near it
// receives. Ordered by descending score: scoreForDatePosition returns the score
// of the first (=highest) keyword found in the window. Kept short and flat on
// purpose — no weighting engine, GoBD-traceable.
type dateKeyword struct {
word string
score float64
}
var dateKeywords = []dateKeyword{
{"rechnungsdatum", 0.9},
{"belegdatum", 0.9},
{"ausstellungsdatum", 0.9},
{"rechnung vom", 0.9},
{"beleg vom", 0.9},
{"datum", 0.75},
{"vom", 0.55},
}
// dateGermanMonths maps lowercased German month names and common abbreviations
// (with the trailing dot already stripped) to their month number.
var dateGermanMonths = map[string]int{
"januar": 1, "jan": 1,
"februar": 2, "feb": 2,
"märz": 3, "maerz": 3, "mär": 3, "mrz": 3,
"april": 4, "apr": 4,
"mai": 5,
"juni": 6, "jun": 6,
"juli": 7, "jul": 7,
"august": 8, "aug": 8,
"september": 9, "sep": 9, "sept": 9,
"oktober": 10, "okt": 10,
"november": 11, "nov": 11,
"dezember": 12, "dez": 12,
}
// dateCandidateRe matches every supported format in a single alternation. Named
// groups keep the branch handling readable. Word boundaries avoid gluing onto
// surrounding digits (e.g. a phone number). Case-insensitive for month names.
var dateCandidateRe = regexp.MustCompile(
`(?i)(?:\b(?P<gd>\d{1,2})\.(?P<gm>\d{1,2})\.(?P<gy>\d{4}|\d{2})\b)` +
`|(?:\b(?P<sd>\d{1,2})/(?P<sm>\d{1,2})/(?P<sy>\d{4}|\d{2})\b)` +
`|(?:\b(?P<iy>\d{4})-(?P<im>\d{1,2})-(?P<id>\d{1,2})\b)` +
`|(?:\b(?P<td>\d{1,2})\.?\s+(?P<tmon>[A-Za-zäöüÄÖÜ]+)\.?\s+(?P<ty>\d{4})\b)`,
)
// sameDate reports whether two optional dates refer to the same calendar day
// (or are both nil). Used by reprocess to skip a no-op document_date update.
func sameDate(a, b *time.Time) bool {
if a == nil || b == nil {
return a == nil && b == nil
}
ay, am, ad := a.Date()
by, bm, bd := b.Date()
return ay == by && am == bm && ad == bd
}
// scoreForDatePosition returns the confidence for a date match found at byte
// offset start in text, based on signal words within dateWindowRadius chars.
func scoreForDatePosition(lowerText string, start int) float64 {
lo := start - dateWindowRadius
if lo < 0 {
lo = 0
}
hi := start + dateWindowRadius
if hi > len(lowerText) {
hi = len(lowerText)
}
window := lowerText[lo:hi]
for _, kw := range dateKeywords {
if strings.Contains(window, kw.word) {
return kw.score
}
}
return dateScoreNoContext
}
// parseDateMatch turns one regex submatch into a validated calendar date, or
// returns ok=false if the match is implausible.
func parseDateMatch(names, m []string) (time.Time, bool) {
now := time.Now()
maxDate := now.AddDate(0, 0, dateFutureToleranceDays)
var day, month, year int
var monthName string
for i, name := range names {
if m[i] == "" {
continue
}
switch name {
case "gd", "sd", "id", "td":
day, _ = strconv.Atoi(m[i])
case "gm", "sm", "im":
month, _ = strconv.Atoi(m[i])
case "gy", "sy":
y, _ := strconv.Atoi(m[i])
if len(m[i]) == 2 {
// Two-digit year: interpret as 2000-2099. Anything above the
// future tolerance is rejected below.
y += 2000
}
year = y
case "iy", "ty":
year, _ = strconv.Atoi(m[i])
case "tmon":
monthName = m[i]
}
}
if monthName != "" {
mn, ok := dateGermanMonths[strings.ToLower(monthName)]
if !ok {
return time.Time{}, false
}
month = mn
}
if year < dateMinYear {
return time.Time{}, false
}
if month < 1 || month > 12 {
return time.Time{}, false
}
if day < 1 || day > 31 {
return time.Time{}, false
}
d := time.Date(year, time.Month(month), day, 0, 0, 0, 0, time.UTC)
// Reject normalised-away impossible days (e.g. 31.02. -> 03.03.).
if d.Day() != day || int(d.Month()) != month || d.Year() != year {
return time.Time{}, false
}
// No future dates beyond today + tolerance.
if d.After(maxDate) {
return time.Time{}, false
}
return d, true
}
// extractDocumentDateWithScore returns the best belegdatum candidate and its
// confidence. found=false when no plausible date exists in the text.
func extractDocumentDateWithScore(ocrText string) (best time.Time, score float64, found bool) {
if ocrText == "" {
return time.Time{}, 0, false
}
lower := strings.ToLower(ocrText)
idxMatches := dateCandidateRe.FindAllStringSubmatchIndex(ocrText, -1)
names := dateCandidateRe.SubexpNames()
for _, loc := range idxMatches {
m := make([]string, len(names))
for g := range names {
s, e := loc[2*g], loc[2*g+1]
if s >= 0 {
m[g] = ocrText[s:e]
}
}
d, ok := parseDateMatch(names, m)
if !ok {
continue
}
sc := scoreForDatePosition(lower, loc[0])
// Strictly greater keeps the earliest occurrence on a tie (matches are
// returned in reading order).
if !found || sc > score {
best, score, found = d, sc, true
}
}
return best, score, found
}
// extractDocumentDate returns just the best belegdatum candidate (or nil),
// preserving the original signature for callers that do not need the score.
func extractDocumentDate(ocrText string) *time.Time {
d, _, found := extractDocumentDateWithScore(ocrText)
if !found {
return nil
}
return &d
}
@@ -0,0 +1,426 @@
package api
import (
"archive/zip"
"bytes"
"encoding/csv"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// maxBulkExportDocuments caps how many documents a single bulk export may
// stream. Explicit ID lists beyond this are rejected with 400; a filter
// selection resolving to more documents is likewise rejected so the caller
// narrows the filter instead of silently receiving a truncated archive
// (GoBD-Vollständigkeit: a partial export must never look complete).
const maxBulkExportDocuments = 500
// bulkExportRequest is the POST /api/documents/export body. Either IDs or
// Filter is used — IDs takes precedence when both are present.
//
// {"ids": [1,2,3]}
// {"filter": {"doc_type_id": 4, "tag_ids": [7,9],
// "document_date_from": "2026-01-01", "document_date_to": "2026-03-31"}}
type bulkExportRequest struct {
IDs []int64 `json:"ids"`
Filter *bulkExportFilter `json:"filter"`
}
// bulkExportFilter mirrors the filter dimensions the list/search endpoints
// already expose (doc type, correspondent, tags, time range). It is applied on
// top of the tenant- and ACL-scoped result of Store.ListDocuments, so no new
// SQL predicate — and no new place a tenant_id filter could be forgotten.
type bulkExportFilter struct {
DocTypeID *int64 `json:"doc_type_id"`
CorrespondentID *int64 `json:"correspondent_id"`
TagIDs []int64 `json:"tag_ids"`
DocumentDateFrom string `json:"document_date_from"` // YYYY-MM-DD, inclusive
DocumentDateTo string `json:"document_date_to"` // YYYY-MM-DD, inclusive
UploadedFrom string `json:"uploaded_from"` // YYYY-MM-DD, inclusive
UploadedTo string `json:"uploaded_to"` // YYYY-MM-DD, inclusive (whole day)
}
// bulkExportCSVHeader is the index.csv header. Column names are deliberately
// identical to the metadata.json field names (snake_case) so the later DATEV
// formatter can map from one shared vocabulary.
var bulkExportCSVHeader = []string{
"document_id", "title", "doc_type", "correspondent",
"document_date", "tags", "uploaded_at",
}
// handleBulkExportDocuments streams a multi-document ZIP
// (POST /api/documents/export):
//
// doc-<id>/<title>.<ext> original WORM file
// doc-<id>/metadata.json same shape as the single-document export
// doc-<id>/ocr_text.txt only when OCR text exists
// index.csv one row per exported document
// errors.txt only when documents were skipped
//
// ACL: identical to the single export — tenant scoping plus, for role 'user',
// the per-document document_visibility check. Documents the caller may not see
// (or that fail to read) are skipped and listed in errors.txt rather than
// aborting the whole request.
//
// Audit: exactly ONE EventDocumentBulkExport entry per call, carrying the
// exported/skipped counts.
func (s *Server) handleBulkExportDocuments(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
tenantID := *sess.TenantID
fail := func(status int, msg, detail string) {
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentBulkExport, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: detail,
})
writeError(w, status, msg)
}
var req bulkExportRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&req); err != nil {
fail(http.StatusBadRequest, "invalid request body", "decode_failed: "+err.Error())
return
}
if len(req.IDs) == 0 && req.Filter == nil {
fail(http.StatusBadRequest, "ids oder filter erforderlich", "empty_selection")
return
}
if len(req.IDs) > maxBulkExportDocuments {
fail(http.StatusBadRequest,
fmt.Sprintf("maximal %d Dokumente pro Export (angefragt: %d)", maxBulkExportDocuments, len(req.IDs)),
fmt.Sprintf("too_many_ids: %d", len(req.IDs)))
return
}
// Role 'user' gets the group-resolved ACL applied by the store; domain
// admins and superadmins see the whole tenant (roles are the outer boundary).
var aclUserID *int64
if !auth.HasRole(sess.Role, userstore.RoleDomainAdmin) {
uid := sess.UserID
aclUserID = &uid
}
// skipped collects IDs that were requested but not exported, with a reason.
skipped := make([]string, 0, 8)
// Resolve the selection into a concrete, ordered document list.
docs := make([]storage.Document, 0, len(req.IDs))
if len(req.IDs) > 0 {
for _, id := range req.IDs {
doc, err := s.store.GetDocument(r.Context(), id, tenantID)
if err != nil || doc == nil {
skipped = append(skipped, fmt.Sprintf("%d: nicht gefunden", id))
continue
}
if aclUserID != nil {
visible, err := s.store.IsDocumentVisible(r.Context(), id, tenantID, *aclUserID)
if err != nil {
skipped = append(skipped, fmt.Sprintf("%d: Sichtbarkeitsprüfung fehlgeschlagen", id))
continue
}
if !visible {
skipped = append(skipped, fmt.Sprintf("%d: nicht sichtbar", id))
continue
}
}
docs = append(docs, *doc)
}
} else {
all, err := s.store.ListDocuments(r.Context(), tenantID, aclUserID)
if err != nil {
fail(http.StatusInternalServerError, "export failed", "list_failed: "+err.Error())
return
}
filtered, err := s.filterBulkExportDocs(r, tenantID, all, req.Filter)
if err != nil {
fail(http.StatusBadRequest, err.Error(), "filter_invalid: "+err.Error())
return
}
if len(filtered) > maxBulkExportDocuments {
fail(http.StatusBadRequest,
fmt.Sprintf("Filter trifft %d Dokumente, maximal %d pro Export — Filter eingrenzen", len(filtered), maxBulkExportDocuments),
fmt.Sprintf("filter_too_broad: %d", len(filtered)))
return
}
docs = filtered
}
if len(docs) == 0 && len(skipped) == 0 {
fail(http.StatusNotFound, "keine Dokumente für den Export gefunden", "empty_result")
return
}
ts := time.Now().UTC().Format("20060102-150405")
w.Header().Set("Content-Type", "application/zip")
w.Header().Set("Content-Disposition", "attachment; filename=\"export-bulk-"+ts+".zip\"")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(http.StatusOK)
zw := zip.NewWriter(w)
writeEntry := func(name string, rd io.Reader) error {
entry, err := zw.Create(name)
if err != nil {
return err
}
_, err = io.Copy(entry, rd)
return err
}
csvBuf := &bytes.Buffer{}
// UTF-8 BOM so Excel opens the CSV with correct umlauts.
csvBuf.WriteString("\xef\xbb\xbf")
cw := csv.NewWriter(csvBuf)
cw.Comma = ';'
_ = cw.Write(bulkExportCSVHeader)
exported := 0
var streamErr error
for i := range docs {
doc := docs[i]
row, err := s.writeBulkExportDoc(r, writeEntry, sess.Username, tenantID, &doc)
if err != nil {
// A single unreadable document must not kill the archive — unless the
// ZIP writer itself failed, which we detect on Close below.
s.logger.Warn("bulk export: document skipped", "document_id", doc.ID, "tenant_id", tenantID, "err", err)
skipped = append(skipped, fmt.Sprintf("%d: %v", doc.ID, err))
continue
}
_ = cw.Write(row)
exported++
}
cw.Flush()
if streamErr == nil {
streamErr = writeEntry("index.csv", bytes.NewReader(csvBuf.Bytes()))
}
if streamErr == nil && len(skipped) > 0 {
var b strings.Builder
b.WriteString("Übersprungene Dokumente (nicht sichtbar, nicht gefunden oder Lesefehler):\n")
for _, line := range skipped {
b.WriteString(line)
b.WriteString("\n")
}
streamErr = writeEntry("errors.txt", strings.NewReader(b.String()))
}
if closeErr := zw.Close(); streamErr == nil {
streamErr = closeErr
}
detail := fmt.Sprintf("zip_bulk_export: exported=%d skipped=%d", exported, len(skipped))
if streamErr != nil {
// Headers are already out — audit the partial export, no HTTP error.
s.logger.Warn("bulk document export stream failed", "tenant_id", tenantID, "err", streamErr)
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentBulkExport, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: detail + " stream_failed: " + streamErr.Error(),
})
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentBulkExport, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: detail,
})
}
// writeBulkExportDoc writes the doc-<id>/ folder of one document and returns
// its index.csv row. Any error means "skip this document" — the caller keeps
// the archive going and records the ID in errors.txt.
func (s *Server) writeBulkExportDoc(
r *http.Request,
writeEntry func(string, io.Reader) error,
username string,
tenantID int64,
doc *storage.Document,
) ([]string, error) {
ctx := r.Context()
prefix := "doc-" + strconv.FormatInt(doc.ID, 10) + "/"
docTypeName, correspondentName, err := s.store.DocumentTaxonomyNames(ctx, doc.ID, tenantID)
if err != nil {
return nil, fmt.Errorf("Taxonomie nicht lesbar: %w", err)
}
// Bestandsschutz: fall back to the deprecated free-text columns.
if docTypeName == "" {
docTypeName = doc.DocType
}
if correspondentName == "" {
correspondentName = doc.Correspondent
}
tagEntities, err := s.store.ListDocumentTags(ctx, doc.ID, tenantID)
if err != nil {
return nil, fmt.Errorf("Tags nicht lesbar: %w", err)
}
tags := make([]string, 0, len(tagEntities))
for _, t := range tagEntities {
tags = append(tags, t.Name)
}
fieldValues, err := s.store.ListDocumentFieldValues(ctx, doc.ID, tenantID)
if err != nil {
return nil, fmt.Errorf("Zusatzfelder nicht lesbar: %w", err)
}
f, err := os.Open(doc.StoragePath)
if err != nil {
return nil, fmt.Errorf("Datei nicht lesbar: %w", err)
}
defer f.Close()
meta := documentExportMetadata{
DocumentID: doc.ID,
TenantID: doc.TenantID,
Title: doc.Title,
DocType: docTypeName,
Correspondent: correspondentName,
Tags: tags,
DocumentDateScore: doc.DocumentDateScore,
UploadedAt: doc.CreatedAt,
UpdatedAt: doc.UpdatedAt,
CreatedBy: s.exportCreatorName(doc),
ContentHash: doc.ContentHash,
OriginalFilename: filepath.Base(doc.StoragePath),
CustomFields: exportCustomFields(fieldValues),
ExportedAt: time.Now().UTC(),
ExportedBy: username,
}
docDate := ""
if doc.DocumentDate != nil {
docDate = doc.DocumentDate.Format("2006-01-02")
meta.DocumentDate = &docDate
}
if doc.RetainUntil != nil {
d := doc.RetainUntil.Format("2006-01-02")
meta.RetainUntil = &d
}
metaJSON, err := json.MarshalIndent(meta, "", " ")
if err != nil {
return nil, fmt.Errorf("Metadaten nicht serialisierbar: %w", err)
}
ext := filepath.Ext(doc.StoragePath)
if err := writeEntry(prefix+safeDownloadName(doc.Title, ext), f); err != nil {
return nil, fmt.Errorf("ZIP-Eintrag fehlgeschlagen: %w", err)
}
if err := writeEntry(prefix+"metadata.json", bytes.NewReader(metaJSON)); err != nil {
return nil, fmt.Errorf("ZIP-Eintrag fehlgeschlagen: %w", err)
}
if doc.OCRText != "" {
if err := writeEntry(prefix+"ocr_text.txt", strings.NewReader(doc.OCRText)); err != nil {
return nil, fmt.Errorf("ZIP-Eintrag fehlgeschlagen: %w", err)
}
}
return []string{
strconv.FormatInt(doc.ID, 10),
doc.Title,
docTypeName,
correspondentName,
docDate,
strings.Join(tags, ", "),
doc.CreatedAt.UTC().Format(time.RFC3339),
}, nil
}
// filterBulkExportDocs narrows an already tenant- and ACL-scoped document list
// by the requested filter. Tag filtering needs a per-document lookup, so it is
// applied last, after the cheap in-memory predicates.
func (s *Server) filterBulkExportDocs(r *http.Request, tenantID int64, docs []storage.Document, f *bulkExportFilter) ([]storage.Document, error) {
docFrom, err := parseBulkExportDate(f.DocumentDateFrom)
if err != nil {
return nil, fmt.Errorf("ungültiges document_date_from (erwartet YYYY-MM-DD)")
}
docTo, err := parseBulkExportDate(f.DocumentDateTo)
if err != nil {
return nil, fmt.Errorf("ungültiges document_date_to (erwartet YYYY-MM-DD)")
}
upFrom, err := parseBulkExportDate(f.UploadedFrom)
if err != nil {
return nil, fmt.Errorf("ungültiges uploaded_from (erwartet YYYY-MM-DD)")
}
upTo, err := parseBulkExportDate(f.UploadedTo)
if err != nil {
return nil, fmt.Errorf("ungültiges uploaded_to (erwartet YYYY-MM-DD)")
}
out := make([]storage.Document, 0, len(docs))
for i := range docs {
d := docs[i]
if f.DocTypeID != nil && (d.DocTypeID == nil || *d.DocTypeID != *f.DocTypeID) {
continue
}
if f.CorrespondentID != nil && (d.CorrespondentID == nil || *d.CorrespondentID != *f.CorrespondentID) {
continue
}
if docFrom != nil || docTo != nil {
if d.DocumentDate == nil {
continue
}
day := d.DocumentDate.UTC().Truncate(24 * time.Hour)
if docFrom != nil && day.Before(*docFrom) {
continue
}
if docTo != nil && day.After(*docTo) {
continue
}
}
if upFrom != nil && d.CreatedAt.UTC().Before(*upFrom) {
continue
}
if upTo != nil && d.CreatedAt.UTC().After(upTo.Add(24*time.Hour-time.Nanosecond)) {
continue
}
out = append(out, d)
}
if len(f.TagIDs) == 0 {
return out, nil
}
want := make(map[int64]struct{}, len(f.TagIDs))
for _, id := range f.TagIDs {
want[id] = struct{}{}
}
tagged := make([]storage.Document, 0, len(out))
for i := range out {
tags, err := s.store.ListDocumentTags(r.Context(), out[i].ID, tenantID)
if err != nil {
return nil, fmt.Errorf("Tag-Filter fehlgeschlagen")
}
for _, t := range tags {
if _, ok := want[t.ID]; ok {
tagged = append(tagged, out[i])
break
}
}
}
return tagged, nil
}
// parseBulkExportDate parses an optional YYYY-MM-DD filter bound (UTC).
func parseBulkExportDate(s string) (*time.Time, error) {
s = strings.TrimSpace(s)
if s == "" {
return nil, nil
}
t, err := time.ParseInLocation("2006-01-02", s, time.UTC)
if err != nil {
return nil, fmt.Errorf("parse date %q: %w", s, err)
}
return &t, nil
}
+265
View File
@@ -0,0 +1,265 @@
package api
import (
"archive/zip"
"bytes"
"encoding/json"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"time"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// documentExportMetadata is the metadata.json payload of a single-document
// export. Field names are snake_case and mirror the API's document JSON so an
// exported package stays readable/parsable without the API at hand (GoBD:
// Verständlichkeit/Nachvollziehbarkeit of the exported archive package).
type documentExportMetadata struct {
DocumentID int64 `json:"document_id"`
TenantID int64 `json:"tenant_id"`
Title string `json:"title"`
DocType string `json:"doc_type"`
Correspondent string `json:"correspondent"`
Tags []string `json:"tags"`
DocumentDate *string `json:"document_date"`
DocumentDateScore *float64 `json:"document_date_score"`
UploadedAt time.Time `json:"uploaded_at"`
UpdatedAt time.Time `json:"updated_at"`
CreatedBy string `json:"created_by"`
ContentHash string `json:"content_hash"`
OriginalFilename string `json:"original_filename"`
RetainUntil *string `json:"retain_until"`
CustomFields []exportedCustomField `json:"custom_fields"`
ExportedAt time.Time `json:"exported_at"`
ExportedBy string `json:"exported_by"`
}
// exportedCustomField is one custom-field value in metadata.json. Exactly one
// of the value pointers is populated, matching the field's type.
type exportedCustomField struct {
Name string `json:"name"`
Label string `json:"label"`
FieldType string `json:"field_type"`
Currency string `json:"currency,omitempty"`
ValueText *string `json:"value_text,omitempty"`
ValueNumber *float64 `json:"value_number,omitempty"`
ValueDate *string `json:"value_date,omitempty"`
ValueBool *bool `json:"value_bool,omitempty"`
}
// handleExportDocument streams a ZIP package for a single document
// (GET /api/documents/{id}/export) containing:
//
// <title>.<ext> the original file, read from the WORM store via this handler
// (never handing out storage_path itself)
// metadata.json title, taxonomy, tags, belegdatum + score, timestamps,
// creator and custom-field values
// ocr_text.txt the OCR full text, only when the document has one
//
// ACL: tenant scoping via GetDocument (WHERE tenant_id) plus — for role 'user' —
// the same document_visibility rule as the list endpoint (IsDocumentVisible).
// domain_admin/superadmin skip the per-document check, roles being the outer
// boundary, exactly like handleListDocuments.
//
// Unlike the plain download/preview endpoints this IS audit-logged (success and
// failure): a complete metadata+content package leaving the system is treated
// like a mutation for GoBD traceability, consistent with the compliance export
// and the accounting pull API.
func (s *Server) handleExportDocument(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
idStr := r.PathValue("id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
fail := func(status int, msg, detail string) {
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentExport, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: idStr, Success: false, Detail: detail,
})
writeError(w, status, msg)
}
doc, err := s.store.GetDocument(r.Context(), id, *sess.TenantID)
if err != nil {
fail(http.StatusNotFound, "document not found", "not_found")
return
}
// Group-resolved ACL for plain users; 404 (not 403) so the endpoint never
// reveals the existence of a document the caller may not see.
if !auth.HasRole(sess.Role, userstore.RoleDomainAdmin) {
visible, err := s.store.IsDocumentVisible(r.Context(), id, *sess.TenantID, sess.UserID)
if err != nil {
fail(http.StatusInternalServerError, "export failed", "visibility_check_failed: "+err.Error())
return
}
if !visible {
fail(http.StatusNotFound, "document not found", "not_visible")
return
}
}
// Gather metadata BEFORE any byte is written — once the ZIP stream has
// started, the status code can no longer be changed.
docTypeName, correspondentName, err := s.store.DocumentTaxonomyNames(r.Context(), id, *sess.TenantID)
if err != nil {
fail(http.StatusInternalServerError, "export failed", "taxonomy_lookup_failed: "+err.Error())
return
}
// Bestandsschutz: fall back to the deprecated free-text columns when no
// structured entity is assigned.
if docTypeName == "" {
docTypeName = doc.DocType
}
if correspondentName == "" {
correspondentName = doc.Correspondent
}
tagEntities, err := s.store.ListDocumentTags(r.Context(), id, *sess.TenantID)
if err != nil {
fail(http.StatusInternalServerError, "export failed", "tag_lookup_failed: "+err.Error())
return
}
tags := make([]string, 0, len(tagEntities))
for _, t := range tagEntities {
tags = append(tags, t.Name)
}
fieldValues, err := s.store.ListDocumentFieldValues(r.Context(), id, *sess.TenantID)
if err != nil {
fail(http.StatusInternalServerError, "export failed", "field_lookup_failed: "+err.Error())
return
}
f, err := os.Open(doc.StoragePath)
if err != nil {
s.logger.Error("export: document file open failed", "document_id", doc.ID, "tenant_id", *sess.TenantID, "err", err)
fail(http.StatusInternalServerError, "file unavailable", "file_open_failed: "+err.Error())
return
}
defer f.Close()
meta := documentExportMetadata{
DocumentID: doc.ID,
TenantID: doc.TenantID,
Title: doc.Title,
DocType: docTypeName,
Correspondent: correspondentName,
Tags: tags,
DocumentDateScore: doc.DocumentDateScore,
UploadedAt: doc.CreatedAt,
UpdatedAt: doc.UpdatedAt,
CreatedBy: s.exportCreatorName(doc),
ContentHash: doc.ContentHash,
OriginalFilename: filepath.Base(doc.StoragePath),
CustomFields: exportCustomFields(fieldValues),
ExportedAt: time.Now().UTC(),
ExportedBy: sess.Username,
}
if doc.DocumentDate != nil {
d := doc.DocumentDate.Format("2006-01-02")
meta.DocumentDate = &d
}
if doc.RetainUntil != nil {
d := doc.RetainUntil.Format("2006-01-02")
meta.RetainUntil = &d
}
metaJSON, err := json.MarshalIndent(meta, "", " ")
if err != nil {
fail(http.StatusInternalServerError, "export failed", "metadata_marshal_failed: "+err.Error())
return
}
ext := filepath.Ext(doc.StoragePath)
w.Header().Set("Content-Type", "application/zip")
w.Header().Set("Content-Disposition", "attachment; filename=\"export-"+idStr+".zip\"")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(http.StatusOK)
zw := zip.NewWriter(w)
writeEntry := func(name string, r io.Reader) error {
entry, err := zw.Create(name)
if err != nil {
return err
}
_, err = io.Copy(entry, r)
return err
}
var streamErr error
// 1. Original file (read through this handler, never exposing storage_path).
if streamErr = writeEntry(safeDownloadName(doc.Title, ext), f); streamErr == nil {
// 2. metadata.json
streamErr = writeEntry("metadata.json", bytes.NewReader(metaJSON))
}
// 3. ocr_text.txt (only when OCR text exists)
if streamErr == nil && doc.OCRText != "" {
streamErr = writeEntry("ocr_text.txt", bytes.NewReader([]byte(doc.OCRText)))
}
if closeErr := zw.Close(); streamErr == nil {
streamErr = closeErr
}
if streamErr != nil {
// Headers are already out — log + audit the partial export, no HTTP error.
s.logger.Warn("document export stream failed", "document_id", doc.ID, "err", streamErr)
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentExport, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: idStr, Success: false, Detail: "stream_failed: " + streamErr.Error(),
})
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventDocumentExport, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: idStr, Success: true, Detail: "zip_export",
})
}
// exportCreatorName resolves the uploading user's username for metadata.json.
// Returns "" when the document has no created_by (e.g. SFTP watcher ingest) or
// the user has since been deleted — the export must never fail over this.
func (s *Server) exportCreatorName(doc *storage.Document) string {
if doc.CreatedBy == nil || s.users == nil {
return ""
}
u, err := s.users.GetByID(*doc.CreatedBy)
if err != nil || u == nil {
return ""
}
return u.Username
}
// exportCustomFields maps stored custom-field values to their export shape,
// normalising dates to ISO strings. Always a non-nil slice so metadata.json
// carries [] rather than null.
func exportCustomFields(values []storage.DocumentFieldValue) []exportedCustomField {
out := make([]exportedCustomField, 0, len(values))
for _, v := range values {
e := exportedCustomField{
Name: v.Name,
Label: v.Label,
FieldType: v.FieldType,
Currency: v.Currency,
ValueText: v.ValueText,
ValueNumber: v.ValueNumber,
ValueBool: v.ValueBool,
}
if v.ValueDate != nil {
d := v.ValueDate.Format("2006-01-02")
e.ValueDate = &d
}
out = append(out, e)
}
return out
}
File diff suppressed because it is too large Load Diff
+111
View File
@@ -0,0 +1,111 @@
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// handleListDocumentNotes returns all free-text notes on a document
// (GET /api/documents/{id}/notes). Tenant-scoped: ListDocumentNotes filters
// WHERE tenant_id, so a foreign-tenant id simply yields an empty list. Pure
// read — no audit entry, consistent with the other GET handlers.
func (s *Server) handleListDocumentNotes(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
notes, err := s.store.ListDocumentNotes(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list notes failed")
return
}
writeJSON(w, http.StatusOK, map[string]any{"notes": notes})
}
type createNoteRequest struct {
Text string `json:"text"`
}
// handleCreateDocumentNote adds a free-text note to a document
// (POST /api/documents/{id}/notes, body {"text": "..."}). The author is the
// authenticated user. CreateDocumentNote verifies the document belongs to the
// tenant before inserting (IDOR guard).
func (s *Server) handleCreateDocumentNote(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
var req createNoteRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
text := strings.TrimSpace(req.Text)
if text == "" {
writeError(w, http.StatusBadRequest, "text is required")
return
}
note, err := s.store.CreateDocumentNote(r.Context(), id, *sess.TenantID, sess.UserID, text)
if err != nil {
status := http.StatusInternalServerError
msg := "create note failed"
if errors.Is(err, storage.ErrDocumentNotFound) {
status = http.StatusNotFound
msg = "document not found"
}
s.audlog.Log(audit.Entry{EventType: audit.EventNoteCreate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventNoteCreate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true, Detail: "note_id=" + strconv.FormatInt(note.ID, 10)})
writeJSON(w, http.StatusCreated, note)
}
// handleDeleteDocumentNote hard-deletes a note
// (DELETE /api/documents/{id}/notes/{noteId}). Only the note's author or a
// domain admin may delete it. 403 when not permitted, 404 when the note does
// not exist for this document/tenant.
func (s *Server) handleDeleteDocumentNote(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
noteID, err := strconv.ParseInt(r.PathValue("noteId"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid note id")
return
}
isAdmin := auth.HasRole(sess.Role, userstore.RoleDomainAdmin)
if err := s.store.DeleteDocumentNote(r.Context(), noteID, id, *sess.TenantID, sess.UserID, isAdmin); err != nil {
status := http.StatusInternalServerError
msg := "delete note failed"
if errors.Is(err, storage.ErrNoteForbidden) {
status = http.StatusForbidden
msg = "not allowed to delete this note"
} else if errors.Is(err, storage.ErrNoteNotFound) {
status = http.StatusNotFound
msg = "note not found"
}
s.audlog.Log(audit.Entry{EventType: audit.EventNoteDelete, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "note_id=" + r.PathValue("noteId") + ": " + err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventNoteDelete, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true, Detail: "note_id=" + r.PathValue("noteId")})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
+168
View File
@@ -0,0 +1,168 @@
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/ldapstore"
"archivdms/internal/userstore"
)
// resolveLDAPTenant determines which tenant's LDAP config the request targets.
// domain_admin is pinned to its own session tenant. superadmin (which has no
// session tenant) must pass ?tenant_id=; may also override to inspect any
// tenant. Returns the tenant ID and false when the request is not authorised
// or the tenant cannot be determined (the caller has already written nothing).
func (s *Server) resolveLDAPTenant(w http.ResponseWriter, r *http.Request) (int64, bool) {
sess := sessionFromCtx(r.Context())
// superadmin may target any tenant via ?tenant_id=.
if sess.Role == userstore.RoleSuperAdmin {
q := r.URL.Query().Get("tenant_id")
if q == "" {
writeError(w, http.StatusBadRequest, "tenant_id query parameter required for superadmin")
return 0, false
}
tid, err := strconv.ParseInt(q, 10, 64)
if err != nil || tid <= 0 {
writeError(w, http.StatusBadRequest, "invalid tenant_id")
return 0, false
}
return tid, true
}
// domain_admin: always scoped to its own tenant (IDOR-safe: query params
// are ignored, the tenant comes from the signed session).
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "no tenant context")
return 0, false
}
return *sess.TenantID, true
}
// handleGetLDAPConfig returns the tenant's LDAP config WITHOUT the bind
// password (only bind_password_set). Returns 404 when none is configured.
func (s *Server) handleGetLDAPConfig(w http.ResponseWriter, r *http.Request) {
if s.ldapStore == nil {
writeError(w, http.StatusServiceUnavailable, "ldap not configured on this server")
return
}
tenantID, ok := s.resolveLDAPTenant(w, r)
if !ok {
return
}
cfg, err := s.ldapStore.Get(r.Context(), tenantID)
if errors.Is(err, ldapstore.ErrNotFound) {
writeError(w, http.StatusNotFound, "no ldap config for this tenant")
return
}
if err != nil {
writeError(w, http.StatusInternalServerError, "load ldap config failed")
return
}
writeJSON(w, http.StatusOK, cfg)
}
type upsertLDAPConfigRequest struct {
Enabled bool `json:"enabled"`
Host string `json:"host"`
Port int `json:"port"`
UseTLS string `json:"use_tls"`
BindDN string `json:"bind_dn"`
BindPassword *string `json:"bind_password"` // nil = keep existing
BaseDN string `json:"base_dn"`
UserFilter string `json:"user_filter"`
AttrUsername string `json:"attr_username"`
AttrEmail string `json:"attr_email"`
AttrName string `json:"attr_name"`
GroupBaseDN string `json:"group_base_dn"`
GroupFilter string `json:"group_filter"`
AdminGroupDN string `json:"admin_group_dn"`
}
// handleUpsertLDAPConfig creates or updates the tenant's LDAP config. The bind
// password is optional (omit to keep the stored one); on creation it is
// mandatory. Every attempt — success or failure — is audit-logged.
func (s *Server) handleUpsertLDAPConfig(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if s.ldapStore == nil {
writeError(w, http.StatusServiceUnavailable, "ldap not configured on this server")
return
}
tenantID, ok := s.resolveLDAPTenant(w, r)
if !ok {
return
}
logFail := func(detail string) {
tid := tenantID
s.audlog.Log(audit.Entry{
EventType: audit.EventLdapConfigChanged, Username: sess.Username,
TenantID: &tid, Success: false, Detail: detail,
})
}
var req upsertLDAPConfigRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
logFail("invalid_body")
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Host == "" || req.BindDN == "" || req.BaseDN == "" {
logFail("missing_fields")
writeError(w, http.StatusBadRequest, "host, bind_dn and base_dn are required")
return
}
// Defaults mirroring the schema so a minimal request still yields a
// working config.
if req.UserFilter == "" {
req.UserFilter = "(uid=%s)"
}
if req.AttrUsername == "" {
req.AttrUsername = "uid"
}
if req.AttrEmail == "" {
req.AttrEmail = "mail"
}
if req.AttrName == "" {
req.AttrName = "cn"
}
cfg := ldapstore.Config{
TenantID: tenantID,
Enabled: req.Enabled,
Host: req.Host,
Port: req.Port,
UseTLS: req.UseTLS,
BindDN: req.BindDN,
BaseDN: req.BaseDN,
UserFilter: req.UserFilter,
AttrUsername: req.AttrUsername,
AttrEmail: req.AttrEmail,
AttrName: req.AttrName,
GroupBaseDN: req.GroupBaseDN,
GroupFilter: req.GroupFilter,
AdminGroupDN: req.AdminGroupDN,
}
saved, err := s.ldapStore.Upsert(r.Context(), cfg, req.BindPassword)
if err != nil {
logFail("upsert_failed")
writeError(w, http.StatusBadRequest, "save ldap config failed: "+err.Error())
return
}
tid := tenantID
s.audlog.Log(audit.Entry{
EventType: audit.EventLdapConfigChanged, Username: sess.Username,
TenantID: &tid, Success: true,
Detail: "ldap_config_saved host:" + saved.Host,
})
writeJSON(w, http.StatusOK, saved)
}
@@ -0,0 +1,154 @@
// Heuristic metadata-suggestion HTTP handlers (see
// internal/storage/metadata_suggestions.go):
//
// POST /api/documents/{id}/suggest-metadata
// GET /api/documents/{id}/suggest-metadata
// POST /api/documents/{id}/suggest-metadata/{suggestionId}/reviewed
//
// All three are normal authenticated tenant actions (s.auth). Suggestions are
// rule-based (no LLM) and NON-binding: nothing here applies a suggested field —
// accepting one goes through the normal edit endpoints (PATCH title,
// tag-attach, ...). Ownership is enforced in the store layer (id+tenant_id).
package api
import (
"errors"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// handleGenerateSuggestions handles POST /api/documents/{id}/suggest-metadata.
// Triggers a fresh heuristic suggestion run and returns the persisted result.
func (s *Server) handleGenerateSuggestions(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
docRef := strconv.FormatInt(docID, 10)
// provider selects the suggestion engine: "heuristic" (default, rule-based,
// always available) or "ollama" (external LLM, only when the tenant has it
// enabled). On an Ollama failure there is NO silent fallback to heuristic —
// the error is surfaced so the frontend knows which provider did not answer.
provider := r.URL.Query().Get("provider")
if provider == "" {
provider = "heuristic"
}
var sug *storage.MetadataSuggestion
switch provider {
case "heuristic":
sug, err = s.store.GenerateHeuristicSuggestions(r.Context(), docID, *sess.TenantID, &sess.UserID)
case "ollama":
cfg, cfgErr := s.store.GetOllamaConfig(r.Context(), *sess.TenantID)
if cfgErr != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventSuggestionGenerated, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docRef, Success: false, Detail: "suggest_metadata ollama config err:" + cfgErr.Error()})
writeError(w, http.StatusInternalServerError, "load ollama config failed")
return
}
if !cfg.Enabled {
s.audlog.Log(audit.Entry{EventType: audit.EventSuggestionGenerated, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docRef, Success: false, Detail: "suggest_metadata ollama not enabled"})
writeError(w, http.StatusBadRequest, "ollama provider is not enabled for this tenant")
return
}
sug, err = s.store.GenerateOllamaSuggestions(r.Context(), docID, *sess.TenantID, &sess.UserID, *cfg)
case "naive_bayes":
// Trained, dependency-free ML classifier (internal/classifier). Yields no
// candidates for a kind whose model is untrained/below threshold — that is
// not an error. Any real failure is surfaced (no silent fallback).
sug, err = s.store.GenerateNaiveBayesSuggestions(r.Context(), docID, *sess.TenantID, &sess.UserID)
default:
writeError(w, http.StatusBadRequest, "unknown provider (use 'heuristic', 'ollama' or 'naive_bayes')")
return
}
if err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrDocumentNotFound) {
status = http.StatusNotFound
} else if provider == "ollama" {
// Ollama unreachable/timeout/bad-response: a dependency failure, not a
// server bug. 502 signals "upstream provider failed".
status = http.StatusBadGateway
}
s.audlog.Log(audit.Entry{EventType: audit.EventSuggestionGenerated, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docRef, Success: false, Detail: "suggest_metadata provider:" + provider + " err:" + err.Error()})
writeError(w, status, "generate metadata suggestions failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventSuggestionGenerated, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: docRef, Success: true, Detail: "suggest_metadata provider:" + provider + " id:" + strconv.FormatInt(sug.ID, 10),
})
writeJSON(w, http.StatusOK, sug)
}
// handleGetLatestSuggestion handles GET /api/documents/{id}/suggest-metadata.
// Returns the most recent suggestion run for the document.
func (s *Server) handleGetLatestSuggestion(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
sug, err := s.store.GetLatestSuggestion(r.Context(), docID, *sess.TenantID)
if err != nil {
if errors.Is(err, storage.ErrSuggestionNotFound) {
writeError(w, http.StatusNotFound, "no metadata suggestion found")
return
}
writeError(w, http.StatusInternalServerError, "get metadata suggestion failed")
return
}
writeJSON(w, http.StatusOK, sug)
}
// handleMarkSuggestionReviewed handles
// POST /api/documents/{id}/suggest-metadata/{suggestionId}/reviewed. Flags a
// suggestion as reviewed (the user acted on it in the UI). Which fields were
// accepted went through the normal edit endpoints, not this call.
func (s *Server) handleMarkSuggestionReviewed(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
suggestionID, err := strconv.ParseInt(r.PathValue("suggestionId"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid suggestion id")
return
}
docRef := strconv.FormatInt(docID, 10)
if err := s.store.MarkSuggestionReviewed(r.Context(), suggestionID, *sess.TenantID, sess.UserID); err != nil {
status := http.StatusNotFound
if !errors.Is(err, storage.ErrSuggestionNotFound) {
status = http.StatusInternalServerError
}
s.audlog.Log(audit.Entry{EventType: audit.EventSuggestionGenerated, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docRef, Success: false, Detail: "suggest_metadata_reviewed id:" + strconv.FormatInt(suggestionID, 10) + " err:" + err.Error()})
writeError(w, status, "mark suggestion reviewed failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventSuggestionGenerated, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: docRef, Success: true, Detail: "suggest_metadata_reviewed id:" + strconv.FormatInt(suggestionID, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "reviewed"})
}
+75
View File
@@ -0,0 +1,75 @@
package api
import (
"net/http"
"strconv"
)
// ocrWordResponse is the wire shape of one OCR word box. It is a dedicated DTO
// (not storage.OCRWord, which carries no json tags and exposes the internal row
// id / document_id) so the overlay renderer gets a compact, stable payload.
// Coordinates are in the original file's coordinate space — see
// internal/ocr/coords.go.
type ocrWordResponse struct {
Text string `json:"text"`
Left int `json:"left"`
Top int `json:"top"`
Width int `json:"width"`
Height int `json:"height"`
Confidence float64 `json:"confidence"`
Page int `json:"page"`
Block int `json:"block"`
Par int `json:"par"`
Line int `json:"line"`
}
// handleListDocumentOCRWords returns the stored word-level bounding boxes of a
// document (GET /api/documents/{id}/ocr-words), Phase 3 of the OCR
// text-highlight/overlay feature.
//
// Tenant/ACL: ocr_words has no tenant_id column, access is only ever mediated
// through document_id. The handler therefore performs the exact same ownership
// check as handleDocumentAuditLog / handleGetDocumentFile — GetDocument(id,
// tenantID) filters WHERE tenant_id and yields 404 for both "unknown id" and
// "foreign tenant", so the endpoint never reveals whether a document exists
// outside the caller's tenant — BEFORE any ocr_words row is read.
//
// Pure read: no audit entry, consistent with the other document GET handlers.
// Empty result serializes as [] (never null).
func (s *Server) handleListDocumentOCRWords(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
// ACL/tenant check first — must precede the ocr_words lookup.
if _, err := s.store.GetDocument(r.Context(), id, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
words, err := s.store.ListOCRWords(r.Context(), id)
if err != nil {
s.logger.Error("list ocr words failed", "document_id", id, "tenant_id", *sess.TenantID, "err", err)
writeError(w, http.StatusInternalServerError, "list ocr words failed")
return
}
out := make([]ocrWordResponse, 0, len(words))
for _, wd := range words {
out = append(out, ocrWordResponse{
Text: wd.Word,
Left: wd.Left,
Top: wd.Top,
Width: wd.Width,
Height: wd.Height,
Confidence: wd.Confidence,
Page: wd.Page,
Block: wd.Block,
Par: wd.Par,
Line: wd.Line,
})
}
writeJSON(w, http.StatusOK, out)
}
+134
View File
@@ -0,0 +1,134 @@
// Per-tenant configuration for an EXTERNAL, already-running Ollama server
// (never installed on the archivdms host — the base URL/port comes from the
// tenant admin). Gates the optional 'ollama' metadata-suggestion provider.
//
// GET /api/ollama-config
// PUT /api/ollama-config
//
// Both are admin-only (domain_admin manages its own tenant; superadmin must
// pass ?tenant_id=), mirroring the LDAP-config and tenant-settings handlers.
// The base URL is an internal network URL, not a secret, and is returned as-is.
package api
import (
"encoding/json"
"net/http"
"time"
"archivdms/internal/audit"
"archivdms/internal/llm"
)
// handleGetOllamaConfig returns the tenant's Ollama connection config. A tenant
// that has never configured Ollama gets a zero/default (disabled) config, not a
// 404 — the frontend always renders an editable form.
func (s *Server) handleGetOllamaConfig(w http.ResponseWriter, r *http.Request) {
tenantID, ok := s.resolveTenantSettingsTenant(w, r)
if !ok {
return
}
cfg, err := s.store.GetOllamaConfig(r.Context(), tenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "load ollama config failed")
return
}
writeJSON(w, http.StatusOK, cfg)
}
// handleListOllamaModels queries an Ollama server for the models actually
// installed there, so the frontend can offer a picklist instead of a free-text
// field. Live call, no caching. Prefers the not-yet-saved ?base_url= query
// param (lets the admin test a URL before hitting "Speichern"); falls back to
// the persisted config's base_url when the param is absent. 400 if neither is
// set. When the external Ollama server is unreachable the failure is the
// external dependency's, not ours → 502 Bad Gateway, not 500.
func (s *Server) handleListOllamaModels(w http.ResponseWriter, r *http.Request) {
tenantID, ok := s.resolveTenantSettingsTenant(w, r)
if !ok {
return
}
cfg, err := s.store.GetOllamaConfig(r.Context(), tenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "load ollama config failed")
return
}
baseURL := r.URL.Query().Get("base_url")
if baseURL == "" {
baseURL = cfg.BaseURL
}
if baseURL == "" {
writeError(w, http.StatusBadRequest, "Server-URL muss zuerst eingetragen werden")
return
}
// Short, listing-specific timeout — independent of the (possibly long)
// generate timeout. Cap the stored value so a large generate timeout does
// not make the picklist request hang for minutes.
timeout := 10 * time.Second
if cfg.TimeoutSeconds > 0 && cfg.TimeoutSeconds < 10 {
timeout = time.Duration(cfg.TimeoutSeconds) * time.Second
}
models, err := llm.ListModels(r.Context(), baseURL, timeout)
if err != nil {
writeError(w, http.StatusBadGateway, "ollama nicht erreichbar: "+err.Error())
return
}
writeJSON(w, http.StatusOK, map[string][]string{"models": models})
}
type upsertOllamaConfigRequest struct {
Enabled bool `json:"enabled"`
BaseURL string `json:"base_url"`
Model string `json:"model"`
TimeoutSeconds int `json:"timeout_seconds"`
}
// handleUpsertOllamaConfig creates or updates the tenant's Ollama connection
// config. Validation (enabled requires base_url+model, http(s) prefix, timeout
// range) lives in the store. Every attempt — success or failure — is
// audit-logged (GoBD).
func (s *Server) handleUpsertOllamaConfig(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
tenantID, ok := s.resolveTenantSettingsTenant(w, r)
if !ok {
return
}
logFail := func(detail string) {
tid := tenantID
s.audlog.Log(audit.Entry{
EventType: audit.EventOllamaConfigUpdate, Username: sess.Username,
TenantID: &tid, Success: false, Detail: detail,
})
}
var req upsertOllamaConfigRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
logFail("ollama_config invalid_body")
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if err := s.store.UpsertOllamaConfig(r.Context(), tenantID, req.Enabled, req.BaseURL, req.Model, req.TimeoutSeconds); err != nil {
logFail("ollama_config upsert_failed:" + err.Error())
writeError(w, http.StatusBadRequest, err.Error())
return
}
tid := tenantID
s.audlog.Log(audit.Entry{
EventType: audit.EventOllamaConfigUpdate, Username: sess.Username,
TenantID: &tid, Success: true, Detail: "ollama_config_saved",
})
// Reload so the response reflects the persisted (normalised) state.
cfg, err := s.store.GetOllamaConfig(r.Context(), tenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "load ollama config failed")
return
}
writeJSON(w, http.StatusOK, cfg)
}
+458
View File
@@ -0,0 +1,458 @@
// Permission-model HTTP handlers (see internal/storage/permissions.go):
//
// POST/GET/DELETE /api/permission-groups
// GET/POST/DELETE /api/permission-groups/{id}/members
// GET/POST/DELETE /api/document-types/{id}/grants
// GET/POST/DELETE /api/tags/{id}/grants
// GET/POST/DELETE /api/documents/{id}/grants (access may be 'deny')
//
// Group and grant administration require domain_admin (s.authAdmin). Ownership
// is enforced in the store layer (id + tenant_id). Every mutation is
// audit-logged (EventPermissionGrantChanged), including failures.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// logGrant records a permission mutation, success or failure.
func (s *Server) logGrant(r *http.Request, tenantID *int64, username, detail string, ok bool) {
s.audlog.Log(audit.Entry{
EventType: audit.EventPermissionGrantChanged, Username: username, TenantID: tenantID,
IPAddress: s.remoteIP(r), Success: ok, Detail: detail,
})
}
// grantStatus maps store errors to an HTTP status.
func grantStatus(err error) int {
switch {
case errors.Is(err, storage.ErrPermissionGroupNotFound):
return http.StatusNotFound
case errors.Is(err, storage.ErrGrantNotFound):
return http.StatusNotFound
case errors.Is(err, storage.ErrDuplicatePermissionGroup):
return http.StatusConflict
default:
return http.StatusInternalServerError
}
}
// --- permission groups ---
type permissionGroupRequest struct {
Name string `json:"name"`
}
// handleListPermissionGroups handles GET /api/permission-groups.
func (s *Server) handleListPermissionGroups(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
groups, err := s.store.ListPermissionGroups(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list permission groups failed")
return
}
writeJSON(w, http.StatusOK, groups)
}
// handleCreatePermissionGroup handles POST /api/permission-groups (domain_admin+).
func (s *Server) handleCreatePermissionGroup(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req permissionGroupRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
g, err := s.store.CreatePermissionGroup(r.Context(), *sess.TenantID, req.Name)
if err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "group_create name:"+req.Name+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "create permission group failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "group_create id:"+strconv.FormatInt(g.ID, 10)+" name:"+g.Name, true)
writeJSON(w, http.StatusCreated, g)
}
// handleDeletePermissionGroup handles DELETE /api/permission-groups/{id} (domain_admin+).
func (s *Server) handleDeletePermissionGroup(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.DeletePermissionGroup(r.Context(), id, *sess.TenantID); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "group_delete id:"+strconv.FormatInt(id, 10)+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "delete permission group failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "group_delete id:"+strconv.FormatInt(id, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// --- group membership ---
type groupMemberRequest struct {
UserID int64 `json:"user_id"`
}
// handleAddGroupMember handles POST /api/permission-groups/{id}/members (domain_admin+).
func (s *Server) handleAddGroupMember(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
groupID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid group id")
return
}
var req groupMemberRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.UserID == 0 {
writeError(w, http.StatusBadRequest, "user_id is required")
return
}
if err := s.store.AddGroupMember(r.Context(), groupID, req.UserID, *sess.TenantID); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "member_add group:"+strconv.FormatInt(groupID, 10)+" user:"+strconv.FormatInt(req.UserID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "add group member failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "member_add group:"+strconv.FormatInt(groupID, 10)+" user:"+strconv.FormatInt(req.UserID, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "added"})
}
// handleRemoveGroupMember handles DELETE /api/permission-groups/{id}/members/{userId} (domain_admin+).
func (s *Server) handleRemoveGroupMember(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
groupID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid group id")
return
}
userID, err := strconv.ParseInt(r.PathValue("userId"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid user id")
return
}
if err := s.store.RemoveGroupMember(r.Context(), groupID, userID, *sess.TenantID); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "member_remove group:"+strconv.FormatInt(groupID, 10)+" user:"+strconv.FormatInt(userID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "remove group member failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "member_remove group:"+strconv.FormatInt(groupID, 10)+" user:"+strconv.FormatInt(userID, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "removed"})
}
// handleListGroupMembers handles GET /api/permission-groups/{id}/members (domain_admin+).
func (s *Server) handleListGroupMembers(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
groupID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid group id")
return
}
members, err := s.store.ListGroupMembersDetailed(r.Context(), groupID, *sess.TenantID)
if err != nil {
writeError(w, grantStatus(err), "list group members failed")
return
}
writeJSON(w, http.StatusOK, members)
}
// --- grants (document-type / tag / document) ---
type grantRequest struct {
GroupID int64 `json:"group_id"`
Access string `json:"access"`
}
// handleSetDocumentTypeGrant handles POST /api/document-types/{id}/grants (domain_admin+).
func (s *Server) handleSetDocumentTypeGrant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docTypeID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document type id")
return
}
var req grantRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Access == "" {
req.Access = "read"
}
if err := s.store.SetDocumentTypeGrant(r.Context(), *sess.TenantID, docTypeID, req.GroupID, req.Access); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "type_grant_set type:"+strconv.FormatInt(docTypeID, 10)+" group:"+strconv.FormatInt(req.GroupID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatusValidated(err), "set document type grant failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "type_grant_set type:"+strconv.FormatInt(docTypeID, 10)+" group:"+strconv.FormatInt(req.GroupID, 10)+" access:"+req.Access, true)
writeJSON(w, http.StatusOK, map[string]string{"status": "granted"})
}
// handleListDocumentTypeGrants handles GET /api/document-types/{id}/grants (domain_admin+).
func (s *Server) handleListDocumentTypeGrants(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docTypeID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document type id")
return
}
grants, err := s.store.ListDocumentTypeGrants(r.Context(), *sess.TenantID, docTypeID)
if err != nil {
writeError(w, grantStatus(err), "list document type grants failed")
return
}
writeJSON(w, http.StatusOK, grants)
}
// handleDeleteDocumentTypeGrant handles DELETE /api/document-types/{id}/grants (domain_admin+).
func (s *Server) handleDeleteDocumentTypeGrant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docTypeID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document type id")
return
}
groupID, err := grantGroupID(r)
if err != nil {
writeError(w, http.StatusBadRequest, "group_id is required")
return
}
if err := s.store.DeleteDocumentTypeGrant(r.Context(), *sess.TenantID, docTypeID, groupID); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "type_grant_delete type:"+strconv.FormatInt(docTypeID, 10)+" group:"+strconv.FormatInt(groupID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "delete document type grant failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "type_grant_delete type:"+strconv.FormatInt(docTypeID, 10)+" group:"+strconv.FormatInt(groupID, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "revoked"})
}
// handleSetTagGrant handles POST /api/tags/{id}/grants (domain_admin+).
func (s *Server) handleSetTagGrant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
tagID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid tag id")
return
}
var req grantRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Access == "" {
req.Access = "read"
}
if err := s.store.SetTagGrant(r.Context(), *sess.TenantID, tagID, req.GroupID, req.Access); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "tag_grant_set tag:"+strconv.FormatInt(tagID, 10)+" group:"+strconv.FormatInt(req.GroupID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatusValidated(err), "set tag grant failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "tag_grant_set tag:"+strconv.FormatInt(tagID, 10)+" group:"+strconv.FormatInt(req.GroupID, 10)+" access:"+req.Access, true)
writeJSON(w, http.StatusOK, map[string]string{"status": "granted"})
}
// handleListTagGrants handles GET /api/tags/{id}/grants (domain_admin+).
func (s *Server) handleListTagGrants(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
tagID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid tag id")
return
}
grants, err := s.store.ListTagGrants(r.Context(), *sess.TenantID, tagID)
if err != nil {
writeError(w, grantStatus(err), "list tag grants failed")
return
}
writeJSON(w, http.StatusOK, grants)
}
// handleDeleteTagGrant handles DELETE /api/tags/{id}/grants (domain_admin+).
func (s *Server) handleDeleteTagGrant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
tagID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid tag id")
return
}
groupID, err := grantGroupID(r)
if err != nil {
writeError(w, http.StatusBadRequest, "group_id is required")
return
}
if err := s.store.DeleteTagGrant(r.Context(), *sess.TenantID, tagID, groupID); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "tag_grant_delete tag:"+strconv.FormatInt(tagID, 10)+" group:"+strconv.FormatInt(groupID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "delete tag grant failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "tag_grant_delete tag:"+strconv.FormatInt(tagID, 10)+" group:"+strconv.FormatInt(groupID, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "revoked"})
}
// handleSetDocumentGrant handles POST /api/documents/{id}/grants (domain_admin+).
// access may be 'read', 'write' or 'deny'.
func (s *Server) handleSetDocumentGrant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
var req grantRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Access == "" {
writeError(w, http.StatusBadRequest, "access is required (read|write|deny)")
return
}
if err := s.store.SetDocumentGrant(r.Context(), *sess.TenantID, docID, req.GroupID, sess.UserID, req.Access); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "doc_grant_set doc:"+strconv.FormatInt(docID, 10)+" group:"+strconv.FormatInt(req.GroupID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatusValidated(err), "set document grant failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "doc_grant_set doc:"+strconv.FormatInt(docID, 10)+" group:"+strconv.FormatInt(req.GroupID, 10)+" access:"+req.Access, true)
writeJSON(w, http.StatusOK, map[string]string{"status": "granted"})
}
// handleListDocumentGrants handles GET /api/documents/{id}/grants (domain_admin+).
// access may be 'read', 'write' or 'deny'.
func (s *Server) handleListDocumentGrants(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
grants, err := s.store.ListDocumentGrants(r.Context(), *sess.TenantID, docID)
if err != nil {
writeError(w, grantStatus(err), "list document grants failed")
return
}
writeJSON(w, http.StatusOK, grants)
}
// handleDeleteDocumentGrant handles DELETE /api/documents/{id}/grants (domain_admin+).
func (s *Server) handleDeleteDocumentGrant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
groupID, err := grantGroupID(r)
if err != nil {
writeError(w, http.StatusBadRequest, "group_id is required")
return
}
if err := s.store.DeleteDocumentGrant(r.Context(), *sess.TenantID, docID, groupID); err != nil {
s.logGrant(r, sess.TenantID, sess.Username, "doc_grant_delete doc:"+strconv.FormatInt(docID, 10)+" group:"+strconv.FormatInt(groupID, 10)+" err:"+err.Error(), false)
writeError(w, grantStatus(err), "delete document grant failed")
return
}
s.logGrant(r, sess.TenantID, sess.Username, "doc_grant_delete doc:"+strconv.FormatInt(docID, 10)+" group:"+strconv.FormatInt(groupID, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "revoked"})
}
// grantGroupID resolves the target group for a grant DELETE, from either the
// ?group_id query param or a JSON body {"group_id": N}.
func grantGroupID(r *http.Request) (int64, error) {
if q := r.URL.Query().Get("group_id"); q != "" {
return strconv.ParseInt(q, 10, 64)
}
var req grantRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
return 0, err
}
if req.GroupID == 0 {
return 0, errors.New("group_id is required")
}
return req.GroupID, nil
}
// grantStatusValidated maps an "invalid access" validation error to 400,
// otherwise defers to grantStatus.
func grantStatusValidated(err error) int {
if err == nil {
return http.StatusInternalServerError
}
if strings.Contains(err.Error(), "invalid access") {
return http.StatusBadRequest
}
return grantStatus(err)
}
+122
View File
@@ -0,0 +1,122 @@
package api
import (
"errors"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// processingJobResponse ist die schlanke Sicht auf einen Queue-Job, die das
// Frontend für Statusbadge und Retry-Button braucht. Bewusst nicht der volle
// storage.ProcessingJob: interne Felder (derive_title, next_attempt_at,
// tenant_id) haben in der UI nichts zu suchen.
type processingJobResponse struct {
DocumentID int64 `json:"document_id"`
Status string `json:"status"`
RetryCount int `json:"retry_count"`
ErrorMessage string `json:"error_message,omitempty"`
}
// handleGetProcessingJob liefert den Verarbeitungsstatus eines Dokuments
// (GET /api/documents/{id}/processing-job).
//
// ACL wie bei allen Dokument-Sub-Routen: erst GetDocument(id, tenantID) — der
// tenant_id-Filter dort ist der IDOR-Guard, ein fremdes Dokument liefert 404
// noch bevor irgendein Job gelesen wird.
//
// Hat ein Dokument keinen Job (kompletter Altbestand vor Einführung der
// Queue), wird KEIN 404 geliefert, sondern der processing_status des
// Dokuments selbst (Spalten-Default 'done'). Damit muss das Frontend keinen
// Sonderfall kennen: es bekommt immer einen Status.
func (s *Server) handleGetProcessingJob(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
tenantID := *sess.TenantID
doc, err := s.store.GetDocument(r.Context(), id, tenantID)
if err != nil {
if errors.Is(err, storage.ErrDocumentNotFound) {
writeError(w, http.StatusNotFound, "document not found")
return
}
writeError(w, http.StatusInternalServerError, "load document failed")
return
}
job, err := s.store.GetJobForDocument(r.Context(), id, tenantID)
if err != nil {
if errors.Is(err, storage.ErrNoJob) {
status := doc.ProcessingStatus
if status == "" {
status = storage.JobStatusDone
}
writeJSON(w, http.StatusOK, processingJobResponse{DocumentID: id, Status: status})
return
}
writeError(w, http.StatusInternalServerError, "load processing job failed")
return
}
writeJSON(w, http.StatusOK, processingJobResponse{
DocumentID: id,
Status: job.Status,
RetryCount: job.RetryCount,
ErrorMessage: job.ErrorMessage,
})
}
// handleRetryProcessingJob stellt einen dauerhaft fehlgeschlagenen Job manuell
// zurück in die Queue (POST /api/documents/{id}/processing-job/retry).
//
// Nur aus dem Status 'failed' heraus erlaubt — ein laufender oder bereits
// fertiger Job darf nicht zurückgesetzt werden (409), sonst könnte ein Klick
// eine gerade laufende Verarbeitung doppelt anstoßen. Der eigentliche Retry
// läuft danach ganz normal über den Dispatcher.
func (s *Server) handleRetryProcessingJob(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
tenantID := *sess.TenantID
if _, err := s.store.GetDocument(r.Context(), id, tenantID); err != nil {
if errors.Is(err, storage.ErrDocumentNotFound) {
writeError(w, http.StatusNotFound, "document not found")
return
}
writeError(w, http.StatusInternalServerError, "load document failed")
return
}
job, err := s.store.GetJobForDocument(r.Context(), id, tenantID)
if err != nil {
if errors.Is(err, storage.ErrNoJob) {
writeError(w, http.StatusNotFound, "no processing job for document")
return
}
writeError(w, http.StatusInternalServerError, "load processing job failed")
return
}
if job.Status != storage.JobStatusFailed {
writeError(w, http.StatusConflict, "processing job is not in failed state")
return
}
if err := s.store.RequeueJob(r.Context(), job.ID, tenantID); err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentProcessed, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "manual retry failed: " + err.Error()})
writeError(w, http.StatusInternalServerError, "requeue failed")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentProcessed, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true, Detail: "manual retry requeued job_id=" + strconv.FormatInt(job.ID, 10)})
writeJSON(w, http.StatusOK, processingJobResponse{DocumentID: id, Status: storage.JobStatusQueued})
}
+281
View File
@@ -0,0 +1,281 @@
// Public (unauthenticated) share-link handlers. These are wired into the mux
// WITHOUT the s.auth middleware (see server.go): the share token itself is the
// only credential. Every lookup goes through the SHA-256 token_hash, never an
// id; every attempt is rate-limited per client IP and recorded in
// document_share_accesses (and, for downloads, the audit log). The archived
// file is streamed straight from the WORM store — storage_path/content_hash are
// never exposed to the client.
//
// GET /public/share/{token} metadata (title, whether a password is needed)
// POST /public/share/{token}/download body optional {password}; streams the file
package api
import (
"encoding/json"
"errors"
"io"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// publicShareMeta is the safe, minimal public view of a share.
type publicShareMeta struct {
Title string `json:"title"`
RequiresPassword bool `json:"requires_password"`
ExpiresAt time.Time `json:"expires_at"`
}
// publicDownloadRequest is the optional JSON body for the download endpoint.
type publicDownloadRequest struct {
Password string `json:"password"`
}
// handlePublicShareMeta handles GET /public/share/{token}. It reveals only the
// document title, whether a password is required, and the expiry — never the
// file. Revoked/expired/max-reached shares are reported as such but never leak
// the title.
func (s *Server) handlePublicShareMeta(w http.ResponseWriter, r *http.Request) {
ip := s.remoteIP(r)
if !s.shareLimiter.allow(ip) {
writeError(w, http.StatusTooManyRequests, "too many requests")
return
}
token := r.PathValue("token")
rs, err := s.store.ResolveShareByToken(r.Context(), token)
if err != nil {
// Unknown token: indistinguishable 404, nothing to log (no share_id).
writeError(w, http.StatusNotFound, "share not found")
return
}
if _, stateErr := rs.VerifyState(time.Now()); stateErr != nil {
writeError(w, shareStateStatus(stateErr), shareStateMessage(stateErr))
return
}
writeJSON(w, http.StatusOK, publicShareMeta{
Title: rs.DocumentTitle,
RequiresPassword: rs.HasPassword(),
ExpiresAt: rs.ExpiresAt(),
})
}
// handlePublicShareDownload handles POST /public/share/{token}/download. Check
// order: rate-limit -> resolve -> revoked -> expired -> max_accesses ->
// password -> deliver (atomic access_count++). Every branch records an access
// row and the terminal outcome is audit-logged (EventShareAccessed).
func (s *Server) handlePublicShareDownload(w http.ResponseWriter, r *http.Request) {
ip := s.remoteIP(r)
token := r.PathValue("token")
rs, err := s.store.ResolveShareByToken(r.Context(), token)
if err != nil {
// Unknown token: 404, no share to attach an access row to.
writeError(w, http.StatusNotFound, "share not found")
return
}
// Rate limit now that we have a share_id to log a 'rate_limited' attempt.
if !s.shareLimiter.allow(ip) {
s.recordShareAccess(r, rs, ip, storage.ShareResultRateLimited)
writeError(w, http.StatusTooManyRequests, "too many requests")
return
}
if result, stateErr := rs.VerifyState(time.Now()); stateErr != nil {
s.recordShareAccess(r, rs, ip, result)
writeError(w, shareStateStatus(stateErr), shareStateMessage(stateErr))
return
}
var body publicDownloadRequest
if r.Body != nil {
_ = json.NewDecoder(r.Body).Decode(&body) // body is optional
}
if err := rs.VerifyPassword(body.Password); err != nil {
s.recordShareAccess(r, rs, ip, storage.ShareResultBadPassword)
writeError(w, http.StatusUnauthorized, "password required or incorrect")
return
}
// Atomically claim one access slot (closes the max_accesses race).
ok, err := s.store.IncrementShareAccess(r.Context(), rs.ShareID())
if err != nil {
s.logger.Error("share access increment failed", "share_id", rs.ShareID(), "err", err)
writeError(w, http.StatusInternalServerError, "download failed")
return
}
if !ok {
// Lost the race (revoked/expired/max between our check and the update).
s.recordShareAccess(r, rs, ip, storage.ShareResultMaxReached)
writeError(w, http.StatusForbidden, "share no longer available")
return
}
f, err := os.Open(rs.StoragePath())
if err != nil {
s.logger.Error("share file open failed", "share_id", rs.ShareID(), "err", err)
writeError(w, http.StatusInternalServerError, "download failed")
return
}
defer f.Close()
s.recordShareAccess(r, rs, ip, storage.ShareResultSuccess)
ext := filepath.Ext(rs.StoragePath())
w.Header().Set("Content-Type", detectMimeType("", ext, rs.StoragePath()))
w.Header().Set("Content-Disposition", "attachment; filename=\""+safeDownloadName(rs.DocumentTitle, ext)+"\"")
w.Header().Set("X-Content-Type-Options", "nosniff")
if _, err := io.Copy(w, f); err != nil {
s.logger.Warn("share file stream interrupted", "share_id", rs.ShareID(), "err", err)
}
}
// recordShareAccess writes the per-attempt access row and mirrors the outcome
// into the audit log (EventShareAccessed). Never blocks the response path.
func (s *Server) recordShareAccess(r *http.Request, rs *storage.ResolvedShare, ip, result string) {
if err := s.store.LogShareAccess(r.Context(), rs.ShareID(), ip, r.UserAgent(), result); err != nil {
s.logger.Error("share access log failed", "share_id", rs.ShareID(), "err", err)
}
tenantID := rs.TenantID()
s.audlog.Log(audit.Entry{
EventType: audit.EventShareAccessed,
Username: "public",
IPAddress: ip,
TenantID: &tenantID,
DocumentID: strconv.FormatInt(rs.DocumentID(), 10),
Success: result == storage.ShareResultSuccess,
Detail: "share:" + strconv.FormatInt(rs.ShareID(), 10) + " result:" + result,
})
}
// shareStateStatus maps a share-state error to an HTTP status.
func shareStateStatus(err error) int {
switch {
case errors.Is(err, storage.ErrShareRevoked):
return http.StatusForbidden
case errors.Is(err, storage.ErrShareExpired):
return http.StatusGone
case errors.Is(err, storage.ErrShareMaxReached):
return http.StatusForbidden
default:
return http.StatusForbidden
}
}
func shareStateMessage(err error) string {
switch {
case errors.Is(err, storage.ErrShareRevoked):
return "share revoked"
case errors.Is(err, storage.ErrShareExpired):
return "share expired"
case errors.Is(err, storage.ErrShareMaxReached):
return "share access limit reached"
default:
return "share not available"
}
}
// safeDownloadName builds a Content-Disposition filename from the document
// title, stripping anything that could break the header or the client's
// filesystem, and appending the stored extension.
func safeDownloadName(title, ext string) string {
title = strings.TrimSpace(title)
if title == "" {
title = "document"
}
var b strings.Builder
for _, ch := range title {
switch {
case ch >= 'a' && ch <= 'z', ch >= 'A' && ch <= 'Z', ch >= '0' && ch <= '9':
b.WriteRune(ch)
case ch == '-', ch == '_', ch == '.', ch == ' ':
b.WriteRune(ch)
default:
b.WriteRune('_')
}
}
name := strings.TrimSpace(b.String())
if name == "" {
name = "document"
}
if ext != "" && !strings.HasSuffix(strings.ToLower(name), strings.ToLower(ext)) {
name += ext
}
return name
}
// --- per-IP token-bucket rate limiter ---
// ipRateLimiter is a minimal in-memory per-IP token-bucket limiter (no external
// dependency). Each IP gets its own bucket of `burst` tokens, refilled at
// `refillPerSec` tokens per second. Buckets are created lazily and swept when
// they have been idle and full for a while.
type ipRateLimiter struct {
mu sync.Mutex
buckets map[string]*tokenBucket
burst float64
refillPerSec float64
lastSweep time.Time
}
type tokenBucket struct {
tokens float64
last time.Time
}
func newIPRateLimiter(burst, refillPerSec float64) *ipRateLimiter {
return &ipRateLimiter{
buckets: make(map[string]*tokenBucket),
burst: burst,
refillPerSec: refillPerSec,
lastSweep: time.Now(),
}
}
// allow consumes one token for ip, returning false when the bucket is empty.
func (l *ipRateLimiter) allow(ip string) bool {
now := time.Now()
l.mu.Lock()
defer l.mu.Unlock()
l.sweepLocked(now)
b, ok := l.buckets[ip]
if !ok {
b = &tokenBucket{tokens: l.burst, last: now}
l.buckets[ip] = b
}
// Refill based on elapsed time.
elapsed := now.Sub(b.last).Seconds()
b.tokens += elapsed * l.refillPerSec
if b.tokens > l.burst {
b.tokens = l.burst
}
b.last = now
if b.tokens < 1 {
return false
}
b.tokens--
return true
}
// sweepLocked drops idle, full buckets roughly once a minute to bound memory.
func (l *ipRateLimiter) sweepLocked(now time.Time) {
if now.Sub(l.lastSweep) < time.Minute {
return
}
l.lastSweep = now
for ip, b := range l.buckets {
if now.Sub(b.last) > 10*time.Minute {
delete(l.buckets, ip)
}
}
}
+171
View File
@@ -0,0 +1,171 @@
// Wiedervorlage (reminder) HTTP handlers:
// POST /api/documents/{id}/reminders
// GET /api/reminders?status=
// PATCH /api/reminders/{id}
// DELETE /api/reminders/{id}
//
// All routes require s.auth(...) (authenticated + tenant context). Ownership
// is enforced in the store layer (id+tenant_id+user_id). Every mutation is
// audit-logged, including failures.
package api
import (
"encoding/json"
"net/http"
"strconv"
"time"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
type createReminderRequest struct {
DueDate string `json:"due_date"` // RFC3339
Note string `json:"note"`
}
// handleCreateReminder handles POST /api/documents/{id}/reminders.
func (s *Server) handleCreateReminder(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
var req createReminderRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
dueDate, err := time.Parse(time.RFC3339, req.DueDate)
if err != nil {
writeError(w, http.StatusBadRequest, "due_date must be RFC3339")
return
}
// Verify the document exists and belongs to the caller's tenant before
// attaching a reminder to it (the FK alone would only stop a fully
// nonexistent document_id, not a cross-tenant one).
if _, err := s.store.GetDocument(r.Context(), docID, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
rem, err := s.store.CreateReminder(r.Context(), docID, *sess.TenantID, sess.UserID, dueDate, req.Note)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventReminderCreate, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: false, Detail: err.Error(),
})
writeError(w, http.StatusInternalServerError, "create reminder failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventReminderCreate, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: true,
Detail: "reminder_id:" + strconv.FormatInt(rem.ID, 10),
})
writeJSON(w, http.StatusCreated, rem)
}
// handleListReminders handles GET /api/reminders?status=.
func (s *Server) handleListReminders(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
status := r.URL.Query().Get("status")
if status != "" && status != storage.ReminderStatusOpen && status != storage.ReminderStatusDone && status != storage.ReminderStatusDismissed {
writeError(w, http.StatusBadRequest, "invalid status filter")
return
}
reminders, err := s.store.ListReminders(r.Context(), *sess.TenantID, sess.UserID, status)
if err != nil {
writeError(w, http.StatusInternalServerError, "list reminders failed")
return
}
writeJSON(w, http.StatusOK, reminders)
}
type updateReminderRequest struct {
Status string `json:"status"`
}
// handleUpdateReminder handles PATCH /api/reminders/{id}.
func (s *Server) handleUpdateReminder(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid reminder id")
return
}
var req updateReminderRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
switch req.Status {
case storage.ReminderStatusOpen, storage.ReminderStatusDone, storage.ReminderStatusDismissed:
default:
writeError(w, http.StatusBadRequest, "invalid status")
return
}
rem, err := s.store.UpdateReminderStatus(r.Context(), id, *sess.TenantID, sess.UserID, req.Status)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventReminderStatusChange, Username: sess.Username, TenantID: sess.TenantID,
Detail: "reminder_id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(), Success: false,
})
writeError(w, http.StatusNotFound, "reminder not found")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventReminderStatusChange, Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(rem.DocumentID, 10), Success: true,
Detail: "reminder_id:" + strconv.FormatInt(rem.ID, 10) + " status:" + rem.Status,
})
writeJSON(w, http.StatusOK, rem)
}
// handleDeleteReminder handles DELETE /api/reminders/{id}.
func (s *Server) handleDeleteReminder(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid reminder id")
return
}
if err := s.store.DeleteReminder(r.Context(), id, *sess.TenantID, sess.UserID); err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventReminderDelete, Username: sess.Username, TenantID: sess.TenantID,
Detail: "reminder_id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(), Success: false,
})
writeError(w, http.StatusNotFound, "reminder not found")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventReminderDelete, Username: sess.Username, TenantID: sess.TenantID,
Detail: "reminder_id:" + strconv.FormatInt(id, 10), Success: true,
})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
+247
View File
@@ -0,0 +1,247 @@
// GoBD retention-rule ("Aufbewahrungsregeln") HTTP handlers (see
// internal/storage/retention_rules.go):
//
// GET /api/retention-rules list all rules of the tenant
// POST /api/retention-rules create a rule
// PATCH /api/retention-rules/{id} update a rule
// DELETE /api/retention-rules/{id} delete a rule
// GET /api/retention-rules/eligible documents eligible for disposition
// GET /api/retention-rules/preview dry-run of ApplyRetentionRules (no write)
//
// Rules are compliance-critical (they define how long documents must be kept),
// so create/update/delete require domain_admin (s.authAdmin). Reading (list,
// eligible, preview) is a normal tenant action (s.auth). Ownership is enforced
// in the store layer (id+tenant_id). Every mutation is audit-logged, including
// failures.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// retentionRuleRequest is the JSON body for create/update. Pointers where the
// store field is a pointer, so "not set" round-trips correctly.
type retentionRuleRequest struct {
DocTypeID *int64 `json:"doc_type_id"`
Name string `json:"name"`
TriggerType string `json:"trigger_type"`
TriggerReference string `json:"trigger_reference"`
RetentionYears *int `json:"retention_years"`
RetentionDays *int `json:"retention_days"`
LegalBasis string `json:"legal_basis"`
RequiresApprovalForDestroy *bool `json:"requires_approval_for_destroy"`
DSGVOConflict *bool `json:"dsgvo_conflict"`
Active *bool `json:"active"`
}
// toRule maps the request onto a storage.RetentionRule. requires_approval and
// active default to true when omitted (safe GoBD default: keep approval on).
func (req retentionRuleRequest) toRule() storage.RetentionRule {
requiresApproval := true
if req.RequiresApprovalForDestroy != nil {
requiresApproval = *req.RequiresApprovalForDestroy
}
active := true
if req.Active != nil {
active = *req.Active
}
dsgvo := false
if req.DSGVOConflict != nil {
dsgvo = *req.DSGVOConflict
}
return storage.RetentionRule{
DocTypeID: req.DocTypeID,
Name: req.Name,
TriggerType: req.TriggerType,
TriggerReference: req.TriggerReference,
RetentionYears: req.RetentionYears,
RetentionDays: req.RetentionDays,
LegalBasis: req.LegalBasis,
RequiresApprovalForDestroy: requiresApproval,
DSGVOConflict: dsgvo,
Active: active,
}
}
// handleListRetentionRules handles GET /api/retention-rules.
func (s *Server) handleListRetentionRules(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
rules, err := s.store.ListRetentionRules(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list retention rules failed")
return
}
writeJSON(w, http.StatusOK, rules)
}
// handleCreateRetentionRule handles POST /api/retention-rules (domain_admin+).
func (s *Server) handleCreateRetentionRule(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req retentionRuleRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
rule := req.toRule()
rule.CreatedBy = &sess.UserID
created, err := s.store.CreateRetentionRule(r.Context(), *sess.TenantID, rule)
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventRetentionRuleCreate, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "retention_rule_create err:" + err.Error(),
})
writeError(w, retentionRuleErrStatus(err), retentionRuleErrMsg(err, "create retention rule failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventRetentionRuleCreate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "retention_rule_create id:" + strconv.FormatInt(created.ID, 10) + " name:" + created.Name,
})
writeJSON(w, http.StatusCreated, created)
}
// handleUpdateRetentionRule handles PATCH /api/retention-rules/{id} (domain_admin+).
func (s *Server) handleUpdateRetentionRule(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req retentionRuleRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
updated, err := s.store.UpdateRetentionRule(r.Context(), id, *sess.TenantID, req.toRule())
if err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventRetentionRuleUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "retention_rule_update id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(),
})
writeError(w, retentionRuleErrStatus(err), retentionRuleErrMsg(err, "update retention rule failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventRetentionRuleUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "retention_rule_update id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, updated)
}
// handleDeleteRetentionRule handles DELETE /api/retention-rules/{id} (domain_admin+).
func (s *Server) handleDeleteRetentionRule(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.DeleteRetentionRule(r.Context(), id, *sess.TenantID); err != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventRetentionRuleDelete, Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "retention_rule_delete id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(),
})
writeError(w, retentionRuleErrStatus(err), retentionRuleErrMsg(err, "delete retention rule failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventRetentionRuleDelete, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "retention_rule_delete id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// handleListEligibleForDisposition handles GET /api/retention-rules/eligible:
// documents whose retention has expired but which are not yet in the trash.
func (s *Server) handleListEligibleForDisposition(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docs, err := s.store.ListEligibleForDisposition(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list eligible-for-disposition documents failed")
return
}
writeJSON(w, http.StatusOK, docs)
}
// handlePreviewRetentionRules handles GET /api/retention-rules/preview: a
// dry-run of ApplyRetentionRules for the current tenant. No writes.
func (s *Server) handlePreviewRetentionRules(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
preview, err := s.store.PreviewRetentionRules(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "preview retention rules failed")
return
}
if preview == nil {
preview = []storage.RetentionPreview{}
}
writeJSON(w, http.StatusOK, preview)
}
// retentionRuleErrStatus maps store errors to HTTP status codes. Validation
// errors (bad trigger_type, missing retention period, bad fixed_date) surface as
// 400; not-found as 404; everything else 500.
func retentionRuleErrStatus(err error) int {
if errors.Is(err, storage.ErrRetentionRuleNotFound) {
return http.StatusNotFound
}
if isRetentionValidationErr(err) {
return http.StatusBadRequest
}
return http.StatusInternalServerError
}
// retentionRuleErrMsg returns the validation message verbatim (safe, no PII) so
// the frontend can show it, or a generic fallback otherwise.
func retentionRuleErrMsg(err error, fallback string) string {
if errors.Is(err, storage.ErrRetentionRuleNotFound) {
return "retention rule not found"
}
if isRetentionValidationErr(err) {
return err.Error()
}
return fallback
}
// isRetentionValidationErr reports whether err is a validateRetentionRule
// cross-field error (all prefixed "retention rule:" in the store).
func isRetentionValidationErr(err error) bool {
if err == nil || errors.Is(err, storage.ErrRetentionRuleNotFound) {
return false
}
msg := err.Error()
const prefix = "retention rule: "
return len(msg) >= len(prefix) && msg[:len(prefix)] == prefix
}
+140
View File
@@ -0,0 +1,140 @@
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// handleListSavedViews serves GET /api/saved-views — the caller's own saved
// search views plus every view shared tenant-wide (is_shared). Pure read, not
// audited (consistent with the rest of the project).
func (s *Server) handleListSavedViews(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
views, err := s.store.ListSavedViews(r.Context(), *sess.TenantID, sess.UserID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list saved views failed")
return
}
writeJSON(w, http.StatusOK, map[string]any{"views": views})
}
// savedViewRequest is the create/update body. Filters carries the serialized
// search query verbatim (see index.SearchQuery / handleSearchDocuments); it is
// stored as-is in the saved_views.filters JSONB column so the client can
// re-hydrate it 1:1 into a new search request.
type savedViewRequest struct {
Name string `json:"name"`
Filters json.RawMessage `json:"filters"`
IsShared bool `json:"is_shared"`
}
// handleCreateSavedView serves POST /api/saved-views.
func (s *Server) handleCreateSavedView(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req savedViewRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
name := strings.TrimSpace(req.Name)
if name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
// Default an empty/omitted filters payload to an empty JSON object so the
// NOT NULL JSONB column always receives valid JSON.
if len(req.Filters) == 0 {
req.Filters = json.RawMessage("{}")
}
view, err := s.store.CreateSavedView(r.Context(), *sess.TenantID, sess.UserID, name, req.Filters, req.IsShared)
if err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventSavedViewCreate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: err.Error()})
writeError(w, http.StatusInternalServerError, "create saved view failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventSavedViewCreate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "saved_view_id=" + strconv.FormatInt(view.ID, 10),
})
writeJSON(w, http.StatusCreated, view)
}
// handleUpdateSavedView serves PATCH /api/saved-views/{id}. Only the view's
// creator may update it (403 otherwise).
func (s *Server) handleUpdateSavedView(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid saved view id")
return
}
var req savedViewRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
name := strings.TrimSpace(req.Name)
if name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
if len(req.Filters) == 0 {
req.Filters = json.RawMessage("{}")
}
if err := s.store.UpdateSavedView(r.Context(), id, *sess.TenantID, sess.UserID, name, req.Filters, req.IsShared); err != nil {
status, msg := savedViewErrStatus(err, "update saved view failed")
s.audlog.Log(audit.Entry{EventType: audit.EventSavedViewUpdate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventSavedViewUpdate, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, map[string]string{"status": "updated"})
}
// handleDeleteSavedView serves DELETE /api/saved-views/{id}. Only the view's
// creator may delete it (403 otherwise).
func (s *Server) handleDeleteSavedView(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil || sess.TenantID == nil {
writeError(w, http.StatusBadRequest, "invalid saved view id")
return
}
if err := s.store.DeleteSavedView(r.Context(), id, *sess.TenantID, sess.UserID); err != nil {
status, msg := savedViewErrStatus(err, "delete saved view failed")
s.audlog.Log(audit.Entry{EventType: audit.EventSavedViewDelete, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventSavedViewDelete, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
// savedViewErrStatus maps store errors to an HTTP status + message: not-found
// -> 404, forbidden (exists but owned by another user) -> 403, else 500.
func savedViewErrStatus(err error, defaultMsg string) (int, string) {
switch {
case errors.Is(err, storage.ErrSavedViewNotFound):
return http.StatusNotFound, "saved view not found"
case errors.Is(err, storage.ErrSavedViewForbidden):
return http.StatusForbidden, "not allowed to modify this saved view"
default:
return http.StatusInternalServerError, defaultMsg
}
}
+122
View File
@@ -0,0 +1,122 @@
package api
import (
"errors"
"net/http"
"strconv"
"strings"
"archivdms/internal/auth"
"archivdms/internal/index"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// handleSearchDocuments serves GET /api/documents/search — full-text +
// attribute search backed by the per-tenant Manticore index.
//
// q full-text term (matched against title/ocr_text/tags/
// correspondent/doc_type). Optional; when empty the query
// degrades to a filter-only listing ordered by recency.
// tag repeatable tag id filter (documents carrying ANY given tag).
// doc_type_id restrict to a single document type.
// page 1-based page number (default 1).
// page_size hits per page (default 20, capped at 100).
//
// ACL: role 'user' is filtered against their permission-group memberships
// (ANY(acl_group_ids)); domain_admin/superadmin bypass the ACL, exactly like
// handleListDocuments. When the search backend is not configured the endpoint
// returns 503 rather than a silent empty result.
func (s *Server) handleSearchDocuments(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
tenantID := *sess.TenantID
q := index.SearchQuery{
Query: strings.TrimSpace(r.URL.Query().Get("q")),
Page: parsePositiveInt(r.URL.Query().Get("page"), 1),
PageSize: clampInt(parsePositiveInt(r.URL.Query().Get("page_size"), 20), 1, 100),
TagIDs: parseInt64List(r.URL.Query()["tag"]),
DocTypeID: parseOptionalInt64(r.URL.Query().Get("doc_type_id")),
}
// Roles are the outer ACL boundary (see handleListDocuments): role 'user'
// is filtered against document_visibility via their group memberships;
// domain_admin/superadmin see every document in the tenant (ACLGroupIDs nil).
if !auth.HasRole(sess.Role, userstore.RoleDomainAdmin) {
groupIDs, err := s.store.ListGroupIDsForUser(r.Context(), sess.UserID, tenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "search failed")
return
}
// Non-nil (possibly empty) slice => ACL filter applies. An empty slice
// means the user is in no group and therefore sees nothing.
q.ACLGroupIDs = groupIDs
}
res, err := s.store.SearchDocuments(r.Context(), tenantID, q)
if err != nil {
if errors.Is(err, storage.ErrSearchUnavailable) {
writeError(w, http.StatusServiceUnavailable, "Suche nicht verfügbar, Manticore nicht konfiguriert")
return
}
writeError(w, http.StatusInternalServerError, "search failed")
return
}
writeJSON(w, http.StatusOK, res)
}
// --- small query-param parsing helpers ---
func parsePositiveInt(s string, def int) int {
if s == "" {
return def
}
n, err := strconv.Atoi(s)
if err != nil || n <= 0 {
return def
}
return n
}
func clampInt(n, min, max int) int {
if n < min {
return min
}
if n > max {
return max
}
return n
}
func parseOptionalInt64(s string) *int64 {
if s == "" {
return nil
}
n, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return nil
}
return &n
}
// parseInt64List parses a slice of query values (each possibly comma-separated)
// into positive int64 ids, silently dropping anything unparseable.
func parseInt64List(vals []string) []int64 {
var out []int64
for _, v := range vals {
for _, part := range strings.Split(v, ",") {
part = strings.TrimSpace(part)
if part == "" {
continue
}
if n, err := strconv.ParseInt(part, 10, 64); err == nil && n > 0 {
out = append(out, n)
}
}
}
return out
}
+554
View File
@@ -0,0 +1,554 @@
// Package api is the archivdms HTTP API server, ported from archivmail's
// internal/api/server.go pattern: net/http ServeMux, an s.auth/s.authAdmin
// middleware chain, JWT session extraction, and application-level
// tenant-context propagation (tenantFromCtx). No mail-specific routes.
package api
import (
"context"
"encoding/json"
"log/slog"
"net"
"net/http"
"strings"
"time"
"archivdms/config"
"archivdms/internal/audit"
"archivdms/internal/auth"
"archivdms/internal/ldapauth"
"archivdms/internal/ldapstore"
"archivdms/internal/mailer"
"archivdms/internal/ocr"
"archivdms/internal/pagesplit"
"archivdms/internal/storage"
"archivdms/internal/tenantstore"
"archivdms/internal/thumbnail"
"archivdms/internal/userstore"
)
type contextKey string
const (
sessionKey contextKey = "session"
tenantKey contextKey = "tenant_id"
)
// Server is the archivdms HTTP API server.
type Server struct {
cfg config.APIConfig
storageCfg config.StorageConfig
startTime time.Time
store *storage.Store
authMgr *auth.Manager
users *userstore.Store
audlog *audit.Logger
logger *slog.Logger
mux *http.ServeMux
ocr *ocr.Extractor
thumbs *thumbnail.Generator
// pagesplitter performs barcode separator-page splitting of multi-page PDF
// uploads before archival (internal/pagesplit). May be nil / disabled, in
// which case every upload is archived as a single document as before.
pagesplitter *pagesplit.Detector
tenantStore *tenantstore.Store
mailer *mailer.Mailer
fqdn string
appVersion string
// ldapStore/ldapAuth are wired via SetLDAP. Both may be nil when LDAP is
// unconfigured — the config endpoints then return 503.
ldapStore *ldapstore.Store
ldapAuth *ldapauth.Authenticator
// shareLimiter rate-limits the unauthenticated public share endpoints
// (per client IP) to blunt token/password enumeration.
shareLimiter *ipRateLimiter
// accountingLimiter rate-limits the Bearer-key Buchhaltungs-Pull-API
// (per client IP) to blunt API-key guessing. Separate bucket set from
// shareLimiter so a busy accounting client cannot starve share downloads.
accountingLimiter *ipRateLimiter
}
// SetStorageConfig wires the storage configuration (inbox/store/ocr-tmp
// paths, max upload size) into the API server. Needed by
// handleUploadDocument, which cannot rely solely on the storage.Store
// (that only knows its own base dir, not the inbox/ocr-tmp layout).
func (s *Server) SetStorageConfig(cfg config.StorageConfig) {
s.storageCfg = cfg
}
// SetOCR wires the OCR extractor into the API server. May be nil, in which
// case uploads succeed with an empty ocr_text and an audit warning.
func (s *Server) SetOCR(e *ocr.Extractor) {
s.ocr = e
}
// SetThumbnailer wires the preview-thumbnail generator. May be nil, in which
// case the thumbnail endpoint returns 404 and the UI falls back to a generic
// file icon.
func (s *Server) SetThumbnailer(g *thumbnail.Generator) {
s.thumbs = g
}
// SetPageSplitter wires the barcode separator-page detector used at ingest.
// May be nil or disabled (config.PageSplitConfig.Enabled=false, the default),
// in which case multi-page uploads are archived unsplit as before.
func (s *Server) SetPageSplitter(d *pagesplit.Detector) {
s.pagesplitter = d
}
// SetTenants wires the tenant store into the API server after construction.
func (s *Server) SetTenants(ts *tenantstore.Store) {
s.tenantStore = ts
}
// SetLDAP wires the per-tenant LDAP config store and authenticator into the
// API server. Both may be nil (LDAP unconfigured); the config endpoints then
// respond 503.
func (s *Server) SetLDAP(store *ldapstore.Store, authn *ldapauth.Authenticator) {
s.ldapStore = store
s.ldapAuth = authn
}
// SetMailer wires the outbound mailer into the API server.
func (s *Server) SetMailer(m *mailer.Mailer) {
s.mailer = m
}
// SetFQDN wires the server FQDN for link generation in emails.
func (s *Server) SetFQDN(fqdn string) {
s.fqdn = fqdn
}
// SetVersion wires the app version into the API server.
func (s *Server) SetVersion(v string) {
s.appVersion = v
}
// New creates and wires up a new API server.
func New(
cfg config.APIConfig,
store *storage.Store,
authMgr *auth.Manager,
users *userstore.Store,
audlog *audit.Logger,
logger *slog.Logger,
) *Server {
s := &Server{
cfg: cfg,
store: store,
authMgr: authMgr,
users: users,
audlog: audlog,
logger: logger,
mux: http.NewServeMux(),
startTime: time.Now(),
// 20 requests burst, refilled at 1/sec per client IP.
shareLimiter: newIPRateLimiter(20, 1.0),
// Batch pulls are legitimate here: 60 requests burst, refilled at 5/sec.
accountingLimiter: newIPRateLimiter(60, 5.0),
}
s.routes()
return s
}
// auth wraps a handler with authentication + tenant context propagation.
func (s *Server) auth(h http.HandlerFunc) http.HandlerFunc {
return s.authMiddleware(s.tenantMiddleware(h))
}
// authAdmin wraps a handler requiring at least domain_admin role.
func (s *Server) authAdmin(h http.HandlerFunc) http.HandlerFunc {
return s.authMiddleware(s.tenantMiddleware(s.requireRole(userstore.RoleDomainAdmin, h)))
}
func (s *Server) routes() {
s.mux.HandleFunc("GET /api/health", s.handleHealth)
s.mux.HandleFunc("GET /api/version", s.handleVersion)
s.mux.HandleFunc("POST /api/auth/login", s.handleLogin)
s.mux.HandleFunc("GET /api/auth/me", s.auth(s.handleMe))
s.mux.HandleFunc("POST /api/auth/logout", s.auth(s.handleLogout))
s.mux.HandleFunc("GET /api/users", s.authAdmin(s.handleListUsers))
s.mux.HandleFunc("POST /api/users", s.authAdmin(s.handleCreateUser))
s.mux.HandleFunc("PATCH /api/users/{id}", s.authAdmin(s.handleUpdateUser))
s.mux.HandleFunc("DELETE /api/users/{id}", s.authAdmin(s.handleDeleteUser))
s.mux.HandleFunc("GET /api/audit", s.auth(s.requireRole(userstore.RoleDomainAdmin, s.handleAuditLog)))
// Tenant management: superadmin-only (internal/api/tenant_handlers.go).
s.mux.HandleFunc("POST /api/tenants", s.auth(s.requireRole(userstore.RoleSuperAdmin, s.handleCreateTenant)))
s.mux.HandleFunc("GET /api/tenants", s.auth(s.requireRole(userstore.RoleSuperAdmin, s.handleListTenants)))
// Dashboard (aggregated tenant key figures, internal/api/dashboard_handlers.go)
s.mux.HandleFunc("GET /api/dashboard", s.auth(s.handleDashboard))
// Documents (core model)
s.mux.HandleFunc("GET /api/documents", s.auth(s.handleListDocuments))
s.mux.HandleFunc("POST /api/documents", s.auth(s.handleCreateDocument))
s.mux.HandleFunc("POST /api/documents/upload", s.auth(s.handleUploadDocument))
// Full-text + attribute search (internal/api/search_handlers.go). Registered
// as a literal path; Go 1.22 ServeMux prefers it over /api/documents/{id}.
s.mux.HandleFunc("GET /api/documents/search", s.auth(s.handleSearchDocuments))
// Bulk-Export: ZIP mit doc-<id>/-Ordner je Dokument + index.csv
// (internal/api/document_bulk_export_handlers.go). Literaler Pfad, daher
// kein Konflikt mit /api/documents/{id}.
s.mux.HandleFunc("POST /api/documents/export", s.auth(s.handleBulkExportDocuments))
s.mux.HandleFunc("GET /api/documents/{id}", s.auth(s.handleGetDocument))
s.mux.HandleFunc("GET /api/documents/{id}/file", s.auth(s.handleGetDocumentFile))
s.mux.HandleFunc("GET /api/documents/{id}/thumbnail", s.auth(s.handleGetDocumentThumbnail))
s.mux.HandleFunc("GET /api/documents/{id}/audit", s.auth(s.handleDocumentAuditLog))
// Einzel-Dokument-Export: ZIP (Originaldatei + metadata.json + ocr_text.txt).
s.mux.HandleFunc("GET /api/documents/{id}/export", s.auth(s.handleExportDocument))
// OCR-Wortkoordinaten für das Text-Overlay (internal/api/ocr_word_handlers.go).
s.mux.HandleFunc("GET /api/documents/{id}/ocr-words", s.auth(s.handleListDocumentOCRWords))
s.mux.HandleFunc("PATCH /api/documents/{id}", s.auth(s.handleUpdateDocumentTitle))
s.mux.HandleFunc("PUT /api/documents/{id}/doc-type", s.auth(s.handleSetDocumentDocType))
s.mux.HandleFunc("PUT /api/documents/{id}/correspondent", s.auth(s.handleSetDocumentCorrespondent))
s.mux.HandleFunc("PUT /api/documents/{id}/document-date", s.auth(s.handleSetDocumentDate))
s.mux.HandleFunc("DELETE /api/documents/{id}", s.auth(s.handleDeleteDocument))
s.mux.HandleFunc("POST /api/documents/{id}/reprocess", s.auth(s.handleReprocessDocument))
// Status/manueller Retry der asynchronen Verarbeitungswarteschlange
// (internal/api/processing_job_handlers.go). Das Frontend pollt den
// GET-Endpunkt nur solange ein Dokument nicht 'done' ist.
s.mux.HandleFunc("GET /api/documents/{id}/processing-job", s.auth(s.handleGetProcessingJob))
s.mux.HandleFunc("POST /api/documents/{id}/processing-job/retry", s.auth(s.handleRetryProcessingJob))
// Akte-Zuordnung eines Dokuments (internal/api/akte_handlers.go). Strikt
// 1:n: Zuordnung ist nur documents.akte_id setzen/nullen.
s.mux.HandleFunc("PUT /api/documents/{id}/akte", s.auth(s.handleSetDocumentAkte))
// Freitext-Notizen pro Dokument (internal/api/document_note_handlers.go).
s.mux.HandleFunc("GET /api/documents/{id}/notes", s.auth(s.handleListDocumentNotes))
s.mux.HandleFunc("POST /api/documents/{id}/notes", s.auth(s.handleCreateDocumentNote))
s.mux.HandleFunc("DELETE /api/documents/{id}/notes/{noteId}", s.auth(s.handleDeleteDocumentNote))
// Gespeicherte Suchansichten (SavedViews, Paperless-ngx inspiriert —
// internal/api/saved_view_handlers.go). Tenant-/user-weit, nicht
// dokument-gebunden, daher eigener Block. Liste enthält eigene + geteilte
// Views; PATCH/DELETE nur durch den Ersteller.
s.mux.HandleFunc("GET /api/saved-views", s.auth(s.handleListSavedViews))
s.mux.HandleFunc("POST /api/saved-views", s.auth(s.handleCreateSavedView))
s.mux.HandleFunc("PATCH /api/saved-views/{id}", s.auth(s.handleUpdateSavedView))
s.mux.HandleFunc("DELETE /api/saved-views/{id}", s.auth(s.handleDeleteSavedView))
// Digitale Akten (digitaler Aktenordner — internal/api/akte_handlers.go).
// Strikt 1:n zu Dokumenten via documents.akte_id. Keine eigene ACL — die
// Sichtbarkeit erbt von den enthaltenen Dokumenten (GET .../{id} liefert die
// ACL-gefilterte Dokumentliste). Tenant-scoped (s.auth).
s.mux.HandleFunc("GET /api/akten", s.auth(s.handleListAkten))
s.mux.HandleFunc("POST /api/akten", s.auth(s.handleCreateAkte))
s.mux.HandleFunc("GET /api/akten/{id}", s.auth(s.handleGetAkte))
s.mux.HandleFunc("PATCH /api/akten/{id}", s.auth(s.handleUpdateAkte))
s.mux.HandleFunc("POST /api/akten/{id}/close", s.auth(s.handleCloseAkte))
s.mux.HandleFunc("DELETE /api/akten/{id}", s.auth(s.handleDeleteAkte))
// Trash + gestaffeltes Löschkonzept (internal/api/trash_handlers.go).
// DELETE /api/documents/{id} above is now a soft-delete into the trash.
s.mux.HandleFunc("GET /api/trash", s.auth(s.handleListTrash))
s.mux.HandleFunc("POST /api/trash/{id}/restore", s.auth(s.handleRestoreDocument))
s.mux.HandleFunc("GET /api/trash/{id}/delete-requests", s.auth(s.handleListDeleteRequests))
s.mux.HandleFunc("POST /api/trash/{id}/delete-requests", s.auth(s.handleCreateDeleteRequest))
s.mux.HandleFunc("DELETE /api/trash/{id}/delete-requests/{reqId}", s.auth(s.handleCancelDeleteRequest))
// Confirm executes the physical deletion -> domain_admin (User B) required.
s.mux.HandleFunc("POST /api/trash/{id}/delete-requests/{reqId}/confirm", s.authAdmin(s.handleConfirmDeleteRequest))
// Wiedervorlage (reminders)
s.mux.HandleFunc("POST /api/documents/{id}/reminders", s.auth(s.handleCreateReminder))
s.mux.HandleFunc("GET /api/reminders", s.auth(s.handleListReminders))
s.mux.HandleFunc("PATCH /api/reminders/{id}", s.auth(s.handleUpdateReminder))
s.mux.HandleFunc("DELETE /api/reminders/{id}", s.auth(s.handleDeleteReminder))
// Structured taxonomy entities (tags/document_types/correspondents)
s.mux.HandleFunc("GET /api/tags", s.auth(s.handleListTaxonomy("tags")))
s.mux.HandleFunc("POST /api/tags", s.auth(s.handleCreateTaxonomy("tags")))
s.mux.HandleFunc("PATCH /api/tags/{id}", s.auth(s.handleUpdateTaxonomy("tags")))
s.mux.HandleFunc("DELETE /api/tags/{id}", s.auth(s.handleDeleteTaxonomy("tags")))
s.mux.HandleFunc("GET /api/document-types", s.auth(s.handleListTaxonomy("document_types")))
s.mux.HandleFunc("POST /api/document-types", s.auth(s.handleCreateTaxonomy("document_types")))
s.mux.HandleFunc("PATCH /api/document-types/{id}", s.auth(s.handleUpdateTaxonomy("document_types")))
s.mux.HandleFunc("DELETE /api/document-types/{id}", s.auth(s.handleDeleteTaxonomy("document_types")))
s.mux.HandleFunc("GET /api/correspondents", s.auth(s.handleListTaxonomy("correspondents")))
s.mux.HandleFunc("POST /api/correspondents", s.auth(s.handleCreateTaxonomy("correspondents")))
s.mux.HandleFunc("PATCH /api/correspondents/{id}", s.auth(s.handleUpdateTaxonomy("correspondents")))
s.mux.HandleFunc("DELETE /api/correspondents/{id}", s.auth(s.handleDeleteTaxonomy("correspondents")))
// Manual tag attach/detach on a document
s.mux.HandleFunc("GET /api/documents/{id}/tags", s.auth(s.handleListDocumentTags))
s.mux.HandleFunc("POST /api/documents/{id}/tags/{tagId}", s.auth(s.handleAttachTag))
s.mux.HandleFunc("DELETE /api/documents/{id}/tags/{tagId}", s.auth(s.handleDetachTag))
// Custom fields (definitions, document-type assignments, document values)
s.mux.HandleFunc("GET /api/custom-fields", s.auth(s.handleListCustomFields))
s.mux.HandleFunc("POST /api/custom-fields", s.authAdmin(s.handleCreateCustomField))
s.mux.HandleFunc("PATCH /api/custom-fields/{id}", s.authAdmin(s.handleUpdateCustomField))
s.mux.HandleFunc("DELETE /api/custom-fields/{id}", s.authAdmin(s.handleDeleteCustomField))
s.mux.HandleFunc("GET /api/document-types/{id}/fields", s.auth(s.handleListDocumentTypeFields))
s.mux.HandleFunc("PUT /api/document-types/{id}/fields", s.authAdmin(s.handleSetDocumentTypeFields))
s.mux.HandleFunc("GET /api/documents/{id}/fields", s.auth(s.handleListDocumentFieldValues))
s.mux.HandleFunc("PUT /api/documents/{id}/fields", s.auth(s.handleSetDocumentFieldValues))
// Classification templates (Klassifizierungsvorlagen,
// internal/api/classification_template_handlers.go). CRUD + tag / field-
// default bulk replace are domain_admin-only (s.authAdmin); applying a
// template to a document is a normal authenticated working action (s.auth).
s.mux.HandleFunc("GET /api/classification-templates", s.auth(s.handleListTemplates))
s.mux.HandleFunc("POST /api/classification-templates", s.authAdmin(s.handleCreateTemplate))
s.mux.HandleFunc("GET /api/classification-templates/{id}", s.auth(s.handleGetTemplate))
s.mux.HandleFunc("PUT /api/classification-templates/{id}", s.authAdmin(s.handleUpdateTemplate))
s.mux.HandleFunc("DELETE /api/classification-templates/{id}", s.authAdmin(s.handleDeleteTemplate))
s.mux.HandleFunc("PUT /api/classification-templates/{id}/tags", s.authAdmin(s.handleSetTemplateTags))
s.mux.HandleFunc("PUT /api/classification-templates/{id}/field-defaults", s.authAdmin(s.handleSetTemplateFieldDefaults))
s.mux.HandleFunc("POST /api/documents/{id}/apply-template", s.auth(s.handleApplyTemplate))
// GoBD-Aufbewahrungsregeln (internal/api/retention_rule_handlers.go).
// Lesen (Liste/eligible/preview) ist normale Tenant-Aktion; Anlegen/Ändern/
// Löschen ist compliance-kritisch und erfordert domain_admin.
s.mux.HandleFunc("GET /api/retention-rules", s.auth(s.handleListRetentionRules))
s.mux.HandleFunc("POST /api/retention-rules", s.authAdmin(s.handleCreateRetentionRule))
s.mux.HandleFunc("GET /api/retention-rules/eligible", s.auth(s.handleListEligibleForDisposition))
s.mux.HandleFunc("GET /api/retention-rules/preview", s.auth(s.handlePreviewRetentionRules))
s.mux.HandleFunc("PATCH /api/retention-rules/{id}", s.authAdmin(s.handleUpdateRetentionRule))
s.mux.HandleFunc("DELETE /api/retention-rules/{id}", s.authAdmin(s.handleDeleteRetentionRule))
// GoBD-Verfahrensdokumentation als Markdown-Entwurf
// (internal/api/compliance_handlers.go). domain_admin+ für den eigenen
// Mandanten; superadmin darf per ?tenant_id=N einen fremden Mandanten
// exportieren (Prüfung im Handler).
s.mux.HandleFunc("GET /api/compliance/procedure-documentation", s.authAdmin(s.handleProcedureDocumentation))
// Workflows / Consumption-Regeln (internal/api/workflow_handlers.go).
// Administration (CRUD + action bulk replace) is domain_admin-only
// (s.authAdmin); the dry-run test and the runs overview are normal
// authenticated tenant actions (s.auth). Automatic on_upload execution is
// wired into the upload pipeline (storeUploadedFile), not exposed as a route.
s.mux.HandleFunc("GET /api/workflows", s.auth(s.handleListWorkflows))
s.mux.HandleFunc("POST /api/workflows", s.authAdmin(s.handleCreateWorkflow))
s.mux.HandleFunc("GET /api/workflows/{id}", s.auth(s.handleGetWorkflow))
s.mux.HandleFunc("PUT /api/workflows/{id}", s.authAdmin(s.handleUpdateWorkflow))
s.mux.HandleFunc("DELETE /api/workflows/{id}", s.authAdmin(s.handleDeleteWorkflow))
s.mux.HandleFunc("PUT /api/workflows/{id}/actions", s.authAdmin(s.handleSetWorkflowActions))
s.mux.HandleFunc("POST /api/workflows/{id}/test", s.auth(s.handleTestWorkflow))
s.mux.HandleFunc("GET /api/workflows/{id}/runs", s.auth(s.handleListWorkflowRuns))
// Heuristische Metadaten-Vorschläge (internal/api/metadata_suggestion_handlers.go).
// All three are normal authenticated tenant actions; nothing here applies a
// suggestion — accepting a suggested field goes through the normal edit
// endpoints (PATCH title, tag-attach, ...).
s.mux.HandleFunc("POST /api/documents/{id}/suggest-metadata", s.auth(s.handleGenerateSuggestions))
s.mux.HandleFunc("GET /api/documents/{id}/suggest-metadata", s.auth(s.handleGetLatestSuggestion))
s.mux.HandleFunc("POST /api/documents/{id}/suggest-metadata/{suggestionId}/reviewed", s.auth(s.handleMarkSuggestionReviewed))
// Permission model (group-resolved document ACL, internal/api/permission_handlers.go).
// Group + grant administration is domain_admin-only (s.authAdmin).
s.mux.HandleFunc("GET /api/permission-groups", s.authAdmin(s.handleListPermissionGroups))
s.mux.HandleFunc("POST /api/permission-groups", s.authAdmin(s.handleCreatePermissionGroup))
s.mux.HandleFunc("DELETE /api/permission-groups/{id}", s.authAdmin(s.handleDeletePermissionGroup))
s.mux.HandleFunc("GET /api/permission-groups/{id}/members", s.authAdmin(s.handleListGroupMembers))
s.mux.HandleFunc("POST /api/permission-groups/{id}/members", s.authAdmin(s.handleAddGroupMember))
s.mux.HandleFunc("DELETE /api/permission-groups/{id}/members/{userId}", s.authAdmin(s.handleRemoveGroupMember))
s.mux.HandleFunc("GET /api/document-types/{id}/grants", s.authAdmin(s.handleListDocumentTypeGrants))
s.mux.HandleFunc("POST /api/document-types/{id}/grants", s.authAdmin(s.handleSetDocumentTypeGrant))
s.mux.HandleFunc("DELETE /api/document-types/{id}/grants", s.authAdmin(s.handleDeleteDocumentTypeGrant))
s.mux.HandleFunc("GET /api/tags/{id}/grants", s.authAdmin(s.handleListTagGrants))
s.mux.HandleFunc("POST /api/tags/{id}/grants", s.authAdmin(s.handleSetTagGrant))
s.mux.HandleFunc("DELETE /api/tags/{id}/grants", s.authAdmin(s.handleDeleteTagGrant))
s.mux.HandleFunc("GET /api/documents/{id}/grants", s.authAdmin(s.handleListDocumentGrants))
s.mux.HandleFunc("POST /api/documents/{id}/grants", s.authAdmin(s.handleSetDocumentGrant))
s.mux.HandleFunc("DELETE /api/documents/{id}/grants", s.authAdmin(s.handleDeleteDocumentGrant))
// External share-links (internal/api/share_handlers.go). Create/list/revoke
// are authenticated + tenant-scoped; the tenant-wide overview is domain_admin.
s.mux.HandleFunc("POST /api/documents/{id}/shares", s.auth(s.handleCreateShare))
s.mux.HandleFunc("GET /api/documents/{id}/shares", s.auth(s.handleListDocumentShares))
s.mux.HandleFunc("DELETE /api/shares/{share_id}", s.auth(s.handleRevokeShare))
s.mux.HandleFunc("GET /api/shares", s.authAdmin(s.handleListTenantShares))
// Public share endpoints (internal/api/public_share_handlers.go) — served
// WITHOUT the s.auth wrapper by design: the share token is the credential.
// Lookup is always by token_hash, rate-limited per IP, every attempt logged.
s.mux.HandleFunc("GET /public/share/{token}", s.handlePublicShareMeta)
s.mux.HandleFunc("POST /public/share/{token}/download", s.handlePublicShareDownload)
// Buchhaltungs-Pull-API (internal/api/accounting_handlers.go).
// Key administration runs on the normal session auth and is domain_admin-only
// (a key grants tenant-wide read access to archived documents).
s.mux.HandleFunc("POST /api/accounting/api-keys", s.authAdmin(s.handleCreateAccountingAPIKey))
s.mux.HandleFunc("GET /api/accounting/api-keys", s.authAdmin(s.handleListAccountingAPIKeys))
s.mux.HandleFunc("DELETE /api/accounting/api-keys/{id}", s.authAdmin(s.handleRevokeAccountingAPIKey))
// The pull endpoints themselves are served WITHOUT s.auth by design: the
// Authorization: Bearer <key> API key is the credential, and the tenant id
// comes exclusively from resolving that key (s.accountingAuth) — never from
// a query parameter.
s.mux.HandleFunc("GET /api/v1/accounting/documents", s.accountingAuth(s.handleAccountingListDocuments))
s.mux.HandleFunc("GET /api/v1/accounting/documents/{id}/file", s.accountingAuth(s.handleAccountingDocumentFile))
// Per-tenant LDAP directory config (internal/api/ldap_handlers.go).
// domain_admin manages its own tenant; superadmin may target any tenant
// via ?tenant_id=. The bind password is never returned.
s.mux.HandleFunc("GET /api/ldap-config", s.authAdmin(s.handleGetLDAPConfig))
s.mux.HandleFunc("PUT /api/ldap-config", s.authAdmin(s.handleUpsertLDAPConfig))
// Per-tenant settings (internal/api/tenant_settings_handlers.go).
// domain_admin manages its own tenant; superadmin may target any tenant
// via ?tenant_id=. Currently: the placeholder-title date format.
s.mux.HandleFunc("GET /api/tenant-settings", s.authAdmin(s.handleGetTenantSettings))
s.mux.HandleFunc("PUT /api/tenant-settings", s.authAdmin(s.handleUpdateTenantSettings))
// Per-tenant external-Ollama connection config (internal/api/ollama_config_handlers.go).
// domain_admin manages its own tenant; superadmin may target any tenant via
// ?tenant_id=. Gates the optional 'ollama' metadata-suggestion provider.
s.mux.HandleFunc("GET /api/ollama-config", s.authAdmin(s.handleGetOllamaConfig))
s.mux.HandleFunc("PUT /api/ollama-config", s.authAdmin(s.handleUpsertOllamaConfig))
s.mux.HandleFunc("GET /api/ollama-config/models", s.authAdmin(s.handleListOllamaModels))
// SFTP credentials (embedded per-tenant SFTP server, internal/sftpserver)
s.mux.HandleFunc("POST /api/admin/sftp-credentials", s.authAdmin(s.handleCreateSFTPCredential))
s.mux.HandleFunc("GET /api/admin/sftp-credentials", s.authAdmin(s.handleListSFTPCredentials))
s.mux.HandleFunc("DELETE /api/admin/sftp-credentials/{id}", s.authAdmin(s.handleRevokeSFTPCredential))
}
// ServeHTTP implements http.Handler.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.mux.ServeHTTP(w, r)
}
// --- system handlers ---
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}
func (s *Server) handleVersion(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"version": s.appVersion})
}
// --- middleware ---
const sessionCookieName = "archivdms_session"
func (s *Server) authMiddleware(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
token := ""
if c, err := r.Cookie(sessionCookieName); err == nil {
token = c.Value
}
if token == "" {
token = extractBearerToken(r)
}
if token == "" {
writeError(w, http.StatusUnauthorized, "missing authorization")
return
}
sess, err := s.authMgr.ValidateToken(token)
if err != nil {
writeError(w, http.StatusUnauthorized, "invalid or expired token")
return
}
ctx := context.WithValue(r.Context(), sessionKey, sess)
next(w, r.WithContext(ctx))
}
}
func (s *Server) requireRole(role string, next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess == nil || !auth.HasRole(sess.Role, role) {
writeError(w, http.StatusForbidden, "insufficient permissions")
return
}
next(w, r)
}
}
// --- helpers ---
func writeJSON(w http.ResponseWriter, code int, v interface{}) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(code)
json.NewEncoder(w).Encode(v)
}
func writeError(w http.ResponseWriter, code int, msg string) {
writeJSON(w, code, map[string]string{"error": msg})
}
func extractBearerToken(r *http.Request) string {
h := r.Header.Get("Authorization")
if strings.HasPrefix(h, "Bearer ") {
return strings.TrimPrefix(h, "Bearer ")
}
return ""
}
func sessionFromCtx(ctx context.Context) *auth.Session {
v := ctx.Value(sessionKey)
if v == nil {
return &auth.Session{}
}
if s, ok := v.(*auth.Session); ok {
return s
}
return &auth.Session{}
}
// tenantMiddleware extracts the tenant_id from the session and stores it in
// the request context, making it available to all downstream handlers.
func (s *Server) tenantMiddleware(next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
session := sessionFromCtx(r.Context())
if session != nil && session.TenantID != nil {
ctx := context.WithValue(r.Context(), tenantKey, session.TenantID)
next(w, r.WithContext(ctx))
return
}
next(w, r)
}
}
// tenantFromCtx extracts the tenant_id from context. Returns nil for a
// global (superadmin/tenant-less) context.
func tenantFromCtx(ctx context.Context) *int64 {
v, _ := ctx.Value(tenantKey).(*int64)
return v
}
// remoteIP returns the real client IP. X-Forwarded-For is only trusted when
// the direct connection comes from a configured trusted proxy.
func (s *Server) remoteIP(r *http.Request) string {
directIP, _, _ := net.SplitHostPort(r.RemoteAddr)
if directIP == "" {
directIP = r.RemoteAddr
}
if len(s.cfg.TrustedProxies) > 0 && isTrustedProxy(directIP, s.cfg.TrustedProxies) {
if fwd := r.Header.Get("X-Forwarded-For"); fwd != "" {
return strings.TrimSpace(strings.Split(fwd, ",")[0])
}
}
return directIP
}
func isTrustedProxy(ip string, proxies []string) bool {
parsed := net.ParseIP(ip)
for _, p := range proxies {
if strings.Contains(p, "/") {
_, cidr, err := net.ParseCIDR(p)
if err == nil && cidr.Contains(parsed) {
return true
}
} else if p == ip {
return true
}
}
return false
}
+116
View File
@@ -0,0 +1,116 @@
package api
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// --- Admin CRUD for per-tenant SFTP credentials (internal/sftpserver) ---
//
// Registered behind s.authAdmin (domain_admin/superadmin), tenant-scoped the
// same way handleListUsers/handleCreateUser are: sess.TenantID (from the
// domain_admin's own session) picks the tenant for domain admins, while a
// tenant-less (superadmin) session must specify tenant_id explicitly on
// create and cannot list/revoke without one — SFTP credentials always
// belong to exactly one tenant's inbox.
type createSFTPCredentialRequest struct {
Username string `json:"username"`
TenantID *int64 `json:"tenant_id,omitempty"` // required only for tenant-less (superadmin) sessions
}
type sftpCredentialCreatedResponse struct {
storage.SFTPCredential
Password string `json:"password"` // returned exactly once, in this response only
}
func (s *Server) handleCreateSFTPCredential(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
var req createSFTPCredentialRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Username == "" {
writeError(w, http.StatusBadRequest, "username is required")
return
}
tenantID := sess.TenantID
if tenantID == nil {
tenantID = req.TenantID
}
if tenantID == nil {
writeError(w, http.StatusBadRequest, "tenant_id is required for tenant-less sessions")
return
}
password, err := randomSFTPPassword()
if err != nil {
writeError(w, http.StatusInternalServerError, "generate password failed")
return
}
cred, err := s.store.CreateSFTPCredential(r.Context(), *tenantID, req.Username, password)
if err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventSFTPCredentialCreate, Username: sess.Username, TenantID: tenantID, Success: false, Detail: err.Error()})
writeError(w, http.StatusBadRequest, "create sftp credential failed (username may already be taken)")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventSFTPCredentialCreate, Username: sess.Username, TenantID: tenantID,
Success: true, Detail: "sftp_username:" + cred.Username,
})
writeJSON(w, http.StatusCreated, sftpCredentialCreatedResponse{SFTPCredential: *cred, Password: password})
}
func (s *Server) handleListSFTPCredentials(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
creds, err := s.store.ListSFTPCredentials(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list sftp credentials failed")
return
}
writeJSON(w, http.StatusOK, creds)
}
func (s *Server) handleRevokeSFTPCredential(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid credential id")
return
}
if err := s.store.RevokeSFTPCredential(r.Context(), id, *sess.TenantID); err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventSFTPCredentialRevoke, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, http.StatusNotFound, "sftp credential not found")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventSFTPCredentialRevoke, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, map[string]string{"status": "revoked"})
}
func randomSFTPPassword() (string, error) {
b := make([]byte, 20)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
+159
View File
@@ -0,0 +1,159 @@
// Authenticated share-link handlers (see internal/storage/shares.go and the
// public counterpart in public_share_handlers.go):
//
// POST /api/documents/{id}/shares create a share (expires_at required)
// GET /api/documents/{id}/shares list shares for a document
// DELETE /api/shares/{share_id} revoke a share (soft, never hard-delete)
// GET /api/shares all shares of the tenant (domain_admin+)
//
// Ownership is enforced in the store layer (document/share id + tenant_id), the
// same IDOR guard used by the other document endpoints. Every create/revoke is
// audit-logged (EventShareCreated/EventShareRevoked), including failures. The
// raw token is returned exactly once, in the create response.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"time"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// createShareRequest is the POST body for creating a share. ExpiresAt is
// mandatory (no unbounded shares); MaxAccesses and Password are optional.
type createShareRequest struct {
ExpiresAt time.Time `json:"expires_at"`
MaxAccesses *int `json:"max_accesses,omitempty"`
Password string `json:"password,omitempty"`
}
// createShareResponse embeds the stored share plus the one-time plaintext
// token (only ever returned here).
type createShareResponse struct {
storage.DocumentShare
Token string `json:"token"`
}
func (s *Server) logShare(r *http.Request, event string, tenantID *int64, username, detail string, ok bool) {
s.audlog.Log(audit.Entry{
EventType: event, Username: username, TenantID: tenantID,
IPAddress: s.remoteIP(r), Success: ok, Detail: detail,
})
}
// handleCreateShare handles POST /api/documents/{id}/shares.
func (s *Server) handleCreateShare(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
var req createShareRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.ExpiresAt.IsZero() {
writeError(w, http.StatusBadRequest, "expires_at is required")
return
}
if !req.ExpiresAt.After(time.Now()) {
writeError(w, http.StatusBadRequest, "expires_at must be in the future")
return
}
if req.MaxAccesses != nil && *req.MaxAccesses < 1 {
writeError(w, http.StatusBadRequest, "max_accesses must be at least 1")
return
}
share, token, err := s.store.CreateShare(r.Context(), storage.CreateShareRequest{
TenantID: *sess.TenantID,
DocumentID: docID,
CreatedBy: sess.UserID,
ExpiresAt: req.ExpiresAt,
MaxAccesses: req.MaxAccesses,
Password: req.Password,
})
if err != nil {
s.logShare(r, audit.EventShareCreated, sess.TenantID, sess.Username, "share_create doc:"+strconv.FormatInt(docID, 10)+" err:"+err.Error(), false)
writeError(w, shareStatus(err), "create share failed")
return
}
s.logShare(r, audit.EventShareCreated, sess.TenantID, sess.Username, "share_create doc:"+strconv.FormatInt(docID, 10)+" share:"+strconv.FormatInt(share.ID, 10), true)
writeJSON(w, http.StatusCreated, createShareResponse{DocumentShare: *share, Token: token})
}
// handleListDocumentShares handles GET /api/documents/{id}/shares.
func (s *Server) handleListDocumentShares(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
shares, err := s.store.ListSharesForDocument(r.Context(), docID, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list shares failed")
return
}
writeJSON(w, http.StatusOK, shares)
}
// handleRevokeShare handles DELETE /api/shares/{share_id}.
func (s *Server) handleRevokeShare(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
shareID, err := strconv.ParseInt(r.PathValue("share_id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid share id")
return
}
if err := s.store.RevokeShare(r.Context(), shareID, *sess.TenantID, sess.UserID); err != nil {
s.logShare(r, audit.EventShareRevoked, sess.TenantID, sess.Username, "share_revoke share:"+strconv.FormatInt(shareID, 10)+" err:"+err.Error(), false)
writeError(w, shareStatus(err), "revoke share failed")
return
}
s.logShare(r, audit.EventShareRevoked, sess.TenantID, sess.Username, "share_revoke share:"+strconv.FormatInt(shareID, 10), true)
writeJSON(w, http.StatusOK, map[string]string{"status": "revoked"})
}
// handleListTenantShares handles GET /api/shares (domain_admin+): every share
// of the caller's tenant, document title joined in.
func (s *Server) handleListTenantShares(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
shares, err := s.store.ListSharesForTenant(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list shares failed")
return
}
writeJSON(w, http.StatusOK, shares)
}
// shareStatus maps store errors to an HTTP status for the authenticated
// endpoints.
func shareStatus(err error) int {
if errors.Is(err, storage.ErrShareNotFound) {
return http.StatusNotFound
}
return http.StatusInternalServerError
}
+291
View File
@@ -0,0 +1,291 @@
// Structured-entity HTTP handlers for tags/document_types/correspondents
// (see internal/storage/taxonomy.go) plus manual tag attach/detach:
//
// GET/POST /api/tags PATCH/DELETE /api/tags/{id}
// GET/POST /api/document-types PATCH/DELETE /api/document-types/{id}
// GET/POST /api/correspondents PATCH/DELETE /api/correspondents/{id}
// POST/DELETE /api/documents/{id}/tags/{tagId}
//
// All routes require s.auth(...) (authenticated + tenant context).
// Ownership is enforced in the store layer (id+tenant_id), analogous to
// internal/api/reminder_handlers.go. Every mutation is audit-logged,
// including failures.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
type taxonomyEntityRequest struct {
Name string `json:"name"`
Color string `json:"color"`
MatchAlgorithm string `json:"match_algorithm"`
MatchPattern string `json:"match_pattern"`
CaseSensitive bool `json:"case_sensitive"`
BarcodeValue string `json:"barcode_value"`
}
func taxonomyEventType(kind string) string {
switch kind {
case "tags":
return "tag"
case "document_types":
return "document_type"
case "correspondents":
return "correspondent"
default:
return kind
}
}
// handleListTaxonomy handles GET /api/{tags,document-types,correspondents}.
func (s *Server) handleListTaxonomy(kind string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
entities, err := s.store.ListTaxonomyEntities(r.Context(), kind, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list "+kind+" failed")
return
}
writeJSON(w, http.StatusOK, entities)
}
}
// handleCreateTaxonomy handles POST /api/{tags,document-types,correspondents}.
func (s *Server) handleCreateTaxonomy(kind string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req taxonomyEntityRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
entity, err := s.store.CreateTaxonomyEntity(r.Context(), kind, *sess.TenantID, storage.TaxonomyEntityRequest{
Name: req.Name, Color: req.Color, MatchAlgorithm: req.MatchAlgorithm,
MatchPattern: req.MatchPattern, CaseSensitive: req.CaseSensitive, BarcodeValue: req.BarcodeValue,
})
if err != nil {
status := http.StatusInternalServerError
if errors.Is(err, storage.ErrDuplicateTaxonomyName) {
status = http.StatusConflict
}
s.audlog.Log(audit.Entry{EventType: taxonomyEventType(kind) + "_create", Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: err.Error()})
writeError(w, status, "create "+kind+" failed")
return
}
s.audlog.Log(audit.Entry{
EventType: taxonomyEventType(kind) + "_create", Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "id:" + strconv.FormatInt(entity.ID, 10) + " name:" + entity.Name,
})
writeJSON(w, http.StatusCreated, entity)
}
}
// handleUpdateTaxonomy handles PATCH /api/{tags,document-types,correspondents}/{id}.
func (s *Server) handleUpdateTaxonomy(kind string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req taxonomyEntityRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
entity, err := s.store.UpdateTaxonomyEntity(r.Context(), kind, id, *sess.TenantID, storage.TaxonomyEntityRequest{
Name: req.Name, Color: req.Color, MatchAlgorithm: req.MatchAlgorithm,
MatchPattern: req.MatchPattern, CaseSensitive: req.CaseSensitive, BarcodeValue: req.BarcodeValue,
})
if err != nil {
status := http.StatusNotFound
if errors.Is(err, storage.ErrDuplicateTaxonomyName) {
status = http.StatusConflict
}
s.audlog.Log(audit.Entry{
EventType: taxonomyEventType(kind) + "_update", Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(),
})
writeError(w, status, "update "+kind+" failed")
return
}
s.audlog.Log(audit.Entry{
EventType: taxonomyEventType(kind) + "_update", Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "id:" + strconv.FormatInt(entity.ID, 10),
})
writeJSON(w, http.StatusOK, entity)
}
}
// handleDeleteTaxonomy handles DELETE /api/{tags,document-types,correspondents}/{id}.
func (s *Server) handleDeleteTaxonomy(kind string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.DeleteTaxonomyEntity(r.Context(), kind, id, *sess.TenantID); err != nil {
s.audlog.Log(audit.Entry{
EventType: taxonomyEventType(kind) + "_delete", Username: sess.Username, TenantID: sess.TenantID,
Success: false, Detail: "id:" + strconv.FormatInt(id, 10) + " err:" + err.Error(),
})
writeError(w, http.StatusNotFound, "delete "+kind+" failed")
return
}
s.audlog.Log(audit.Entry{
EventType: taxonomyEventType(kind) + "_delete", Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
}
// handleAttachTag handles POST /api/documents/{id}/tags/{tagId} (manual
// tag attach). Verifies both the document and the tag belong to the
// caller's tenant before inserting the document_tags row.
func (s *Server) handleAttachTag(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
tagID, err := strconv.ParseInt(r.PathValue("tagId"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid tag id")
return
}
if _, err := s.store.GetDocument(r.Context(), docID, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
tags, err := s.store.ListTaxonomyEntities(r.Context(), "tags", *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "attach tag failed")
return
}
found := false
for _, t := range tags {
if t.ID == tagID {
found = true
break
}
}
if !found {
writeError(w, http.StatusNotFound, "tag not found")
return
}
if err := s.store.AttachTag(r.Context(), docID, tagID); err != nil {
s.audlog.Log(audit.Entry{
EventType: "tag_attach", Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: false, Detail: err.Error(),
})
writeError(w, http.StatusInternalServerError, "attach tag failed")
return
}
s.audlog.Log(audit.Entry{
EventType: "tag_attach", Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: true, Detail: "tag_id:" + strconv.FormatInt(tagID, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "attached"})
}
// handleDetachTag handles DELETE /api/documents/{id}/tags/{tagId}.
func (s *Server) handleDetachTag(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
tagID, err := strconv.ParseInt(r.PathValue("tagId"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid tag id")
return
}
if _, err := s.store.GetDocument(r.Context(), docID, *sess.TenantID); err != nil {
writeError(w, http.StatusNotFound, "document not found")
return
}
if err := s.store.DetachTag(r.Context(), docID, tagID); err != nil {
s.audlog.Log(audit.Entry{
EventType: "tag_detach", Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: false, Detail: err.Error(),
})
writeError(w, http.StatusInternalServerError, "detach tag failed")
return
}
s.audlog.Log(audit.Entry{
EventType: "tag_detach", Username: sess.Username, TenantID: sess.TenantID,
DocumentID: strconv.FormatInt(docID, 10), Success: true, Detail: "tag_id:" + strconv.FormatInt(tagID, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "detached"})
}
// handleListDocumentTags handles GET /api/documents/{id}/tags.
func (s *Server) handleListDocumentTags(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docID, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
tags, err := s.store.ListDocumentTags(r.Context(), docID, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list document tags failed")
return
}
writeJSON(w, http.StatusOK, tags)
}
+50
View File
@@ -0,0 +1,50 @@
package api
import (
"encoding/json"
"net/http"
"archivdms/internal/audit"
)
type createTenantRequest struct {
Name string `json:"name"`
Slug string `json:"slug"`
Domain string `json:"domain"`
}
// handleCreateTenant creates a new tenant. Restricted to superadmin via
// requireRole in server.go's route registration.
func (s *Server) handleCreateTenant(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
var req createTenantRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" || req.Slug == "" {
writeError(w, http.StatusBadRequest, "name and slug are required")
return
}
tenant, err := s.tenantStore.Create(r.Context(), req.Name, req.Slug, req.Domain)
if err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventTenantMgmt, Username: sess.Username, Success: false, Detail: "create_tenant_failed"})
writeError(w, http.StatusBadRequest, "create tenant failed")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventTenantMgmt, Username: sess.Username, Success: true, Detail: "tenant_created:" + tenant.Slug})
writeJSON(w, http.StatusCreated, tenant)
}
// handleListTenants lists all tenants. Restricted to superadmin via
// requireRole in server.go's route registration.
func (s *Server) handleListTenants(w http.ResponseWriter, r *http.Request) {
tenants, err := s.tenantStore.List(r.Context())
if err != nil {
writeError(w, http.StatusInternalServerError, "list tenants failed")
return
}
writeJSON(w, http.StatusOK, tenants)
}
+233
View File
@@ -0,0 +1,233 @@
package api
import (
"encoding/json"
"net/http"
"strconv"
"strings"
"archivdms/internal/audit"
"archivdms/internal/dateformat"
"archivdms/internal/storage"
"archivdms/internal/userstore"
)
// resolveTenantSettingsTenant determines which tenant the request targets,
// mirroring resolveLDAPTenant: domain_admin is pinned to its own signed
// session tenant (IDOR-safe, query params ignored); superadmin (no session
// tenant) must pass ?tenant_id=. Returns the tenant ID and false when the
// request is not authorised or the tenant cannot be determined (the caller
// has already written the response).
func (s *Server) resolveTenantSettingsTenant(w http.ResponseWriter, r *http.Request) (int64, bool) {
sess := sessionFromCtx(r.Context())
if sess.Role == userstore.RoleSuperAdmin {
q := r.URL.Query().Get("tenant_id")
if q == "" {
writeError(w, http.StatusBadRequest, "tenant_id query parameter required for superadmin")
return 0, false
}
tid, err := strconv.ParseInt(q, 10, 64)
if err != nil || tid <= 0 {
writeError(w, http.StatusBadRequest, "invalid tenant_id")
return 0, false
}
return tid, true
}
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "no tenant context")
return 0, false
}
return *sess.TenantID, true
}
// availableScanTitleFormats returns the example token patterns the frontend can
// offer as clickable starting points. These are suggestions only — any free
// token pattern is accepted (see dateformat.Translate) — not a validation
// constraint. Returned as a fresh slice so callers never share backing storage.
func availableScanTitleFormats() []string {
out := make([]string, len(exampleScanTitleFormats))
copy(out, exampleScanTitleFormats)
return out
}
type tenantSettingsResponse struct {
ScanTitleDateFormat string `json:"scan_title_date_format"`
ScanTitlePrefix string `json:"scan_title_prefix"`
// DefaultTitleTemplate is the tenant-wide fallback title template (Go
// text/template) used when an applied classification template carries no
// own title_template. Empty string means "no tenant default".
DefaultTitleTemplate string `json:"default_title_template"`
// AvailableFormats is a list of example patterns (suggestions), not an
// allow-list; the field name is kept for frontend compatibility.
AvailableFormats []string `json:"available_formats"`
}
// effectiveScanTitlePrefix returns the tenant's stored prefix, or the default
// for empty/legacy rows, so the response always reflects what would actually be
// used at upload time.
func effectiveScanTitlePrefix(stored string) string {
if p := strings.TrimSpace(stored); p != "" {
return p
}
return defaultScanTitlePrefix
}
// effectiveScanTitleFormat returns the tenant's stored token pattern, or the
// default for empty/invalid rows, so the response always reflects the pattern
// that would actually be used at upload time.
func effectiveScanTitleFormat(stored string) string {
if s := strings.TrimSpace(stored); s != "" {
if _, err := dateformat.Translate(s); err == nil {
return s
}
}
return defaultScanTitleDateFormat
}
// handleGetTenantSettings returns the tenant's own settings (currently the
// placeholder-title date format) plus a list of example format patterns
// (suggestions for the frontend, not an allow-list).
func (s *Server) handleGetTenantSettings(w http.ResponseWriter, r *http.Request) {
if s.tenantStore == nil {
writeError(w, http.StatusServiceUnavailable, "tenant store not configured")
return
}
tenantID, ok := s.resolveTenantSettingsTenant(w, r)
if !ok {
return
}
t, err := s.tenantStore.GetByID(r.Context(), tenantID)
if err != nil || t == nil {
writeError(w, http.StatusInternalServerError, "load tenant settings failed")
return
}
writeJSON(w, http.StatusOK, tenantSettingsResponse{
ScanTitleDateFormat: effectiveScanTitleFormat(t.ScanTitleDateFormat),
ScanTitlePrefix: effectiveScanTitlePrefix(t.ScanTitlePrefix),
DefaultTitleTemplate: t.DefaultTitleTemplate,
AvailableFormats: availableScanTitleFormats(),
})
}
// updateTenantSettingsRequest uses pointer fields for PATCH-like semantics:
// only the fields present in the JSON body are updated, so the two settings
// can be changed independently of one another.
type updateTenantSettingsRequest struct {
ScanTitleDateFormat *string `json:"scan_title_date_format"`
ScanTitlePrefix *string `json:"scan_title_prefix"`
DefaultTitleTemplate *string `json:"default_title_template"`
}
// handleUpdateTenantSettings persists the tenant's placeholder-title date
// format. The submitted value is a free token pattern (e.g. "DD.MM.YYYY HH:mm")
// validated via dateformat.Translate; an unparseable pattern yields 400 with
// the translator's error message. Every attempt — success or failure — is
// audit-logged (GoBD).
func (s *Server) handleUpdateTenantSettings(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if s.tenantStore == nil {
writeError(w, http.StatusServiceUnavailable, "tenant store not configured")
return
}
tenantID, ok := s.resolveTenantSettingsTenant(w, r)
if !ok {
return
}
logFail := func(detail string) {
tid := tenantID
s.audlog.Log(audit.Entry{
EventType: audit.EventTenantMgmt, Username: sess.Username,
TenantID: &tid, Success: false, Detail: detail,
})
}
var req updateTenantSettingsRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
logFail("tenant_settings invalid_body")
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.ScanTitleDateFormat == nil && req.ScanTitlePrefix == nil && req.DefaultTitleTemplate == nil {
logFail("tenant_settings no_fields")
writeError(w, http.StatusBadRequest, "no settings provided")
return
}
// Validate everything before persisting anything, so a bad prefix can never
// leave a half-applied update.
if req.ScanTitleDateFormat != nil {
if _, err := dateformat.Translate(*req.ScanTitleDateFormat); err != nil {
logFail("tenant_settings invalid_scan_title_date_format:" + *req.ScanTitleDateFormat)
writeError(w, http.StatusBadRequest, err.Error())
return
}
}
if req.ScanTitlePrefix != nil && strings.TrimSpace(*req.ScanTitlePrefix) == "" {
logFail("tenant_settings empty_scan_title_prefix")
writeError(w, http.StatusBadRequest, "scan_title_prefix must not be empty")
return
}
// An empty default_title_template is allowed (clears the tenant default);
// a non-empty one must parse as a valid Go text/template.
if req.DefaultTitleTemplate != nil {
if err := storage.ValidateTitleTemplate(*req.DefaultTitleTemplate); err != nil {
logFail("tenant_settings invalid_default_title_template")
writeError(w, http.StatusBadRequest, err.Error())
return
}
}
var detail string
if req.ScanTitleDateFormat != nil {
if err := s.tenantStore.UpdateScanTitleDateFormat(r.Context(), tenantID, *req.ScanTitleDateFormat); err != nil {
logFail("tenant_settings update_failed")
writeError(w, http.StatusInternalServerError, "save tenant settings failed")
return
}
detail += " scan_title_date_format:" + *req.ScanTitleDateFormat
}
if req.ScanTitlePrefix != nil {
if err := s.tenantStore.UpdateScanTitlePrefix(r.Context(), tenantID, *req.ScanTitlePrefix); err != nil {
// Store-level validation (length) or DB error.
logFail("tenant_settings update_failed")
writeError(w, http.StatusBadRequest, "invalid scan_title_prefix")
return
}
detail += " scan_title_prefix:" + strings.TrimSpace(*req.ScanTitlePrefix)
}
if req.DefaultTitleTemplate != nil {
if err := s.tenantStore.UpdateDefaultTitleTemplate(r.Context(), tenantID, *req.DefaultTitleTemplate); err != nil {
logFail("tenant_settings update_failed")
writeError(w, http.StatusBadRequest, "invalid default_title_template")
return
}
detail += " default_title_template:" + strings.TrimSpace(*req.DefaultTitleTemplate)
}
tid := tenantID
s.audlog.Log(audit.Entry{
EventType: audit.EventTenantMgmt, Username: sess.Username,
TenantID: &tid, Success: true,
Detail: "tenant_settings" + detail,
})
// Reload so the response reflects the effective persisted state for both
// fields, regardless of which one this request changed.
t, err := s.tenantStore.GetByID(r.Context(), tenantID)
if err != nil || t == nil {
writeError(w, http.StatusInternalServerError, "load tenant settings failed")
return
}
writeJSON(w, http.StatusOK, tenantSettingsResponse{
ScanTitleDateFormat: effectiveScanTitleFormat(t.ScanTitleDateFormat),
ScanTitlePrefix: effectiveScanTitlePrefix(t.ScanTitlePrefix),
DefaultTitleTemplate: t.DefaultTitleTemplate,
AvailableFormats: availableScanTitleFormats(),
})
}
+237
View File
@@ -0,0 +1,237 @@
// Trash + staged-deletion HTTP handlers (see internal/storage/trash.go):
//
// GET /api/trash list soft-deleted docs
// POST /api/trash/{id}/restore restore from trash
// POST /api/trash/{id}/delete-requests request final deletion (User A)
// GET /api/trash/{id}/delete-requests request status/history
// POST /api/trash/{id}/delete-requests/{reqId}/confirm confirm + execute (User B, domain_admin)
// DELETE /api/trash/{id}/delete-requests/{reqId} withdraw a pending request
//
// The soft-delete itself lives on DELETE /api/documents/{id}
// (handleDeleteDocument). Ownership is enforced in the store layer via
// id+tenant_id; the two-person rule (requester != confirmer) is enforced in
// ConfirmDeleteRequest. Every phase emits its own append-only audit entry.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"time"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// handleListTrash handles GET /api/trash.
func (s *Server) handleListTrash(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
docs, err := s.store.ListTrash(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list trash failed")
return
}
writeJSON(w, http.StatusOK, docs)
}
// handleRestoreDocument handles POST /api/trash/{id}/restore.
func (s *Server) handleRestoreDocument(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
if err := s.store.RestoreDocument(r.Context(), id, *sess.TenantID, sess.UserID); err != nil {
status := http.StatusInternalServerError
msg := "restore failed"
if errors.Is(err, storage.ErrDocumentNotInTrash) {
status = http.StatusNotFound
msg = "document not found in trash"
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentRestore, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentRestore, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true})
writeJSON(w, http.StatusOK, map[string]string{"status": "restored"})
}
// handleListDeleteRequests handles GET /api/trash/{id}/delete-requests.
func (s *Server) handleListDeleteRequests(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
reqs, err := s.store.ListDeleteRequests(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list delete requests failed")
return
}
writeJSON(w, http.StatusOK, reqs)
}
// handleCreateDeleteRequest handles POST /api/trash/{id}/delete-requests.
// User A requests final deletion; retention is re-checked, and a blocked
// attempt is still recorded (409).
func (s *Server) handleCreateDeleteRequest(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid document id")
return
}
req, err := s.store.CreateDeleteRequest(r.Context(), id, *sess.TenantID, sess.UserID)
if err != nil {
if errors.Is(err, storage.ErrRetentionActive) {
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteBlocked, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "delete request blocked by retention"})
writeError(w, http.StatusConflict, "document is under retention and cannot be deleted yet")
return
}
status := http.StatusInternalServerError
msg := "create delete request failed"
if errors.Is(err, storage.ErrDocumentNotInTrash) {
status = http.StatusNotFound
msg = "document not found in trash"
} else if errors.Is(err, storage.ErrDeleteRequestExists) {
status = http.StatusConflict
msg = "a pending delete request already exists"
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteRequest, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteRequest, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true, Detail: "request id:" + strconv.FormatInt(req.ID, 10)})
writeJSON(w, http.StatusCreated, req)
}
// handleCancelDeleteRequest handles DELETE /api/trash/{id}/delete-requests/{reqId}.
func (s *Server) handleCancelDeleteRequest(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
reqID, err2 := strconv.ParseInt(r.PathValue("reqId"), 10, 64)
if err != nil || err2 != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.CancelDeleteRequest(r.Context(), id, reqID, *sess.TenantID, sess.UserID); err != nil {
status := http.StatusInternalServerError
msg := "cancel delete request failed"
if errors.Is(err, storage.ErrDeleteRequestNotFound) {
status = http.StatusNotFound
msg = "pending delete request not found"
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteRequest, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: false, Detail: "cancel req:" + r.PathValue("reqId") + " err:" + err.Error()})
writeError(w, status, msg)
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteRequest, Username: sess.Username, TenantID: sess.TenantID, DocumentID: r.PathValue("id"), Success: true, Detail: "cancelled req:" + r.PathValue("reqId")})
writeJSON(w, http.StatusOK, map[string]string{"status": "cancelled"})
}
// handleConfirmDeleteRequest handles
// POST /api/trash/{id}/delete-requests/{reqId}/confirm (domain_admin, User B).
// The store enforces requester != confirmer and re-checks retention before it
// removes the physical WORM file and tombstones the DB row.
func (s *Server) handleConfirmDeleteRequest(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
reqID, err2 := strconv.ParseInt(r.PathValue("reqId"), 10, 64)
if err != nil || err2 != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
docIDStr := r.PathValue("id")
exec, err := s.store.ConfirmDeleteRequest(r.Context(), id, reqID, *sess.TenantID, sess.UserID)
if err != nil {
if errors.Is(err, storage.ErrRetentionActive) {
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteBlocked, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docIDStr, Success: false, Detail: "confirm blocked by retention req:" + r.PathValue("reqId")})
writeError(w, http.StatusConflict, "document is under retention and cannot be deleted yet")
return
}
status := http.StatusInternalServerError
msg := "confirm delete request failed"
if errors.Is(err, storage.ErrSelfConfirm) {
status = http.StatusForbidden
msg = "delete request must be confirmed by a different user"
} else if errors.Is(err, storage.ErrDeleteRequestNotFound) {
status = http.StatusNotFound
msg = "pending delete request not found"
} else if errors.Is(err, storage.ErrDocumentNotInTrash) {
status = http.StatusNotFound
msg = "document not found in trash"
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteConfirm, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docIDStr, Success: false, Detail: "req:" + r.PathValue("reqId") + " err:" + err.Error()})
writeError(w, status, msg)
return
}
// Two-person rule: emit a confirm entry (User B) and an execute entry that
// records the requester (User A) whose request was finally carried out.
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteConfirm, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docIDStr, Success: true, Detail: "confirmed req:" + strconv.FormatInt(exec.RequestID, 10)})
// GoBD-taugliches Löschprotokoll (ecoDMS-Muster): the execute entry carries
// a structured, self-contained record of the final, irreversible deletion —
// who requested it (User A) and when, who confirmed/executed it (User B, the
// current session) and when, which document (title + content_hash as the
// tamper-evident fingerprint of the removed WORM file), the retention state
// at execution time, and the legal basis (Vier-Augen-Prinzip + elapsed/absent
// retention). Encoded as JSON in the audit Detail field so the append-only
// audit_log (DB + JSON-Lines mirror) remains the single source of truth
// without a dedicated table.
protokoll := map[string]any{
"loeschprotokoll": true,
"document_id": docIDStr,
"title": exec.Title,
"content_hash": exec.ContentHash,
"worm_file_removed": exec.StoragePath,
"request_id": exec.RequestID,
"requested_by_user_id": exec.RequestedBy,
"requested_at": exec.RequestedAt.UTC().Format(time.RFC3339),
"confirmed_by_user_id": sess.UserID,
"confirmed_by_username": sess.Username,
"executed_at": time.Now().UTC().Format(time.RFC3339),
"rechtsgrundlage": "Vier-Augen-Prinzip erfuellt; Aufbewahrungsfrist (retain_until) abgelaufen oder nicht gesetzt",
}
if exec.RetainUntil != nil {
protokoll["retain_until"] = exec.RetainUntil.UTC().Format(time.RFC3339)
} else {
protokoll["retain_until"] = nil
}
detail := "executed req:" + strconv.FormatInt(exec.RequestID, 10) +
" requested_by_user_id:" + strconv.FormatInt(exec.RequestedBy, 10) +
" removed:" + exec.StoragePath
if b, jerr := json.Marshal(protokoll); jerr == nil {
detail = string(b)
}
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentDeleteExecute, Username: sess.Username, TenantID: sess.TenantID, DocumentID: docIDStr, Success: true, Detail: detail})
writeJSON(w, http.StatusOK, map[string]string{"status": "executed"})
}
+129
View File
@@ -0,0 +1,129 @@
package api
import (
"encoding/json"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/userstore"
)
func (s *Server) handleListUsers(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
var users []*userstore.User
var err error
if sess.TenantID != nil {
users, err = s.users.ListByTenant(r.Context(), *sess.TenantID)
} else {
users, err = s.users.List("")
}
if err != nil {
writeError(w, http.StatusInternalServerError, "list users failed")
return
}
writeJSON(w, http.StatusOK, users)
}
type createUserRequest struct {
Username string `json:"username"`
Email string `json:"email"`
Password string `json:"password"`
Role string `json:"role"`
// TenantID is only ever evaluated for a superadmin caller (see
// handleCreateUser). For any other role it is silently ignored and the
// caller's own sess.TenantID is enforced instead — this is a deliberate
// IDOR guard: a domain_admin must never be able to steer a created user
// into a tenant other than their own by sending a different tenant_id.
TenantID *int64 `json:"tenant_id"`
}
func (s *Server) handleCreateUser(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
var req createUserRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Role == "" {
req.Role = userstore.RoleUser
}
// Tenant assignment is rollenabhängig (security-critical, see plan):
// - superadmin: may set tenant_id explicitly from the request body,
// including nil for another tenant-less superadmin.
// - everyone else (domain_admin, user): tenant_id is ALWAYS hard-forced
// to the caller's own sess.TenantID; any tenant_id in the request
// body is completely ignored, not merely validated, to close the
// IDOR hole where a domain_admin could otherwise create a user in a
// tenant they don't administer.
tenantID := sess.TenantID
if sess.Role == userstore.RoleSuperAdmin {
tenantID = req.TenantID
}
user, err := s.users.Create(userstore.CreateUserRequest{
Username: req.Username,
Email: req.Email,
Password: req.Password,
Role: req.Role,
TenantID: tenantID,
})
if err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventUserMgmt, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "create_user_failed"})
writeError(w, http.StatusBadRequest, "create user failed")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventUserMgmt, Username: sess.Username, TenantID: sess.TenantID, Success: true, Detail: "user_created:" + user.Username})
writeJSON(w, http.StatusCreated, user)
}
type updateUserRequest struct {
Email *string `json:"email"`
Role *string `json:"role"`
Active *bool `json:"active"`
Password *string `json:"password"`
}
func (s *Server) handleUpdateUser(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid user id")
return
}
var req updateUserRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
user, err := s.users.Update(id, userstore.UpdateUserRequest{
Email: req.Email, Role: req.Role, Active: req.Active, Password: req.Password,
})
if err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventUserMgmt, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "update_user_failed"})
writeError(w, http.StatusBadRequest, "update user failed")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventUserMgmt, Username: sess.Username, TenantID: sess.TenantID, Success: true, Detail: "user_updated:" + user.Username})
writeJSON(w, http.StatusOK, user)
}
func (s *Server) handleDeleteUser(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid user id")
return
}
if err := s.users.Delete(id); err != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventUserMgmt, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "delete_user_failed"})
writeError(w, http.StatusBadRequest, "delete user failed")
return
}
s.audlog.Log(audit.Entry{EventType: audit.EventUserMgmt, Username: sess.Username, TenantID: sess.TenantID, Success: true, Detail: "user_deleted"})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
+327
View File
@@ -0,0 +1,327 @@
// Workflow ("Consumption-Regeln") HTTP handlers (see
// internal/storage/workflows.go):
//
// GET/POST /api/workflows GET/PUT/DELETE /api/workflows/{id}
// PUT /api/workflows/{id}/actions
// POST /api/workflows/{id}/test
// GET /api/workflows/{id}/runs
//
// Workflow administration (CRUD + action bulk replace) requires domain_admin
// (s.authAdmin, enforced in server.go). The dry-run test and the runs overview
// are normal authenticated tenant actions (s.auth). Ownership is enforced in
// the store layer (id+tenant_id). Every mutation is audit-logged, incl. failures.
package api
import (
"encoding/json"
"errors"
"net/http"
"strconv"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
type workflowRequest struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
TriggerType string `json:"trigger_type"`
ConditionTree json.RawMessage `json:"condition_tree"`
Priority int `json:"priority"`
}
// handleListWorkflows handles GET /api/workflows (without actions).
func (s *Server) handleListWorkflows(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
workflows, err := s.store.ListWorkflows(r.Context(), *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "list workflows failed")
return
}
writeJSON(w, http.StatusOK, workflows)
}
// handleGetWorkflow handles GET /api/workflows/{id} (resolved with actions).
func (s *Server) handleGetWorkflow(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
wf, err := s.store.GetWorkflow(r.Context(), id, *sess.TenantID)
if err != nil {
if errors.Is(err, storage.ErrWorkflowNotFound) {
writeError(w, http.StatusNotFound, "workflow not found")
return
}
writeError(w, http.StatusInternalServerError, "get workflow failed")
return
}
writeJSON(w, http.StatusOK, wf)
}
// handleCreateWorkflow handles POST /api/workflows (domain_admin+).
func (s *Server) handleCreateWorkflow(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
var req workflowRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
wf, err := s.store.CreateWorkflow(r.Context(), *sess.TenantID, storage.CreateWorkflowRequest{
Name: req.Name, Enabled: req.Enabled, TriggerType: req.TriggerType,
ConditionTree: req.ConditionTree, Priority: req.Priority, CreatedBy: &sess.UserID,
})
if err != nil {
status := workflowErrorStatus(err)
s.audlog.Log(audit.Entry{EventType: audit.EventWorkflowCreate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "workflow_create err:" + err.Error()})
writeError(w, status, workflowErrorMessage(err, "create workflow failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventWorkflowCreate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "workflow_create id:" + strconv.FormatInt(wf.ID, 10) + " name:" + wf.Name,
})
writeJSON(w, http.StatusCreated, wf)
}
// handleUpdateWorkflow handles PUT /api/workflows/{id} (domain_admin+).
func (s *Server) handleUpdateWorkflow(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req workflowRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.Name == "" {
writeError(w, http.StatusBadRequest, "name is required")
return
}
err = s.store.UpdateWorkflow(r.Context(), id, *sess.TenantID, storage.UpdateWorkflowRequest{
Name: req.Name, Enabled: req.Enabled, TriggerType: req.TriggerType,
ConditionTree: req.ConditionTree, Priority: req.Priority,
})
if err != nil {
status := workflowErrorStatus(err)
s.audlog.Log(audit.Entry{EventType: audit.EventWorkflowUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "workflow_update id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, workflowErrorMessage(err, "update workflow failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventWorkflowUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "workflow_update id:" + strconv.FormatInt(id, 10),
})
wf, err := s.store.GetWorkflow(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "reload workflow failed")
return
}
writeJSON(w, http.StatusOK, wf)
}
// handleDeleteWorkflow handles DELETE /api/workflows/{id} (domain_admin+).
func (s *Server) handleDeleteWorkflow(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
if err := s.store.DeleteWorkflow(r.Context(), id, *sess.TenantID); err != nil {
status := http.StatusNotFound
if !errors.Is(err, storage.ErrWorkflowNotFound) {
status = http.StatusInternalServerError
}
s.audlog.Log(audit.Entry{EventType: audit.EventWorkflowDelete, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "workflow_delete id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, "delete workflow failed")
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventWorkflowDelete, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "workflow_delete id:" + strconv.FormatInt(id, 10),
})
writeJSON(w, http.StatusOK, map[string]string{"status": "deleted"})
}
type workflowActionsRequest struct {
Actions []struct {
ActionType string `json:"action_type"`
ActionConfig json.RawMessage `json:"action_config"`
} `json:"actions"`
}
// handleSetWorkflowActions handles PUT /api/workflows/{id}/actions (bulk
// replace, domain_admin+).
func (s *Server) handleSetWorkflowActions(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req workflowActionsRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
actions := make([]storage.WorkflowActionInput, 0, len(req.Actions))
for _, a := range req.Actions {
actions = append(actions, storage.WorkflowActionInput{ActionType: a.ActionType, ActionConfig: a.ActionConfig})
}
if err := s.store.SetWorkflowActions(r.Context(), id, *sess.TenantID, actions); err != nil {
status := workflowErrorStatus(err)
s.audlog.Log(audit.Entry{EventType: audit.EventWorkflowUpdate, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "workflow_actions_set id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, workflowErrorMessage(err, "set workflow actions failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventWorkflowUpdate, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "workflow_actions_set id:" + strconv.FormatInt(id, 10) + " count:" + strconv.Itoa(len(actions)),
})
wf, err := s.store.GetWorkflow(r.Context(), id, *sess.TenantID)
if err != nil {
writeError(w, http.StatusInternalServerError, "reload workflow failed")
return
}
writeJSON(w, http.StatusOK, wf)
}
type workflowTestRequest struct {
DocumentID *int64 `json:"document_id"`
RawText string `json:"raw_text"`
}
// handleTestWorkflow handles POST /api/workflows/{id}/test. Pure dry-run: it
// reports which leaves matched and which actions WOULD run, never executing
// them. Evaluated against an existing document (document_id) or a synthetic
// document built from raw_text.
func (s *Server) handleTestWorkflow(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req workflowTestRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
res, err := s.store.TestWorkflow(r.Context(), id, *sess.TenantID, req.DocumentID, req.RawText)
if err != nil {
status := workflowErrorStatus(err)
if errors.Is(err, storage.ErrDocumentNotFound) {
status = http.StatusNotFound
}
s.audlog.Log(audit.Entry{EventType: audit.EventWorkflowRun, Username: sess.Username, TenantID: sess.TenantID, Success: false, Detail: "workflow_test id:" + strconv.FormatInt(id, 10) + " err:" + err.Error()})
writeError(w, status, workflowErrorMessage(err, "test workflow failed"))
return
}
s.audlog.Log(audit.Entry{
EventType: audit.EventWorkflowRun, Username: sess.Username, TenantID: sess.TenantID,
Success: true, Detail: "workflow_test id:" + strconv.FormatInt(id, 10) + " matched:" + strconv.FormatBool(res.Matched),
})
writeJSON(w, http.StatusOK, res)
}
// handleListWorkflowRuns handles GET /api/workflows/{id}/runs (optional
// ?limit=).
func (s *Server) handleListWorkflowRuns(w http.ResponseWriter, r *http.Request) {
sess := sessionFromCtx(r.Context())
if sess.TenantID == nil {
writeError(w, http.StatusForbidden, "tenant context required")
return
}
id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
limit := 0
if raw := r.URL.Query().Get("limit"); raw != "" {
n, err := strconv.Atoi(raw)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid limit")
return
}
limit = n
}
runs, err := s.store.ListWorkflowRuns(r.Context(), id, *sess.TenantID, limit)
if err != nil {
if errors.Is(err, storage.ErrWorkflowNotFound) {
writeError(w, http.StatusNotFound, "workflow not found")
return
}
writeError(w, http.StatusInternalServerError, "list workflow runs failed")
return
}
writeJSON(w, http.StatusOK, runs)
}
// workflowErrorStatus maps store errors from the workflow write path to HTTP
// status codes.
func workflowErrorStatus(err error) int {
switch {
case errors.Is(err, storage.ErrWorkflowNotFound):
return http.StatusNotFound
case errors.Is(err, storage.ErrDuplicateWorkflowName):
return http.StatusConflict
case errors.Is(err, storage.ErrInvalidConditionTree), errors.Is(err, storage.ErrInvalidWorkflowAction):
return http.StatusBadRequest
default:
return http.StatusInternalServerError
}
}
// workflowErrorMessage returns the store error's message for the client on the
// validation cases (safe, user-actionable), otherwise the generic fallback.
func workflowErrorMessage(err error, fallback string) string {
switch {
case errors.Is(err, storage.ErrInvalidConditionTree), errors.Is(err, storage.ErrInvalidWorkflowAction):
return err.Error()
case errors.Is(err, storage.ErrWorkflowNotFound):
return "workflow not found"
case errors.Is(err, storage.ErrDuplicateWorkflowName):
return "workflow with this name already exists"
default:
return fallback
}
}
+507
View File
@@ -0,0 +1,507 @@
// Package audit is a PostgreSQL-backed, append-only audit log, ported 1:1
// from archivmail's internal/audit pattern (including the DB-level
// immutability trigger and the tamper-evident JSON-Lines mirror file), with
// the mail-specific fields (mail_id, query) replaced by a generic document_id
// so the log fits archivdms's document-centric core model.
package audit
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
// Event type constants.
const (
EventLogin = "login"
EventLogout = "logout"
EventUserMgmt = "user_mgmt"
EventTenantMgmt = "tenant_mgmt"
// EventDocumentCreate/Update/Delete cover document lifecycle changes.
EventDocumentCreate = "document_create"
EventDocumentUpdate = "document_update"
EventDocumentDelete = "document_delete"
// EventDocumentReprocessed records a re-processing run on an already-archived
// document (POST /api/documents/{id}/reprocess): OCR is re-run on the stored
// WORM file and ocr_text refreshed, followed by best-effort auto-assignment
// and on_upload workflows. The file itself stays untouched (WORM).
EventDocumentReprocessed = "document_reprocessed"
// EventDocumentProcessed records the asynchronous post-upload processing
// run of the tenant job queue (internal/jobqueue -> api.ProcessDocumentJob):
// OCR extraction, title/belegdatum derivation, taxonomy auto-assignment and
// on_upload workflows on an already-staged, already-archived document.
// Logged on success AND failure (retry/backoff attempts included) so the
// GoBD trail covers the whole ingest chain, not just the synchronous
// staging step (EventDocumentCreate). The WORM file is never touched.
EventDocumentProcessed = "document_processed"
// Manual taxonomy assignment on a single document (doc_type/correspondent).
// Auto-assignment during upload is covered by EventDocumentCreate.
EventDocTypeSet = "document_doctype_set"
EventCorrespondentSet = "document_correspondent_set"
// Trash / staged deletion workflow (Papierkorb + Vier-Augen-Prinzip).
// Each phase produces its own append-only entry; confirm/execute emit
// separate entries for requester (User A) and confirmer (User B).
EventDocumentTrash = "document_trash" // soft-delete into trash
EventDocumentRestore = "document_restore" // restored from trash
EventDocumentDeleteRequest = "document_delete_request" // final-deletion requested (User A)
EventDocumentDeleteConfirm = "document_delete_confirm" // final-deletion confirmed (User B)
EventDocumentDeleteExecute = "document_delete_execute" // WORM file removed, tombstone kept
EventDocumentDeleteBlocked = "document_delete_blocked_retention" // blocked by retain_until
// Reminder ("Wiedervorlage") events.
EventReminderCreate = "reminder_create"
EventReminderStatusChange = "reminder_status_change"
EventReminderDelete = "reminder_delete"
EventReminderNotify = "reminder_notify" // cron: due-date notification sent
// SFTP credential lifecycle + login events (internal/sftpserver,
// internal/api/sftp_handlers.go).
EventSFTPCredentialCreate = "sftp_credential_create"
EventSFTPCredentialRevoke = "sftp_credential_revoke"
EventSFTPLogin = "sftp_login" // logged on every attempt, success and failure
// Permission model (group-resolved document ACL, internal/storage/permissions.go,
// internal/api/permission_handlers.go). Covers group/member changes and all
// three grant layers (document-type / tag / per-document, incl. 'deny').
EventPermissionGrantChanged = "permission_grant_changed"
// External document share-links (internal/storage/shares.go,
// internal/api/share_handlers.go + public_share_handlers.go). Create/Revoke
// are authenticated tenant actions; Accessed is logged on the public
// download endpoint (success and failure — see also the per-attempt
// document_share_accesses table for the full public-access trail).
EventShareCreated = "share_created"
EventShareRevoked = "share_revoked"
EventShareAccessed = "share_accessed"
// LDAP directory integration (internal/ldapstore, internal/ldapauth,
// internal/api/ldap_handlers.go). ConfigChanged covers create/update/delete
// and the test action; LoginSuccess/Failed are logged on every LDAP bind
// attempt (incl. JIT provisioning); RoleSync records a role change applied by
// group membership re-synchronisation (including downgrades).
EventLdapConfigChanged = "ldap_config_changed"
EventLdapLoginSuccess = "ldap_login_success"
EventLdapLoginFailed = "ldap_login_failed"
EventLdapRoleSync = "ldap_role_sync"
// EventOllamaConfigUpdate records a change to a tenant's external-Ollama
// connection config (GET/PUT /api/ollama-config). Logged on every attempt,
// success and failure (GoBD-Nachvollziehbarkeit).
EventOllamaConfigUpdate = "ollama_config_update"
// Classification templates (Klassifizierungsvorlagen,
// internal/storage/classification_templates.go,
// internal/api/classification_template_handlers.go). Create/Update/Delete
// cover template administration (incl. tag / field-default bulk replace);
// Applied records applying a template to a document — logged on success and
// failure, and also when a retain_until shortening attempt was rejected
// (RetainUntilBlocked) so GoBD traceability shows the rejection explicitly.
EventTemplateCreate = "classification_template_create"
EventTemplateUpdate = "classification_template_update"
EventTemplateDelete = "classification_template_delete"
EventTemplateApplied = "classification_template_applied"
// Workflows / Consumption-Regeln (internal/storage/workflows.go,
// internal/api/workflow_handlers.go). Create/Update/Delete cover workflow
// administration (incl. action bulk replace and dry-run test). EventWorkflowRun
// records a manual (test-endpoint) or automatic (on_upload) evaluation. Note
// automatic runs are additionally recorded document-scoped in the
// workflow_runs table for GoBD reproducibility.
EventWorkflowCreate = "workflow_create"
EventWorkflowUpdate = "workflow_update"
EventWorkflowDelete = "workflow_delete"
EventWorkflowRun = "workflow_run"
// Heuristische Metadaten-Vorschläge (internal/storage/metadata_suggestions.go,
// internal/api/metadata_suggestion_handlers.go). Records a (non-binding)
// suggestion run for a document; accepting a suggested field goes through the
// normal edit endpoints, not through this event.
EventSuggestionGenerated = "metadata_suggestion_generated"
// Freitext-Notizen pro Dokument (internal/storage/document_notes.go,
// internal/api/document_note_handlers.go). Create/Delete cover the note
// lifecycle; pure reads (listing notes) are not audited, consistent with the
// rest of this project. Notes are hard-deleted (not GoBD documents), but the
// deletion itself is still recorded for Nachvollziehbarkeit.
EventNoteCreate = "document_note_create"
EventNoteDelete = "document_note_delete"
// EventMLRetrain records a Naive-Bayes classifier retraining run
// (internal/classifier, cmd/archivdms/cmd_classify_retrain.go, cron-driven).
// Logged once per tenant per retrain, success and failure — a per-tenant
// failure is isolated and does not block the other tenants. Detail carries
// the per-kind document counts / skip reasons for GoBD-Nachvollziehbarkeit.
EventMLRetrain = "ml_classifier_retrain"
// Gespeicherte Suchansichten (SavedViews, Paperless-ngx inspiriert —
// internal/storage/saved_views.go, internal/api/saved_view_handlers.go).
// Create/Update/Delete cover a user's named, reusable search/filter view.
// Views can be private (own) or shared tenant-wide (is_shared); only the
// creator may update or delete a view. Pure reads (listing views) are not
// audited, consistent with the rest of this project.
EventSavedViewCreate = "saved_view_create"
EventSavedViewUpdate = "saved_view_update"
EventSavedViewDelete = "saved_view_delete"
// EventRetentionApplied records a batch run of the GoBD retention-rules
// engine (internal/storage/retention_rules.go ApplyRetentionRules,
// cmd/archivdms/cmd_retention_apply.go, cron-driven). One summary entry per
// run per tenant (tenant + count of documents whose retain_until was
// computed and set), NOT per document — batch-summary style to avoid audit
// log spam. Logged on success and failure.
EventRetentionApplied = "retention_applied"
// EventRetentionRuleCreate/Update/Delete record CRUD changes to GoBD
// retention rules (internal/storage/retention_rules.go,
// internal/api/retention_rule_handlers.go). Compliance-critical: changing a
// retention period alters how long documents must be kept, so every mutation
// — success and failure — is audit-logged.
EventRetentionRuleCreate = "retention_rule_create"
EventRetentionRuleUpdate = "retention_rule_update"
EventRetentionRuleDelete = "retention_rule_delete"
// Digitale Akten (digitaler Aktenordner — internal/storage/akten.go,
// internal/api/akte_handlers.go). Create/Update/Close/Delete cover the akte
// lifecycle; DocumentAdd/DocumentRemove record assigning/removing a document
// to/from an akte (PUT /api/documents/{id}/akte). An akte has no own ACL —
// its visibility derives from the documents it contains (see
// project_akte_konzept_plan.md).
EventAkteCreate = "akte_create"
EventAkteUpdate = "akte_update"
EventAkteClose = "akte_close"
EventAkteDelete = "akte_delete"
EventAkteDocumentAdd = "akte_document_add"
EventAkteDocumentRemove = "akte_document_remove"
// EventDocumentSplit records a barcode-separator-page split at ingest
// (internal/pagesplit, internal/api/document_handlers.go storeUploadedFile).
// GoBD-Nachvollziehbarkeit: the uploaded multi-page original is NOT archived
// as such — it is replaced by N part documents — so the split itself is the
// only record tying the parts back to the original upload. Detail therefore
// carries the original filename, its SHA-256, the page count, the separator
// page numbers and the resulting document IDs. Logged on success and on
// failure (Success:false when the split was attempted but aborted, in which
// case the original is archived unsplit).
EventDocumentSplit = "document_split"
// EventDocumentExport records a single-document export
// (GET /api/documents/{id}/export): a ZIP containing the original WORM file,
// metadata.json and, if present, ocr_text.txt. Read-only, but archived
// content plus its full metadata leaves the system as a package, so — like
// EventComplianceExport / EventAccountingPull — it is logged like a mutation,
// including failures (Success:false).
EventDocumentExport = "document_export"
// EventDocumentBulkExport records a multi-document export
// (POST /api/documents/export): one ZIP with a doc-<id>/ folder per
// document plus index.csv. Exactly ONE entry per request (not per
// document) — Detail carries "exported=<n> skipped=<m>", where skipped
// counts documents the caller could not see or that failed to read (also
// listed in the archive's errors.txt). Logged on failure too
// (Success:false), including rejected requests (too many IDs, invalid
// filter) and aborted streams.
EventDocumentBulkExport = "document_bulk_export"
)
// Compliance-Export (internal/api/compliance_handlers.go).
const (
// EventComplianceExport records a generated GoBD-Verfahrensdokumentation
// draft. Read-only, but exported tenant configuration leaves the system, so
// it is logged like a mutation (including failures) — and for a superadmin
// cross-tenant export the Detail carries the target tenant_id.
EventComplianceExport = "compliance_procedure_doc_export"
)
// Buchhaltungs-Pull-API (internal/api/accounting_handlers.go,
// internal/storage/accounting_api_keys.go).
const (
// EventAccountingKeyCreated/Revoked record the lifecycle of a per-tenant
// accounting API key. The plaintext key is NEVER part of Detail — only the
// key id and its label.
EventAccountingKeyCreated = "accounting_key_created"
EventAccountingKeyRevoked = "accounting_key_revoked"
// EventAccountingPull records machine-to-machine reads through an
// accounting API key. Read-only, but archived content leaves the system via
// a non-browser path, so it is logged like a mutation (including failures):
// every file download individually, and every list query with its result
// range (first/last document id).
EventAccountingPull = "accounting_pull"
)
// Entry is a single audit log record.
type Entry struct {
ID int64 `json:"id"`
Timestamp time.Time `json:"timestamp"`
EventType string `json:"event_type"`
Username string `json:"username"`
IPAddress string `json:"ip_address"`
DocumentID string `json:"document_id"`
Success bool `json:"success"`
Detail string `json:"detail"`
// TenantID, when set, records which tenant this event belongs to. nil means
// a tenant-less / system-wide event (e.g. superadmin actions, cron jobs).
TenantID *int64 `json:"tenant_id,omitempty"`
}
// QueryFilter specifies filtering options for audit log queries.
type QueryFilter struct {
Username string
EventType string
DocumentID string
From *time.Time
To *time.Time
TenantID *int64
PageSize int
Page int
}
// Logger is a PostgreSQL-backed, append-only audit log mirrored to a
// tamper-evident JSON-Lines file opened in append-only mode.
type Logger struct {
pool *pgxpool.Pool
logger *slog.Logger
fileMu sync.Mutex
file *os.File
logPath string
}
type fileEntry struct {
Timestamp string `json:"timestamp"`
EventType string `json:"event_type"`
Username string `json:"username"`
IPAddress string `json:"ip_address"`
DocumentID string `json:"document_id,omitempty"`
Success bool `json:"success"`
Detail string `json:"detail,omitempty"`
TenantID *int64 `json:"tenant_id,omitempty"`
}
// New connects to PostgreSQL using the given DSN and initialises the schema.
func New(dsn, logPath string, logger *slog.Logger) (*Logger, error) {
ctx := context.Background()
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("audit: connect: %w", err)
}
if err := initSchema(ctx, pool); err != nil {
pool.Close()
return nil, fmt.Errorf("audit: create schema: %w", err)
}
l := &Logger{pool: pool, logger: logger, logPath: logPath}
l.openLogFile()
return l, nil
}
// initSchema creates the audit_log table and installs the immutability
// trigger. Both operations are idempotent and safe on existing databases.
func initSchema(ctx context.Context, pool *pgxpool.Pool) error {
if _, err := pool.Exec(ctx, `
CREATE TABLE IF NOT EXISTS audit_log (
id BIGSERIAL PRIMARY KEY,
timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
event_type VARCHAR(50) NOT NULL,
username VARCHAR(255) NOT NULL DEFAULT '',
ip_address VARCHAR(45) NOT NULL DEFAULT '',
document_id VARCHAR(64) NOT NULL DEFAULT '',
success BOOLEAN NOT NULL DEFAULT true,
detail TEXT NOT NULL DEFAULT '',
tenant_id BIGINT
);
`); err != nil {
return err
}
// Append-only enforcement at the database level: any UPDATE/DELETE raises.
if _, err := pool.Exec(ctx, `
CREATE OR REPLACE FUNCTION audit_log_no_mutation()
RETURNS trigger AS $$
BEGIN
RAISE EXCEPTION 'audit_log is append-only: % is not permitted', TG_OP
USING ERRCODE = 'integrity_constraint_violation';
END;
$$ LANGUAGE plpgsql;
`); err != nil {
return fmt.Errorf("create trigger function: %w", err)
}
if _, err := pool.Exec(ctx, `DROP TRIGGER IF EXISTS audit_log_immutable ON audit_log;`); err != nil {
return fmt.Errorf("drop trigger: %w", err)
}
if _, err := pool.Exec(ctx, `
CREATE TRIGGER audit_log_immutable
BEFORE UPDATE OR DELETE ON audit_log
FOR EACH ROW EXECUTE FUNCTION audit_log_no_mutation();
`); err != nil {
return fmt.Errorf("create trigger: %w", err)
}
return nil
}
func (l *Logger) openLogFile() {
if l.logPath == "" {
l.logger.Warn("audit: log_path not configured, file logging disabled")
return
}
if dir := filepath.Dir(l.logPath); dir != "" && dir != "." {
_ = os.MkdirAll(dir, 0o750)
}
f, err := os.OpenFile(l.logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o640)
if err != nil {
l.logger.Warn("audit: audit log file not writable, continuing with DB-only logging",
"path", l.logPath, "err", err)
return
}
l.file = f
}
// Log appends an entry to the audit log. Errors are logged but not returned.
func (l *Logger) Log(entry Entry) {
ts := entry.Timestamp
if ts.IsZero() {
ts = time.Now().UTC()
}
ctx := context.Background()
_, err := l.pool.Exec(ctx,
`INSERT INTO audit_log (timestamp, event_type, username, ip_address, document_id, success, detail, tenant_id)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)`,
ts.UTC(), entry.EventType, entry.Username, entry.IPAddress, entry.DocumentID,
entry.Success, entry.Detail, entry.TenantID,
)
if err != nil {
l.logger.Error("audit: insert failed", "err", err)
}
l.writeFile(entry, ts.UTC())
}
func (l *Logger) writeFile(entry Entry, ts time.Time) {
l.fileMu.Lock()
defer l.fileMu.Unlock()
if l.file == nil {
return
}
line, err := json.Marshal(fileEntry{
Timestamp: ts.Format(time.RFC3339),
EventType: entry.EventType,
Username: entry.Username,
IPAddress: entry.IPAddress,
DocumentID: entry.DocumentID,
Success: entry.Success,
Detail: entry.Detail,
TenantID: entry.TenantID,
})
if err != nil {
l.logger.Error("audit: marshal log line failed", "err", err)
return
}
if _, err := l.file.Write(append(line, '\n')); err != nil {
l.logger.Error("audit: write to log file failed", "path", l.logPath, "err", err)
}
}
// Query retrieves audit entries matching the given filter.
func (l *Logger) Query(filter QueryFilter) ([]Entry, int, error) {
pageSize := filter.PageSize
if pageSize <= 0 {
pageSize = 50
}
where, args := buildWhere(filter)
ctx := context.Background()
countSQL := "SELECT COUNT(*) FROM audit_log" + where
var total int
if err := l.pool.QueryRow(ctx, countSQL, args...).Scan(&total); err != nil {
return nil, 0, fmt.Errorf("audit: count: %w", err)
}
offset := filter.Page * pageSize
limitArg := len(args) + 1
offsetArg := len(args) + 2
querySQL := fmt.Sprintf(
"SELECT id, timestamp, event_type, username, ip_address, document_id, success, detail FROM audit_log%s ORDER BY timestamp DESC LIMIT $%d OFFSET $%d",
where, limitArg, offsetArg,
)
allArgs := append(args, pageSize, offset)
rows, err := l.pool.Query(ctx, querySQL, allArgs...)
if err != nil {
return nil, 0, fmt.Errorf("audit: query: %w", err)
}
defer rows.Close()
var entries []Entry
for rows.Next() {
var e Entry
if err := rows.Scan(&e.ID, &e.Timestamp, &e.EventType, &e.Username, &e.IPAddress, &e.DocumentID, &e.Success, &e.Detail); err != nil {
return nil, 0, fmt.Errorf("audit: scan: %w", err)
}
entries = append(entries, e)
}
return entries, total, rows.Err()
}
// Close closes the audit log file and the connection pool.
func (l *Logger) Close() error {
l.fileMu.Lock()
if l.file != nil {
_ = l.file.Sync()
_ = l.file.Close()
l.file = nil
}
l.fileMu.Unlock()
l.pool.Close()
return nil
}
func buildWhere(f QueryFilter) (string, []interface{}) {
var clauses []string
var args []interface{}
n := 1
if f.Username != "" {
clauses = append(clauses, fmt.Sprintf("username = $%d", n))
args = append(args, f.Username)
n++
}
if f.EventType != "" {
clauses = append(clauses, fmt.Sprintf("event_type = $%d", n))
args = append(args, f.EventType)
n++
}
if f.DocumentID != "" {
clauses = append(clauses, fmt.Sprintf("document_id = $%d", n))
args = append(args, f.DocumentID)
n++
}
if f.From != nil {
clauses = append(clauses, fmt.Sprintf("timestamp >= $%d", n))
args = append(args, f.From.UTC())
n++
}
if f.To != nil {
clauses = append(clauses, fmt.Sprintf("timestamp <= $%d", n))
args = append(args, f.To.UTC())
n++
}
if f.TenantID != nil {
clauses = append(clauses, fmt.Sprintf("tenant_id = $%d", n))
args = append(args, *f.TenantID)
n++
}
if len(clauses) == 0 {
return "", args
}
return " WHERE " + strings.Join(clauses, " AND "), args
}
+380
View File
@@ -0,0 +1,380 @@
// Package auth implements login, JWT session issuance/validation, and logout,
// ported from archivmail's internal/auth pattern. LDAP and TOTP are
// intentionally left out of this initial scaffold; they can be re-added later
// following the same shape archivmail uses (Manager.SetTenantLDAP etc.).
package auth
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"strings"
"time"
"github.com/golang-jwt/jwt/v5"
"archivdms/internal/audit"
"archivdms/internal/ldapauth"
"archivdms/internal/ldapstore"
"archivdms/internal/tenantstore"
"archivdms/internal/userstore"
)
// Session holds the claims extracted from a validated JWT.
type Session struct {
UserID int64
Username string
Email string
Role string
JTI string
TenantID *int64
}
// Manager handles login, token issuance, validation, and logout.
type Manager struct {
store *userstore.Store
jwtSecret []byte
// ldap is nil until SetLDAP wires the directory integration. When nil the
// login flow is local-only (bcrypt).
ldap *ldapComponent
}
// ldapComponent bundles the dependencies for LDAP authentication. Kept behind
// a pointer so a deployment without LDAP configured pays zero cost.
type ldapComponent struct {
store *ldapstore.Store
authn *ldapauth.Authenticator
tenants *tenantstore.Store
audlog *audit.Logger
limiter *keyedRateLimiter
}
// New creates a new auth Manager.
func New(store *userstore.Store, jwtSecret string) *Manager {
return &Manager{store: store, jwtSecret: []byte(jwtSecret)}
}
// SetLDAP wires the LDAP directory integration into the auth manager. Called
// once at startup after the ldapstore/tenantstore/audit dependencies exist.
func (m *Manager) SetLDAP(ldapSt *ldapstore.Store, authn *ldapauth.Authenticator, tenants *tenantstore.Store, audlog *audit.Logger) {
m.ldap = &ldapComponent{
store: ldapSt,
authn: authn,
tenants: tenants,
audlog: audlog,
// 10 attempts burst, refilled at 1 / 6s per key (~10/min sustained).
limiter: newKeyedRateLimiter(10, 1.0/6.0),
}
}
// Login verifies credentials and returns a signed JWT token. Kept for callers
// that do not have request context (client IP); prefer LoginFrom.
func (m *Manager) Login(username, password string) (token string, user *userstore.User, err error) {
return m.LoginFrom(context.Background(), username, password, "")
}
// LoginFrom verifies credentials, honouring each account's auth_source and,
// where applicable, the tenant's LDAP configuration. ip is used for
// rate-limiting and audit context.
//
// Decision matrix:
// - account exists, auth_source=local : bcrypt only, LDAP never attempted.
// - account exists, auth_source=ldap : LDAP bind only, no local-password fallback.
// - account absent, tenant LDAP enabled: LDAP bind + JIT provisioning.
// - otherwise : fail (with bcrypt timing burn).
func (m *Manager) LoginFrom(ctx context.Context, identifier, password, ip string) (string, *userstore.User, error) {
rec, err := m.store.FindForLogin(ctx, identifier)
switch {
case err == nil && rec.AuthSource == userstore.AuthSourceLDAP:
if m.ldap == nil {
return "", nil, fmt.Errorf("auth: login: invalid credentials")
}
return m.ldapLogin(ctx, derefTenant(rec.User.TenantID), identifier, password, ip, rec.User)
case err == nil:
if !rec.User.Active {
return "", nil, fmt.Errorf("auth: login: invalid credentials")
}
if cErr := userstore.CompareLocalPassword(rec.Hash, password); cErr != nil {
return "", nil, fmt.Errorf("auth: login: invalid credentials")
}
return m.issueToken(rec.User)
case errors.Is(err, userstore.ErrUserNotFound):
// Possible just-in-time LDAP provisioning.
if tid := m.jitTenant(ctx, identifier); tid != nil {
return m.ldapLogin(ctx, *tid, identifier, password, ip, nil)
}
m.store.BurnPasswordTiming(password)
return "", nil, fmt.Errorf("auth: login: invalid credentials")
default:
return "", nil, fmt.Errorf("auth: login: %w", err)
}
}
// jitTenant returns the tenant ID an unknown identifier may be provisioned
// into: the identifier's email domain must map to a tenant that has an enabled
// LDAP config. Returns nil when JIT is not applicable.
func (m *Manager) jitTenant(ctx context.Context, identifier string) *int64 {
if m.ldap == nil {
return nil
}
at := strings.LastIndex(identifier, "@")
if at < 0 || at == len(identifier)-1 {
return nil
}
domain := strings.ToLower(identifier[at+1:])
tid, err := m.ldap.tenants.GetTenantIDByDomain(ctx, domain)
if err != nil || tid == nil {
return nil
}
cfg, err := m.ldap.store.Get(ctx, *tid)
if err != nil || !cfg.Enabled {
return nil
}
return tid
}
// ldapLogin performs the LDAP bind, applies role mapping, JIT-provisions or
// re-synchronises the local user, and issues a token. existing may be nil (JIT).
func (m *Manager) ldapLogin(ctx context.Context, tenantID int64, loginName, password, ip string, existing *userstore.User) (string, *userstore.User, error) {
tid := tenantID
logFail := func(detail string) {
m.ldap.audlog.Log(audit.Entry{
EventType: audit.EventLdapLoginFailed, Username: loginName, IPAddress: ip,
TenantID: &tid, Success: false, Detail: detail,
})
}
// Rate limit per (tenant, loginName) and per source IP.
userKey := fmt.Sprintf("u:%d:%s", tenantID, strings.ToLower(loginName))
if !m.ldap.limiter.allow(userKey) || (ip != "" && !m.ldap.limiter.allow("ip:"+ip)) {
logFail("rate_limited")
return "", nil, fmt.Errorf("auth: login: too many attempts")
}
cfg, bindPw, err := m.ldap.store.GetWithSecret(ctx, tenantID)
if err != nil {
logFail("config_unavailable")
return "", nil, fmt.Errorf("auth: login: invalid credentials")
}
if !cfg.Enabled {
logFail("ldap_disabled")
return "", nil, fmt.Errorf("auth: login: invalid credentials")
}
res, err := m.ldap.authn.Authenticate(ctx, cfg, bindPw, loginName, password)
if err != nil {
logFail("bind_failed")
return "", nil, fmt.Errorf("auth: login: invalid credentials")
}
// Role mapping: admin group membership -> domain_admin, else user.
// LDAP can NEVER confer superadmin.
role := userstore.RoleUser
if res.IsAdmin {
role = userstore.RoleDomainAdmin
}
var user *userstore.User
if existing == nil {
username := res.Username
if username == "" {
username = loginName
}
email := res.Email
if email == "" {
email = loginName
}
user, err = m.store.CreateLDAPUser(ctx, userstore.LDAPUserRequest{
Username: username, Email: email, Role: role, LdapUID: res.Username, TenantID: &tid,
})
if err != nil {
logFail("provision_failed")
return "", nil, fmt.Errorf("auth: login: provisioning failed")
}
} else {
if existing.Role != role && existing.Role != userstore.RoleSuperAdmin {
m.ldap.audlog.Log(audit.Entry{
EventType: audit.EventLdapRoleSync, Username: existing.Username, IPAddress: ip,
TenantID: &tid, Success: true,
Detail: fmt.Sprintf("role %s -> %s", existing.Role, role),
})
}
// Never downgrade a superadmin via LDAP (defensive; LDAP accounts are
// never superadmin, but guard against manual DB edits).
syncRole := role
if existing.Role == userstore.RoleSuperAdmin {
syncRole = userstore.RoleSuperAdmin
}
email := res.Email
if email == "" {
email = existing.Email
}
if err := m.store.SyncLDAPUser(ctx, existing.ID, email, syncRole); err != nil {
logFail("sync_failed")
return "", nil, fmt.Errorf("auth: login: sync failed")
}
user, err = m.store.GetByID(existing.ID)
if err != nil {
logFail("reload_failed")
return "", nil, fmt.Errorf("auth: login: reload failed")
}
}
m.ldap.audlog.Log(audit.Entry{
EventType: audit.EventLdapLoginSuccess, Username: user.Username, IPAddress: ip,
TenantID: &tid, Success: true, Detail: "role:" + user.Role,
})
return m.issueToken(user)
}
func derefTenant(t *int64) int64 {
if t == nil {
return 0
}
return *t
}
func (m *Manager) issueToken(user *userstore.User) (string, *userstore.User, error) {
jti := generateJTI()
now := time.Now()
claims := jwt.MapClaims{
"sub": user.Username,
"email": user.Email,
"role": user.Role,
"uid": user.ID,
"jti": jti,
"iat": now.Unix(),
"exp": now.Add(8 * time.Hour).Unix(),
}
if user.TenantID != nil {
claims["tenant_id"] = *user.TenantID
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
signed, err := token.SignedString(m.jwtSecret)
if err != nil {
return "", nil, fmt.Errorf("auth: sign token: %w", err)
}
return signed, user, nil
}
// ValidateToken parses and validates the token, checking the blacklist.
func (m *Manager) ValidateToken(tokenStr string) (*Session, error) {
token, err := jwt.Parse(tokenStr, func(t *jwt.Token) (interface{}, error) {
if _, ok := t.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, fmt.Errorf("auth: unexpected signing method: %v", t.Header["alg"])
}
return m.jwtSecret, nil
})
if err != nil {
return nil, fmt.Errorf("auth: invalid token: %w", err)
}
if !token.Valid {
return nil, errors.New("auth: token not valid")
}
claims, ok := token.Claims.(jwt.MapClaims)
if !ok {
return nil, errors.New("auth: bad claims")
}
jti, _ := claims["jti"].(string)
blacklisted, err := m.store.IsBlacklisted(jti)
if err != nil {
return nil, fmt.Errorf("auth: blacklist check: %w", err)
}
if blacklisted {
return nil, errors.New("auth: token revoked")
}
username, _ := claims["sub"].(string)
email, _ := claims["email"].(string)
role, _ := claims["role"].(string)
var userID int64
switch v := claims["uid"].(type) {
case float64:
userID = int64(v)
case int64:
userID = v
}
var tenantID *int64
switch v := claims["tenant_id"].(type) {
case float64:
id := int64(v)
tenantID = &id
case int64:
id := v
tenantID = &id
}
return &Session{
UserID: userID,
Username: username,
Email: email,
Role: role,
JTI: jti,
TenantID: tenantID,
}, nil
}
// Logout revokes the token by adding its JTI to the blacklist.
func (m *Manager) Logout(tokenStr string) error {
token, err := jwt.Parse(tokenStr, func(t *jwt.Token) (interface{}, error) {
if _, ok := t.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, fmt.Errorf("auth: unexpected signing method")
}
return m.jwtSecret, nil
})
if err != nil {
return fmt.Errorf("auth: logout parse: %w", err)
}
claims, ok := token.Claims.(jwt.MapClaims)
if !ok {
return errors.New("auth: bad claims on logout")
}
jti, _ := claims["jti"].(string)
var exp time.Time
switch v := claims["exp"].(type) {
case float64:
exp = time.Unix(int64(v), 0)
case int64:
exp = time.Unix(v, 0)
default:
exp = time.Now().Add(8 * time.Hour)
}
return m.store.BlacklistToken(jti, exp)
}
// HasRole returns true when userRole satisfies the required role level.
// Hierarchy: superadmin > domain_admin > user
func HasRole(userRole, required string) bool {
levels := map[string]int{
userstore.RoleUser: 1,
userstore.RoleDomainAdmin: 2,
userstore.RoleSuperAdmin: 3,
}
return levels[userRole] >= levels[required]
}
// generateJTI returns a cryptographically random identifier for a JWT.
func generateJTI() string {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return fmt.Sprintf("%d", time.Now().UnixNano())
}
return hex.EncodeToString(b)
}
// GetUserStore returns the underlying user store.
func (m *Manager) GetUserStore() *userstore.Store {
return m.store
}
+70
View File
@@ -0,0 +1,70 @@
package auth
import (
"sync"
"time"
)
// keyedRateLimiter is a minimal in-memory token-bucket limiter keyed by an
// arbitrary string (used for per-(tenant,loginName) and per-IP LDAP login
// throttling). No external dependency, no Redis — buckets are created lazily
// and idle ones are swept periodically to bound memory.
type keyedRateLimiter struct {
mu sync.Mutex
buckets map[string]*bucket
burst float64
refillPerSec float64
lastSweep time.Time
}
type bucket struct {
tokens float64
last time.Time
}
func newKeyedRateLimiter(burst, refillPerSec float64) *keyedRateLimiter {
return &keyedRateLimiter{
buckets: make(map[string]*bucket),
burst: burst,
refillPerSec: refillPerSec,
lastSweep: time.Now(),
}
}
// allow consumes one token for key, returning false when the bucket is empty.
func (l *keyedRateLimiter) allow(key string) bool {
now := time.Now()
l.mu.Lock()
defer l.mu.Unlock()
l.sweepLocked(now)
b, ok := l.buckets[key]
if !ok {
b = &bucket{tokens: l.burst, last: now}
l.buckets[key] = b
}
b.tokens += now.Sub(b.last).Seconds() * l.refillPerSec
if b.tokens > l.burst {
b.tokens = l.burst
}
b.last = now
if b.tokens < 1 {
return false
}
b.tokens--
return true
}
func (l *keyedRateLimiter) sweepLocked(now time.Time) {
if now.Sub(l.lastSweep) < time.Minute {
return
}
l.lastSweep = now
for k, b := range l.buckets {
if now.Sub(b.last) > 30*time.Minute {
delete(l.buckets, k)
}
}
}
+64
View File
@@ -0,0 +1,64 @@
// Package barcode wraps the system zbarimg binary (Debian package
// zbar-tools) as a best-effort barcode decoding sidecar, consistent with
// archivdms's general philosophy of shelling out to small CLI tools via
// os/exec instead of adding CGO/native Go dependencies (see
// internal/ocr/ocr.go package comment for the same rationale applied to
// tesseract/poppler-utils).
//
// Barcode decoding failures (binary missing, non-zero exit because no
// barcode was found, timeout) are never fatal to an upload: DecodeBarcodes
// returns an empty slice and a nil error in the "nothing found / binary
// missing" cases, matching how OCR failures are tolerated by callers.
package barcode
import (
"bytes"
"context"
"fmt"
"os/exec"
"strings"
"time"
)
const defaultTimeout = 20 * time.Second
// DecodeBarcodes runs `zbarimg --raw -q <imagePath>` and returns the
// decoded raw values, one per line of output. If zbarimg is not installed,
// this returns an empty slice and a nil error (tolerant, no hard-fail) —
// callers that want to warn about a missing binary should check
// exec.LookPath("zbarimg") themselves if they need to distinguish
// "not installed" from "installed, nothing found".
func DecodeBarcodes(ctx context.Context, imagePath string) ([]string, error) {
if _, err := exec.LookPath("zbarimg"); err != nil {
// Binary not present: tolerated, same as a missing tesseract binary.
return nil, nil
}
cctx, cancel := context.WithTimeout(ctx, defaultTimeout)
defer cancel()
cmd := exec.CommandContext(cctx, "zbarimg", "--raw", "-q", imagePath)
var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
// zbarimg exits non-zero (typically exit code 4) when no barcode is
// found in the image at all — that is a normal, expected outcome for
// the vast majority of scanned documents, not an error condition.
if exitErr, ok := err.(*exec.ExitError); ok {
_ = exitErr
return nil, nil
}
return nil, fmt.Errorf("barcode: zbarimg failed: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
var values []string
for _, line := range strings.Split(out.String(), "\n") {
line = strings.TrimSpace(line)
if line != "" {
values = append(values, line)
}
}
return values, nil
}
+617
View File
@@ -0,0 +1,617 @@
// Package classifier implements a dependency-free multinomial Naive-Bayes text
// classifier that supplements archivdms's rule-based matching engine
// (internal/matching) WITHOUT introducing any LLM dependency. It is designed to
// give clean, self-standing results even when no Ollama/LLM is configured: the
// tokenizer, German stop-word handling and Laplace smoothing are the quality
// levers and are treated as first-class, not minimal.
//
// The model is persisted in the ml_classifier_tokens / ml_classifier_classes
// tables (see internal/storage/ml_classifier.go). Training is a full
// per-tenant/per-kind rebuild (DELETE + bulk insert) — deliberately simple, no
// incremental updates. Classification (Predict) reads that persisted model and
// returns softmax-normalised posterior probabilities so the caller can apply a
// single confidence floor comparable to the heuristic provider's
// suggestionFloor.
//
// This package MUST NOT import internal/storage (storage imports it, for
// Store.GenerateNaiveBayesSuggestions) — it talks to Postgres through the small
// DB interface below, which *pgxpool.Pool satisfies.
package classifier
import (
"context"
"fmt"
"math"
"sort"
"strings"
"unicode"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
)
// DB is the minimal Postgres surface the classifier needs. *pgxpool.Pool (and
// pgx.Tx, for the value passed to Predict from within a transaction) satisfy it.
type DB interface {
Begin(ctx context.Context) (pgx.Tx, error)
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
}
const (
// MinDocsPerClass is the minimum number of labelled training documents a
// class (document_type / correspondent / tag) must have before it is kept in
// the model. Below this the class is silently skipped (no error): too few
// examples produce an over-confident, unreliable token distribution.
MinDocsPerClass = 20
// laplaceAlpha is the additive (Laplace/Lidstone) smoothing constant applied
// to every token count. alpha=1 (classic add-one) keeps unseen tokens from
// zeroing out a whole class's likelihood while staying conservative for
// small vocabularies.
laplaceAlpha = 1.0
// SuggestionFloor is the minimum softmax posterior probability at which a
// class is surfaced as a candidate. Chosen to mirror the heuristic
// provider's suggestionFloor (0.55) so the two engines feel consistent to
// the user: below this the model is essentially undecided between classes.
SuggestionFloor = 0.55
// maxCandidates caps how many classes Predict returns, sorted by posterior
// descending (matches maxSuggestionCandidates in the storage layer).
maxCandidates = 5
// maxExplanationTokens is how many "most decisive" tokens are attached to a
// candidate as a human-readable explanation.
maxExplanationTokens = 5
// minTokenRunes / maxTokenRunes bound token length: 1-rune tokens are noise;
// absurdly long runs are almost always OCR garbage (barcodes, scan
// artefacts) rather than meaningful words.
minTokenRunes = 2
maxTokenRunes = 40
// minNumericTokenRunes: purely numeric tokens shorter than this (page
// numbers, single amounts, "1."/"12") are dropped as noise, but longer
// numeric runs are KEPT — invoice/customer numbers, IBAN fragments and years
// recur across a correspondent's documents and carry real signal.
minNumericTokenRunes = 4
)
// Kinds are the three classifiable taxonomy kinds. Mirrors the CHECK constraint
// on ml_classifier_tokens.kind.
const (
KindDocumentTypes = "document_types"
KindCorrespondents = "correspondents"
KindTags = "tags"
)
// SuggestionCandidate is one scored class prediction. Score is a softmax
// posterior probability in [0,1]. TopTokens lists (at most maxExplanationTokens)
// input tokens that contributed most to selecting this class over the runner-up
// — the model's explanation for GoBD-Nachvollziehbarkeit / user trust.
type SuggestionCandidate struct {
EntityID int64 `json:"entity_id"`
Score float64 `json:"score"`
TopTokens []string `json:"top_tokens"`
}
// Classifier is a thin, stateless wrapper around a DB handle. Safe to construct
// per call.
type Classifier struct {
db DB
}
// New returns a Classifier backed by db.
func New(db DB) *Classifier {
return &Classifier{db: db}
}
// validKind guards the kind against the allowlist so it can be interpolated
// nowhere (all queries parameterise it) but callers still fail fast on typos.
func validKind(kind string) error {
switch kind {
case KindDocumentTypes, KindCorrespondents, KindTags:
return nil
default:
return fmt.Errorf("classifier: unknown kind %q", kind)
}
}
// ---------------------------------------------------------------------------
// Tokenizer
// ---------------------------------------------------------------------------
// germanStopwords is a curated list of high-frequency German function words
// (plus a few ubiquitous document-boilerplate terms) that carry no class
// signal. Removing them sharpens the per-class token distributions. Kept as a
// set for O(1) lookup. Not exhaustive by design — the goal is to strip the
// worst offenders, not to stem the language.
var germanStopwords = func() map[string]struct{} {
words := []string{
"der", "die", "das", "den", "dem", "des", "ein", "eine", "einen", "einem",
"einer", "eines", "und", "oder", "aber", "auch", "sich", "nicht", "mit",
"für", "von", "vom", "im", "in", "am", "an", "auf", "aus", "bei", "bis",
"durch", "gegen", "ohne", "um", "unter", "über", "zwischen", "nach", "zu",
"zur", "zum", "vor", "hinter", "neben", "ist", "sind", "war", "waren",
"wird", "werden", "wurde", "wurden", "sein", "seine", "seiner", "ihre",
"ihrer", "ihren", "haben", "hat", "hatte", "hatten", "kann", "können",
"muss", "müssen", "soll", "sollen", "als", "wie", "wenn", "dann", "dass",
"daß", "weil", "denn", "doch", "nur", "noch", "schon", "sehr", "hier",
"dort", "man", "wir", "sie", "ich", "du", "er", "es", "ihr", "uns",
"euch", "mein", "dein", "unser", "diese", "dieser", "dieses", "diesem",
"jede", "jeder", "jedes", "alle", "allen", "aller", "kein", "keine",
"keinen", "mehr", "sehr", "so", "auch", "wieder", "bitte", "danke",
"gmbh", "seite", "www", "http", "https", "email", "mail", "tel",
}
m := make(map[string]struct{}, len(words))
for _, w := range words {
m[w] = struct{}{}
}
return m
}()
// tokenize normalises text into a bag of meaningful tokens:
// - Unicode-aware lowercasing.
// - A token is a maximal run of letters and/or digits (so "de89370400"
// survives as one token, and dots/slashes/whitespace all split). This keeps
// structured identifiers (IBAN fragments, invoice numbers) intact instead of
// shredding them into single digits.
// - German stop-words are dropped.
// - Tokens shorter than minTokenRunes or longer than maxTokenRunes are dropped.
// - Purely numeric tokens shorter than minNumericTokenRunes are dropped
// (page numbers, trivial amounts), longer ones are kept (they recur and
// carry signal). Mixed letter+digit tokens are always kept.
//
// Returns a frequency map (token -> count in this text), which is exactly what
// the multinomial model consumes.
func tokenize(text string) map[string]int {
freq := make(map[string]int)
var b strings.Builder
flush := func() {
if b.Len() == 0 {
return
}
tok := b.String()
b.Reset()
addToken(freq, tok)
}
for _, r := range text {
if unicode.IsLetter(r) || unicode.IsDigit(r) {
b.WriteRune(unicode.ToLower(r))
continue
}
flush()
}
flush()
return freq
}
// addToken applies the length / numeric / stop-word filters and, if the token
// survives, increments its frequency.
func addToken(freq map[string]int, tok string) {
runes := []rune(tok)
if len(runes) < minTokenRunes || len(runes) > maxTokenRunes {
return
}
if _, stop := germanStopwords[tok]; stop {
return
}
if isAllDigits(runes) && len(runes) < minNumericTokenRunes {
return
}
freq[tok]++
}
func isAllDigits(runes []rune) bool {
for _, r := range runes {
if !unicode.IsDigit(r) {
return false
}
}
return true
}
// ---------------------------------------------------------------------------
// Training
// ---------------------------------------------------------------------------
// classAccum accumulates token statistics for one class during training.
type classAccum struct {
docCount int64
totalTokens int64
tokens map[string]int64
}
// Train rebuilds the Naive-Bayes model for one tenant and one kind from scratch.
// It reads every labelled, non-deleted training document (assignments made
// manually OR by the rule engine — NOT prior ml_accepted ones, to avoid the
// model reinforcing its own past guesses), tokenizes the title+OCR text, counts
// tokens per class, drops classes with fewer than MinDocsPerClass documents, and
// writes the result with a DELETE + bulk COPY inside a single transaction
// (previous model for this tenant/kind is fully replaced).
//
// Returns the number of training documents actually used (across retained
// classes). A kind with no qualifying data trains to an empty model and returns
// 0 — this is not an error.
func (c *Classifier) Train(ctx context.Context, tenantID int64, kind string) (docCount int, err error) {
if err := validKind(kind); err != nil {
return 0, err
}
rows, err := c.db.Query(ctx, trainingQuery(kind), tenantID)
if err != nil {
return 0, fmt.Errorf("classifier: read training data (%s): %w", kind, err)
}
defer rows.Close()
classes := make(map[int64]*classAccum)
for rows.Next() {
var entityID int64
var title, ocr string
if err := rows.Scan(&entityID, &title, &ocr); err != nil {
return 0, fmt.Errorf("classifier: scan training row (%s): %w", kind, err)
}
acc := classes[entityID]
if acc == nil {
acc = &classAccum{tokens: make(map[string]int64)}
classes[entityID] = acc
}
acc.docCount++
for tok, n := range tokenize(title + "\n" + ocr) {
acc.tokens[tok] += int64(n)
acc.totalTokens += int64(n)
}
}
if err := rows.Err(); err != nil {
return 0, fmt.Errorf("classifier: iterate training rows (%s): %w", kind, err)
}
// Keep only classes with enough evidence.
retained := make(map[int64]*classAccum)
used := 0
for id, acc := range classes {
if acc.docCount < MinDocsPerClass {
continue
}
retained[id] = acc
used += int(acc.docCount)
}
if err := c.persist(ctx, tenantID, kind, retained); err != nil {
return 0, err
}
return used, nil
}
// trainingQuery returns the SQL that yields (entity_id, title, ocr_text) rows
// for a kind, restricted to manual/rule-assigned, non-deleted documents.
func trainingQuery(kind string) string {
switch kind {
case KindDocumentTypes:
return `
SELECT d.doc_type_id, d.title, COALESCE(d.ocr_text, '')
FROM documents d
WHERE d.tenant_id = $1
AND d.deleted_at IS NULL
AND d.doc_type_id IS NOT NULL
AND d.doc_type_assigned_via IN ('manual','rule')`
case KindCorrespondents:
return `
SELECT d.correspondent_id, d.title, COALESCE(d.ocr_text, '')
FROM documents d
WHERE d.tenant_id = $1
AND d.deleted_at IS NULL
AND d.correspondent_id IS NOT NULL
AND d.correspondent_assigned_via IN ('manual','rule')`
case KindTags:
return `
SELECT dt.tag_id, d.title, COALESCE(d.ocr_text, '')
FROM document_tags dt
JOIN documents d ON d.id = dt.document_id
WHERE d.tenant_id = $1
AND d.deleted_at IS NULL
AND dt.assigned_via IN ('manual','rule')`
default:
return ""
}
}
// persist replaces the stored model for tenant/kind with the retained classes,
// atomically (DELETE + COPY inside one transaction). An empty retained map still
// clears the previous model — a class that dropped below the threshold must not
// keep serving stale predictions.
func (c *Classifier) persist(ctx context.Context, tenantID int64, kind string, retained map[int64]*classAccum) error {
tx, err := c.db.Begin(ctx)
if err != nil {
return fmt.Errorf("classifier: begin tx: %w", err)
}
defer tx.Rollback(ctx)
if _, err := tx.Exec(ctx, `DELETE FROM ml_classifier_tokens WHERE tenant_id = $1 AND kind = $2`, tenantID, kind); err != nil {
return fmt.Errorf("classifier: clear tokens: %w", err)
}
if _, err := tx.Exec(ctx, `DELETE FROM ml_classifier_classes WHERE tenant_id = $1 AND kind = $2`, tenantID, kind); err != nil {
return fmt.Errorf("classifier: clear classes: %w", err)
}
classRows := make([][]any, 0, len(retained))
tokenRows := make([][]any, 0)
for entityID, acc := range retained {
classRows = append(classRows, []any{tenantID, kind, entityID, acc.docCount, acc.totalTokens})
for tok, cnt := range acc.tokens {
tokenRows = append(tokenRows, []any{tenantID, kind, entityID, tok, cnt})
}
}
if len(classRows) > 0 {
if _, err := tx.CopyFrom(ctx,
pgx.Identifier{"ml_classifier_classes"},
[]string{"tenant_id", "kind", "entity_id", "doc_count", "total_tokens"},
pgx.CopyFromRows(classRows)); err != nil {
return fmt.Errorf("classifier: copy classes: %w", err)
}
}
if len(tokenRows) > 0 {
if _, err := tx.CopyFrom(ctx,
pgx.Identifier{"ml_classifier_tokens"},
[]string{"tenant_id", "kind", "entity_id", "token", "count"},
pgx.CopyFromRows(tokenRows)); err != nil {
return fmt.Errorf("classifier: copy tokens: %w", err)
}
}
if err := tx.Commit(ctx); err != nil {
return fmt.Errorf("classifier: commit tx: %w", err)
}
return nil
}
// ---------------------------------------------------------------------------
// Prediction
// ---------------------------------------------------------------------------
// classStat holds the persisted per-class statistics needed for scoring.
type classStat struct {
entityID int64
docCount int64
totalTokens int64
logLikeAt float64 // running log-likelihood accumulator (built during Predict)
}
// Predict scores the given text against the trained model for tenant/kind and
// returns the classes whose softmax posterior probability is at least
// SuggestionFloor, top maxCandidates, sorted by score descending. Each candidate
// carries its most decisive tokens as an explanation.
//
// Scoring is the standard multinomial Naive-Bayes log-likelihood with Laplace
// smoothing:
//
// logscore(c) = log P(c) + Σ_t freq(t) · log( (count(t,c)+α) / (Σtokens_c + α·V) )
//
// where V is the tenant/kind vocabulary size. The log-scores are then softmaxed
// (max-subtracted for numerical stability) into posterior probabilities so a
// single, interpretable confidence floor can be applied.
func (c *Classifier) Predict(ctx context.Context, tenantID int64, kind string, text string) ([]SuggestionCandidate, error) {
if err := validKind(kind); err != nil {
return nil, err
}
stats, totalDocs, err := c.loadClasses(ctx, tenantID, kind)
if err != nil {
return nil, err
}
if len(stats) == 0 || totalDocs == 0 {
return []SuggestionCandidate{}, nil // untrained kind: no suggestions, not an error
}
vocab, err := c.vocabSize(ctx, tenantID, kind)
if err != nil {
return nil, err
}
freq := tokenize(text)
if len(freq) == 0 {
return []SuggestionCandidate{}, nil
}
tokens := make([]string, 0, len(freq))
for t := range freq {
tokens = append(tokens, t)
}
// tokenCounts[token][entityID] = stored count.
tokenCounts, err := c.loadTokenCounts(ctx, tenantID, kind, tokens)
if err != nil {
return nil, err
}
// Per-token, per-class smoothed log-probability, plus per-class log score.
// perTokenLog[token][entityID] retained for the explanation step.
perTokenLog := make(map[string]map[int64]float64, len(tokens))
for i := range stats {
st := &stats[i]
st.logLikeAt = math.Log(float64(st.docCount) / float64(totalDocs)) // log prior
}
denom := make(map[int64]float64, len(stats))
for i := range stats {
st := &stats[i]
denom[st.entityID] = float64(st.totalTokens) + laplaceAlpha*float64(vocab)
}
for _, tok := range tokens {
perClass := tokenCounts[tok]
logs := make(map[int64]float64, len(stats))
for i := range stats {
st := &stats[i]
cnt := float64(perClass[st.entityID]) // 0 if unseen
logp := math.Log((cnt + laplaceAlpha) / denom[st.entityID])
logs[st.entityID] = logp
st.logLikeAt += float64(freq[tok]) * logp
}
perTokenLog[tok] = logs
}
// Softmax over the class log-scores.
scores := softmax(stats)
cands := make([]SuggestionCandidate, 0, len(stats))
// Rank runner-up for the explanation (second-highest posterior).
for i := range stats {
st := stats[i]
p := scores[st.entityID]
if p < SuggestionFloor {
continue
}
cands = append(cands, SuggestionCandidate{
EntityID: st.entityID,
Score: p,
TopTokens: decisiveTokens(st.entityID, stats, freq, perTokenLog),
})
}
sort.SliceStable(cands, func(i, j int) bool { return cands[i].Score > cands[j].Score })
if len(cands) > maxCandidates {
cands = cands[:maxCandidates]
}
return cands, nil
}
// softmax converts the per-class log scores into posterior probabilities,
// subtracting the max log score first for numerical stability.
func softmax(stats []classStat) map[int64]float64 {
maxLog := math.Inf(-1)
for i := range stats {
if stats[i].logLikeAt > maxLog {
maxLog = stats[i].logLikeAt
}
}
sum := 0.0
exp := make(map[int64]float64, len(stats))
for i := range stats {
e := math.Exp(stats[i].logLikeAt - maxLog)
exp[stats[i].entityID] = e
sum += e
}
out := make(map[int64]float64, len(stats))
if sum == 0 {
return out
}
for id, e := range exp {
out[id] = e / sum
}
return out
}
// decisiveTokens returns the (up to maxExplanationTokens) input tokens that most
// favoured winner over the strongest competing class, weighted by their
// frequency in the text. Positive margin = the token pushed toward winner.
func decisiveTokens(winner int64, stats []classStat, freq map[string]int, perTokenLog map[string]map[int64]float64) []string {
type scored struct {
token string
margin float64
}
out := make([]scored, 0, len(freq))
for tok, logs := range perTokenLog {
winLog, ok := logs[winner]
if !ok {
continue
}
// Best competing class's log-prob for this token.
competitor := math.Inf(-1)
for _, st := range stats {
if st.entityID == winner {
continue
}
if l := logs[st.entityID]; l > competitor {
competitor = l
}
}
if math.IsInf(competitor, -1) {
competitor = winLog // single-class case: no margin
}
margin := float64(freq[tok]) * (winLog - competitor)
if margin <= 0 {
continue
}
out = append(out, scored{token: tok, margin: margin})
}
sort.SliceStable(out, func(i, j int) bool { return out[i].margin > out[j].margin })
tokens := make([]string, 0, maxExplanationTokens)
for _, s := range out {
if len(tokens) >= maxExplanationTokens {
break
}
tokens = append(tokens, s.token)
}
return tokens
}
// loadClasses reads the persisted per-class stats for tenant/kind and the total
// document count across them (the denominator of the class priors).
func (c *Classifier) loadClasses(ctx context.Context, tenantID int64, kind string) ([]classStat, int64, error) {
rows, err := c.db.Query(ctx,
`SELECT entity_id, doc_count, total_tokens
FROM ml_classifier_classes
WHERE tenant_id = $1 AND kind = $2`, tenantID, kind)
if err != nil {
return nil, 0, fmt.Errorf("classifier: load classes: %w", err)
}
defer rows.Close()
var out []classStat
var totalDocs int64
for rows.Next() {
var st classStat
if err := rows.Scan(&st.entityID, &st.docCount, &st.totalTokens); err != nil {
return nil, 0, fmt.Errorf("classifier: scan class: %w", err)
}
totalDocs += st.docCount
out = append(out, st)
}
return out, totalDocs, rows.Err()
}
// vocabSize returns the number of distinct tokens in the tenant/kind model — the
// V in Laplace smoothing.
func (c *Classifier) vocabSize(ctx context.Context, tenantID int64, kind string) (int64, error) {
var v int64
if err := c.db.QueryRow(ctx,
`SELECT COUNT(DISTINCT token) FROM ml_classifier_tokens WHERE tenant_id = $1 AND kind = $2`,
tenantID, kind).Scan(&v); err != nil {
return 0, fmt.Errorf("classifier: vocab size: %w", err)
}
return v, nil
}
// loadTokenCounts fetches the per-class counts for exactly the input tokens
// (one query, token = ANY($3)), returning token -> entityID -> count.
func (c *Classifier) loadTokenCounts(ctx context.Context, tenantID int64, kind string, tokens []string) (map[string]map[int64]int64, error) {
out := make(map[string]map[int64]int64, len(tokens))
if len(tokens) == 0 {
return out, nil
}
rows, err := c.db.Query(ctx,
`SELECT token, entity_id, count
FROM ml_classifier_tokens
WHERE tenant_id = $1 AND kind = $2 AND token = ANY($3)`,
tenantID, kind, tokens)
if err != nil {
return nil, fmt.Errorf("classifier: load token counts: %w", err)
}
defer rows.Close()
for rows.Next() {
var tok string
var entityID, cnt int64
if err := rows.Scan(&tok, &entityID, &cnt); err != nil {
return nil, fmt.Errorf("classifier: scan token count: %w", err)
}
m := out[tok]
if m == nil {
m = make(map[int64]int64)
out[tok] = m
}
m[entityID] = cnt
}
return out, rows.Err()
}
+73
View File
@@ -0,0 +1,73 @@
// Package cryptutil provides authenticated symmetric encryption (AES-256-GCM)
// for secrets that must be stored at rest but read back in plaintext at
// runtime — currently the LDAP service-bind password (internal/ldapstore).
//
// The 256-bit key is derived via HKDF-SHA256 from the application's existing
// master/JWT secret (config.API.Secret), so no additional secret needs to be
// provisioned. A distinct HKDF info label keeps this key independent from the
// JWT signing key even though both originate from the same input secret.
package cryptutil
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"crypto/sha256"
"fmt"
"io"
"golang.org/x/crypto/hkdf"
)
// hkdfInfo domain-separates the secretbox key from every other key derived
// from the same master secret (e.g. the JWT signing key uses "archivdms-jwt-v1").
const hkdfInfo = "archivdms-ldap-secretbox-v1"
// Box performs AES-256-GCM encrypt/decrypt with a key derived from a secret.
type Box struct {
gcm cipher.AEAD
}
// NewBox derives a 256-bit AES key from secret via HKDF-SHA256 and returns a
// ready-to-use Box. secret must be non-empty.
func NewBox(secret string) (*Box, error) {
if secret == "" {
return nil, fmt.Errorf("cryptutil: empty secret")
}
key := make([]byte, 32)
if _, err := io.ReadFull(hkdf.New(sha256.New, []byte(secret), nil, []byte(hkdfInfo)), key); err != nil {
return nil, fmt.Errorf("cryptutil: derive key: %w", err)
}
block, err := aes.NewCipher(key)
if err != nil {
return nil, fmt.Errorf("cryptutil: new cipher: %w", err)
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return nil, fmt.Errorf("cryptutil: new gcm: %w", err)
}
return &Box{gcm: gcm}, nil
}
// Encrypt seals plaintext, returning the ciphertext and the freshly generated
// nonce (stored separately in the DB). Callers persist both.
func (b *Box) Encrypt(plaintext []byte) (ciphertext, nonce []byte, err error) {
nonce = make([]byte, b.gcm.NonceSize())
if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
return nil, nil, fmt.Errorf("cryptutil: nonce: %w", err)
}
ciphertext = b.gcm.Seal(nil, nonce, plaintext, nil)
return ciphertext, nonce, nil
}
// Decrypt opens ciphertext using nonce, returning the original plaintext.
func (b *Box) Decrypt(ciphertext, nonce []byte) ([]byte, error) {
if len(nonce) != b.gcm.NonceSize() {
return nil, fmt.Errorf("cryptutil: bad nonce length %d", len(nonce))
}
plaintext, err := b.gcm.Open(nil, nonce, ciphertext, nil)
if err != nil {
return nil, fmt.Errorf("cryptutil: decrypt: %w", err)
}
return plaintext, nil
}
+61
View File
@@ -0,0 +1,61 @@
// Package dateformat translates a small, user-friendly token pattern (e.g.
// "DD.MM.YYYY HH:mm") into a Go time layout ("02.01.2006 15:04"). It is a
// neutral package imported by both internal/api and internal/tenantstore so
// they can share one validator without importing each other (which would be a
// circular dependency).
//
// Admins may enter an arbitrary token string; anything that is not a known
// token is preserved verbatim as literal text in the resulting layout.
package dateformat
import (
"fmt"
"regexp"
"unicode/utf8"
)
// MaxPatternLen bounds the token pattern length, mirroring the scan-title
// prefix limit so an oversized value can never be persisted.
const MaxPatternLen = 40
// tokenLayout maps each supported token to its Go reference-time fragment.
// Order matters at match time (longest first) so e.g. "YYYY" is not consumed
// as "YY"+"YY"; the regex alternation below encodes that ordering explicitly.
var tokenLayout = map[string]string{
"AM/PM": "PM",
"YYYY": "2006",
"YY": "06",
"MM": "01",
"DD": "02",
"HH": "15",
"hh": "03",
"mm": "04",
"ss": "05",
"PM": "PM",
}
// tokenRE matches known tokens, longest alternatives first so a greedy leftmost
// match never splits a long token into shorter ones.
var tokenRE = regexp.MustCompile(`AM/PM|YYYY|YY|MM|DD|HH|hh|mm|ss|PM`)
// Translate converts a token pattern into a Go time layout. It fails when the
// pattern is empty, too long, or contains no recognised token (a pattern of
// pure literal text would make time.Format return that literal unchanged,
// which is never what the admin intends).
func Translate(pattern string) (goLayout string, err error) {
if pattern == "" {
return "", fmt.Errorf("Format darf nicht leer sein")
}
if utf8.RuneCountInString(pattern) > MaxPatternLen {
return "", fmt.Errorf("Format zu lang (max %d Zeichen)", MaxPatternLen)
}
matched := false
layout := tokenRE.ReplaceAllStringFunc(pattern, func(tok string) string {
matched = true
return tokenLayout[tok]
})
if !matched {
return "", fmt.Errorf("Format muss mindestens einen Datums-/Zeit-Platzhalter enthalten")
}
return layout, nil
}
+94
View File
@@ -0,0 +1,94 @@
// Package index is the (Phase 1) full-text search sync layer for archivdms.
//
// PostgreSQL remains the single source of truth; this package keeps a
// secondary, per-tenant Manticore Search index (Hybrid BM25+Vektor is a later
// phase) in sync with the documents table. Only the write/sync half is
// implemented here — there is deliberately NO search endpoint yet (Phase 2/3).
//
// Design guarantees:
// - The index is best-effort. When Manticore is not configured (empty DSN)
// the whole thing degrades to a no-op: the Indexer is nil and every caller
// skips silently.
// - An index error must NEVER be propagated to the originating HTTP request.
// Callers log and move on. Postgres stays authoritative, so a stale index
// is a recoverable, non-fatal condition (a later reindex CLI, Phase 3,
// rebuilds it).
//
// This package intentionally has NO dependency on internal/storage to avoid an
// import cycle: storage builds DocumentDoc values and calls into here.
package index
import (
"context"
"time"
)
// DocumentDoc is the index representation of a stored document. It is the
// projection of a documents row plus its resolved taxonomy (tags) and ACL
// (visibility group IDs) that the search index needs.
type DocumentDoc struct {
ID int64
TenantID int64
Title string
DocType string // deprecated free-text doc_type
Correspondent string // deprecated free-text correspondent
OCRText string
Tags []string
TagIDs []int64
DocTypeID *int64
CorrespondentID *int64
ACLGroupIDs []int64
RetainUntil *time.Time
CreatedAt time.Time
UpdatedAt time.Time
}
// SearchQuery is the (Phase 3) full-text + attribute query against a single
// tenant's index. It intentionally carries only what the index needs to return
// a ranked list of documents.id values; the caller re-hydrates the full
// document rows from Postgres (the source of truth) afterwards.
type SearchQuery struct {
// Query is the raw user full-text term. It is escaped before it ever
// reaches a MATCH() expression — callers pass it verbatim.
Query string
// TagIDs, when non-empty, restricts hits to documents carrying ANY of
// these tag ids (MVA filter).
TagIDs []int64
// DocTypeID, when non-nil, restricts hits to that document type.
DocTypeID *int64
// ACLGroupIDs applies the group-resolved document ACL: when non-nil, only
// documents visible to ANY of these permission groups are returned. A nil
// slice means "no ACL filter" (domain_admin/superadmin). An explicitly
// empty (non-nil) slice would match nothing — callers must short-circuit
// that case before querying.
ACLGroupIDs []int64
// Page is 1-based; PageSize caps hits per page.
Page int
PageSize int
}
// SearchHit is a single ranked result: a documents.id plus its BM25 score.
type SearchHit struct {
ID int64
Score float64
}
// Indexer syncs a single (tenant-scoped) document index. Implementations must
// never block or fail the calling request on transient backend errors beyond
// returning the error for the caller to log.
type Indexer interface {
// IndexSync inserts or replaces the document (id-based upsert).
IndexSync(ctx context.Context, doc DocumentDoc) error
// Delete removes the document from the index by its documents.id.
Delete(ctx context.Context, id int64) error
// Search runs a full-text + attribute query and returns the ranked hits
// for the requested page plus the total match count (across all pages).
Search(ctx context.Context, q SearchQuery) (hits []SearchHit, total int, err error)
}
// TenantIndexer hands out per-tenant Indexer instances, each backed by its own
// RT table (documents_tenant_<id>).
type TenantIndexer interface {
ForTenant(tenantID int64) Indexer
Close() error
}
+319
View File
@@ -0,0 +1,319 @@
package index
import (
"context"
"database/sql"
"fmt"
"regexp"
"strings"
"sync"
"time"
_ "github.com/go-sql-driver/mysql"
)
// validTableName guards against SQL injection through table-name
// interpolation: only documents_tenant_<digits> is ever a legal RT table.
var validTableName = regexp.MustCompile(`^documents_tenant_\d+$`)
// manticoreIndex implements Indexer against a single Manticore RT table.
type manticoreIndex struct {
db *sql.DB
table string
}
// ManticoreTenantManager implements TenantIndexer using Manticore Search via
// the MySQL wire protocol (port 9306 by default). No CGO required — pure Go
// through database/sql + github.com/go-sql-driver/mysql.
type ManticoreTenantManager struct {
db *sql.DB
mu sync.RWMutex
pool map[int64]*manticoreIndex
}
// NewManticoreTenantManager opens (and pings) a Manticore connection and
// returns a ready manager. Per-tenant RT tables are created lazily on first
// ForTenant use.
func NewManticoreTenantManager(dsn string) (*ManticoreTenantManager, error) {
db, err := sql.Open("mysql", dsn)
if err != nil {
return nil, fmt.Errorf("manticore: open: %w", err)
}
db.SetMaxOpenConns(16)
db.SetMaxIdleConns(4)
db.SetConnMaxLifetime(5 * time.Minute)
if err := db.Ping(); err != nil {
db.Close()
return nil, fmt.Errorf("manticore: ping: %w", err)
}
return &ManticoreTenantManager{
db: db,
pool: make(map[int64]*manticoreIndex),
}, nil
}
// ForTenant returns the Indexer for a tenant, creating its RT table on first
// use. If the table cannot be ensured, a no-op Indexer is returned so callers
// never panic or block — the miss is the caller's to log.
func (m *ManticoreTenantManager) ForTenant(tenantID int64) Indexer {
if tenantID <= 0 {
return noopIndexer{}
}
m.mu.RLock()
idx, ok := m.pool[tenantID]
m.mu.RUnlock()
if ok {
return idx
}
m.mu.Lock()
defer m.mu.Unlock()
if idx, ok = m.pool[tenantID]; ok {
return idx
}
idx = &manticoreIndex{db: m.db, table: manticoreTableName(tenantID)}
if err := idx.ensureTable(); err != nil {
return noopIndexer{}
}
m.pool[tenantID] = idx
return idx
}
// Close closes the shared database connection.
func (m *ManticoreTenantManager) Close() error {
return m.db.Close()
}
// ── manticoreIndex methods ────────────────────────────────────────────────
// ensureTable creates the RT index idempotently if it does not yet exist.
func (idx *manticoreIndex) ensureTable() error {
stmt := fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (
doc_id string,
title text,
doc_type text,
correspondent text,
ocr_text text,
tags text,
tag_ids multi,
doc_type_id bigint,
correspondent_id bigint,
acl_group_ids multi,
retain_until_ts bigint,
created_ts bigint,
updated_ts bigint,
deleted uint
) type='rt' morphology='lemmatize_de_all,stem_en'`, idx.table)
if _, err := idx.db.Exec(stmt); err != nil {
return fmt.Errorf("manticore: ensureTable %s: %w", idx.table, err)
}
return nil
}
// IndexSync upserts a document via REPLACE INTO (id-based, Manticore-typical).
//
// The two MVA (multi) columns tag_ids/acl_group_ids are interpolated inline
// because Manticore does not accept bind placeholders inside the (a,b,c) MVA
// value syntax. This is injection-safe: both lists are rendered from int64
// values only (joinInts), never from free text.
func (idx *manticoreIndex) IndexSync(ctx context.Context, doc DocumentDoc) error {
_, err := idx.db.ExecContext(ctx,
fmt.Sprintf(`REPLACE INTO %s
(id, doc_id, title, doc_type, correspondent, ocr_text, tags, tag_ids, doc_type_id, correspondent_id, acl_group_ids, retain_until_ts, created_ts, updated_ts, deleted)
VALUES (?,?,?,?,?,?,?,(%s),?,?,(%s),?,?,?,?)`, idx.table, joinInts(doc.TagIDs), joinInts(doc.ACLGroupIDs)),
doc.ID,
fmt.Sprintf("%d", doc.ID),
doc.Title,
doc.DocType,
doc.Correspondent,
doc.OCRText,
strings.Join(doc.Tags, " "),
ptrInt64(doc.DocTypeID),
ptrInt64(doc.CorrespondentID),
unixOrZero(doc.RetainUntil),
doc.CreatedAt.Unix(),
doc.UpdatedAt.Unix(),
0,
)
if err != nil {
return fmt.Errorf("manticore: IndexSync %s id=%d: %w", idx.table, doc.ID, err)
}
return nil
}
// Delete removes a document from the RT index by its documents.id.
func (idx *manticoreIndex) Delete(ctx context.Context, id int64) error {
_, err := idx.db.ExecContext(ctx,
fmt.Sprintf("DELETE FROM %s WHERE id = ?", idx.table), id)
if err != nil {
return fmt.Errorf("manticore: Delete %s id=%d: %w", idx.table, id, err)
}
return nil
}
// Search runs a full-text + attribute query against the RT index and returns
// the ranked hits for the requested page plus the overall match count.
//
// The full-text term (if any) is matched against the title, ocr_text, tags,
// correspondent and doc_type fields. It is escaped via escapeMatch before it
// is placed into a MATCH() expression — the only user-controlled string in the
// query; every other filter value is an int64 rendered inline (injection-safe)
// or a bound placeholder.
func (idx *manticoreIndex) Search(ctx context.Context, q SearchQuery) ([]SearchHit, int, error) {
var whereParts []string
var args []any
hasMatch := strings.TrimSpace(q.Query) != ""
if hasMatch {
whereParts = append(whereParts, "MATCH(?)")
args = append(args, "@(title,ocr_text,tags,correspondent,doc_type) "+escapeMatch(q.Query))
}
// Never return purged documents.
whereParts = append(whereParts, "deleted = 0")
// Attribute filters. MVA lists are rendered inline from int64 values only
// (joinInts) — Manticore rejects placeholders inside ANY(...) IN (...).
if len(q.TagIDs) > 0 {
whereParts = append(whereParts, fmt.Sprintf("ANY(tag_ids) IN (%s)", joinInts(q.TagIDs)))
}
if q.DocTypeID != nil {
whereParts = append(whereParts, "doc_type_id = ?")
args = append(args, *q.DocTypeID)
}
if q.ACLGroupIDs != nil {
// A non-nil but empty slice means "no visible groups" — match nothing.
if len(q.ACLGroupIDs) == 0 {
return nil, 0, nil
}
whereParts = append(whereParts, fmt.Sprintf("ANY(acl_group_ids) IN (%s)", joinInts(q.ACLGroupIDs)))
}
whereClause := ""
if len(whereParts) > 0 {
whereClause = "WHERE " + strings.Join(whereParts, " AND ")
}
// Total match count (across all pages) for pagination metadata.
countArgs := make([]any, len(args))
copy(countArgs, args)
countSQL := fmt.Sprintf("SELECT COUNT(*) FROM %s %s OPTION max_matches=1000000", idx.table, whereClause)
var total int
if err := idx.db.QueryRowContext(ctx, countSQL, countArgs...).Scan(&total); err != nil {
return nil, 0, fmt.Errorf("manticore: Search count %s: %w", idx.table, err)
}
pageSize := q.PageSize
if pageSize <= 0 {
pageSize = 20
}
page := q.Page
if page <= 0 {
page = 1
}
offset := (page - 1) * pageSize
scoreExpr := "1 as score"
orderBy := "created_ts DESC"
if hasMatch {
scoreExpr = "WEIGHT() as score"
orderBy = "WEIGHT() DESC, created_ts DESC"
}
selectSQL := fmt.Sprintf(
"SELECT id, %s FROM %s %s ORDER BY %s LIMIT ? OFFSET ? OPTION max_matches=10000",
scoreExpr, idx.table, whereClause, orderBy)
selectArgs := make([]any, len(args))
copy(selectArgs, args)
selectArgs = append(selectArgs, pageSize, offset)
rows, err := idx.db.QueryContext(ctx, selectSQL, selectArgs...)
if err != nil {
return nil, 0, fmt.Errorf("manticore: Search select %s: %w", idx.table, err)
}
defer rows.Close()
var hits []SearchHit
for rows.Next() {
var id int64
var score float64
if err := rows.Scan(&id, &score); err != nil {
return nil, 0, fmt.Errorf("manticore: Search scan %s: %w", idx.table, err)
}
hits = append(hits, SearchHit{ID: id, Score: score})
}
if err := rows.Err(); err != nil {
return nil, 0, fmt.Errorf("manticore: Search rows %s: %w", idx.table, err)
}
return hits, total, nil
}
// escapeMatch escapes characters that carry special meaning in a Manticore
// MATCH() expression, so a user-supplied full-text term can never inject
// operators (query-injection guard). Mirrors the established archivmail
// escapeManticoreMatch pattern.
func escapeMatch(s string) string {
const specials = `\()|!@~"/^$=<`
var b strings.Builder
b.Grow(len(s))
for _, c := range s {
if strings.ContainsRune(specials, c) {
b.WriteRune('\\')
}
b.WriteRune(c)
}
return b.String()
}
// ── helpers ────────────────────────────────────────────────────────────────
// manticoreTableName returns the RT table name for a tenant. Panics on an
// invalid result — that would be a programming error, not a runtime condition.
func manticoreTableName(tenantID int64) string {
name := fmt.Sprintf("documents_tenant_%d", tenantID)
if !validTableName.MatchString(name) {
panic(fmt.Sprintf("manticore: invalid table name: %q", name))
}
return name
}
// joinInts renders an int64 slice as a comma-separated list for a Manticore
// multi (MVA) column value, wrapped in parentheses by the caller's placeholder.
func joinInts(ids []int64) string {
if len(ids) == 0 {
return ""
}
parts := make([]string, len(ids))
for i, v := range ids {
parts[i] = fmt.Sprintf("%d", v)
}
return strings.Join(parts, ",")
}
func ptrInt64(p *int64) int64 {
if p == nil {
return 0
}
return *p
}
func unixOrZero(t *time.Time) int64 {
if t == nil || t.IsZero() {
return 0
}
return t.Unix()
}
// noopIndexer is returned when a tenant table cannot be ensured. Every method
// silently succeeds so a backend hiccup never blocks the calling request.
type noopIndexer struct{}
func (noopIndexer) IndexSync(context.Context, DocumentDoc) error { return nil }
func (noopIndexer) Delete(context.Context, int64) error { return nil }
func (noopIndexer) Search(context.Context, SearchQuery) ([]SearchHit, int, error) {
return nil, 0, nil
}
+258
View File
@@ -0,0 +1,258 @@
// Package jobqueue ist die Mandanten-faire Arbeitswarteschlange für die
// asynchrone Dokument-Nachverarbeitung (OCR, Taxonomie-Autozuordnung,
// on_upload-Workflows).
//
// Architektur (bewusst schlank):
//
// - Backend: Postgres-Tabelle processing_jobs (internal/storage/
// processing_jobs.go), KEIN Redis/AMQP. Job-Insert und documents-Insert
// laufen in derselben Transaktion, damit nie ein Dokument ohne Job
// entsteht.
// - Worker: Goroutinen IM SELBEN Backend-Prozess, kein separater Dienst und
// kein Container. Anzahl aus der Config (jobqueue.workers).
// - Fairness: der Dispatcher arbeitet Round-Robin über die Mandanten. Pro
// Runde wird je Mandant mit fälligen Jobs GENAU EINER gezogen
// (ClaimNextJobForTenant), erst danach beginnt die nächste Runde. Ein
// Mandant mit 5.000 Batch-Scans kann damit die Verarbeitung der anderen
// Mandanten verzögern, aber nicht aushungern (globales FIFO würde genau
// das tun).
// - Locking: FOR UPDATE SKIP LOCKED, dadurch können beliebig viele Worker
// (und theoretisch mehrere Prozesse) parallel ziehen, ohne dass ein Job
// doppelt läuft.
// - Reaper: hängengebliebene 'processing'-Jobs (Prozess-Neustart, toter
// OCR-Subprozess) werden nach jobqueue.job_timeout_seconds zurückgesetzt
// und mit exponentiellem Backoff neu eingeplant; ab max_retries bleiben
// sie dauerhaft 'failed' (kein Automatik-Retry mehr, manueller Retry ist
// Phase 3/Frontend).
//
// WORM bleibt außen vor: der Worker liest die archivierte Datei nur und
// schreibt ausschließlich abgeleitete Metadaten.
package jobqueue
import (
"context"
"errors"
"fmt"
"log/slog"
"sync"
"time"
"archivdms/config"
"archivdms/internal/storage"
)
// ProcessFunc verarbeitet einen einzelnen Job. Implementiert von
// internal/api.(*Server).ProcessDocumentJob und als Funktionswert
// hereingereicht (statt internal/api zu importieren) — dasselbe Muster wie
// sftpserver.UploadFunc, um einen Import-Zyklus zu vermeiden. Die
// Verdrahtung passiert in cmd/archivdms/main.go.
type ProcessFunc func(ctx context.Context, tenantID, documentID int64, deriveTitle bool) error
// Dispatcher zieht Jobs Round-Robin über die Mandanten und verteilt sie an
// einen Pool von Worker-Goroutinen.
type Dispatcher struct {
cfg config.JobQueueConfig
store *storage.Store
process ProcessFunc
logger *slog.Logger
jobs chan *storage.ProcessingJob
stopOnce sync.Once
stopCh chan struct{}
wg sync.WaitGroup
}
// New erzeugt einen Dispatcher. Start startet Worker, Dispatch-Loop und
// Reaper; Stop fährt alles sauber herunter.
func New(cfg config.JobQueueConfig, store *storage.Store, process ProcessFunc, logger *slog.Logger) *Dispatcher {
return &Dispatcher{
cfg: cfg,
store: store,
process: process,
logger: logger,
// Ungepuffert: der Dispatch-Loop blockiert, solange alle Worker
// beschäftigt sind. Genau erwünscht — so werden nur so viele Jobs auf
// 'processing' gesetzt, wie auch tatsächlich gerade laufen können, und
// ein Prozess-Neustart lässt keine unnötig große Menge Jobs im
// Reaper-Timeout hängen.
jobs: make(chan *storage.ProcessingJob),
stopCh: make(chan struct{}),
}
}
// Start startet den Worker-Pool, den Round-Robin-Dispatch-Loop und den
// Reaper. Nicht blockierend.
func (d *Dispatcher) Start(ctx context.Context) {
workers := d.cfg.ResolvedWorkers()
for i := 0; i < workers; i++ {
d.wg.Add(1)
go d.worker(ctx, i+1)
}
d.wg.Add(2)
go d.dispatchLoop(ctx)
go d.reapLoop(ctx)
d.logger.Info("job queue started",
"workers", workers,
"poll_interval", d.cfg.ResolvedPollInterval(),
"job_timeout", d.cfg.ResolvedJobTimeout(),
"max_retries", d.cfg.ResolvedMaxRetries())
}
// Stop signalisiert allen Goroutinen das Ende und wartet auf sie.
func (d *Dispatcher) Stop() {
d.stopOnce.Do(func() { close(d.stopCh) })
d.wg.Wait()
}
// dispatchLoop pollt die Queue und verteilt Jobs Round-Robin über Mandanten.
func (d *Dispatcher) dispatchLoop(ctx context.Context) {
defer d.wg.Done()
ticker := time.NewTicker(d.cfg.ResolvedPollInterval())
defer ticker.Stop()
for {
select {
case <-d.stopCh:
close(d.jobs)
return
case <-ctx.Done():
close(d.jobs)
return
case <-ticker.C:
d.dispatchRound(ctx)
}
}
}
// dispatchRound führt so lange Round-Robin-Runden aus, wie noch Mandanten
// mit fälligen Jobs übrig sind. Pro Runde wird je Mandant genau ein Job
// gezogen und an den Worker-Pool übergeben — dadurch wechseln sich die
// Mandanten ab, statt dass Mandant A komplett leergeräumt wird, bevor
// Mandant B drankommt.
func (d *Dispatcher) dispatchRound(ctx context.Context) {
for {
tenants, err := d.store.TenantsWithDueJobs(ctx)
if err != nil {
d.logger.Warn("job queue: listing tenants with due jobs failed", "err", err)
return
}
if len(tenants) == 0 {
return
}
dispatched := 0
for _, tenantID := range tenants {
select {
case <-d.stopCh:
return
case <-ctx.Done():
return
default:
}
job, err := d.store.ClaimNextJobForTenant(ctx, tenantID)
if err != nil {
if errors.Is(err, storage.ErrNoJob) {
continue // Runde hat sich zwischenzeitlich erledigt
}
d.logger.Warn("job queue: claim failed", "tenant_id", tenantID, "err", err)
continue
}
select {
case d.jobs <- job:
dispatched++
case <-d.stopCh:
// Beim Herunterfahren den bereits geclaimten Job nicht
// verlieren: sofort wieder einreihen (retry_count bleibt
// unangetastet), sonst müsste erst der Reaper-Timeout
// ablaufen.
if rerr := d.store.RequeueJob(context.Background(), job.ID, job.TenantID); rerr != nil {
d.logger.Warn("job queue: requeue on shutdown failed", "job_id", job.ID, "err", rerr)
}
return
case <-ctx.Done():
if rerr := d.store.RequeueJob(context.Background(), job.ID, job.TenantID); rerr != nil {
d.logger.Warn("job queue: requeue on shutdown failed", "job_id", job.ID, "err", rerr)
}
return
}
}
if dispatched == 0 {
return
}
}
}
// worker verarbeitet Jobs aus dem Kanal, einer nach dem anderen.
func (d *Dispatcher) worker(ctx context.Context, num int) {
defer d.wg.Done()
for job := range d.jobs {
d.runJob(ctx, num, job)
}
}
func (d *Dispatcher) runJob(ctx context.Context, worker int, job *storage.ProcessingJob) {
started := time.Now()
// Eigener Timeout je Job, damit ein hängender OCR-Subprozess einen Worker
// nicht dauerhaft belegt. Bewusst NICHT vom Request-Kontext abgeleitet —
// die Verarbeitung ist vom Upload-Request entkoppelt.
jobCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), d.cfg.ResolvedJobTimeout())
defer cancel()
err := d.process(jobCtx, job.TenantID, job.DocumentID, job.DeriveTitle)
if err != nil {
requeued, mErr := d.store.MarkJobFailed(context.WithoutCancel(ctx), job.ID, job.TenantID, job.DocumentID, err.Error(), d.cfg.ResolvedMaxRetries())
if mErr != nil {
d.logger.Error("job queue: recording job failure failed", "job_id", job.ID, "err", mErr)
}
d.logger.Warn("job queue: job failed",
"worker", worker, "job_id", job.ID, "tenant_id", job.TenantID, "document_id", job.DocumentID,
"retry_count", job.RetryCount, "will_retry", requeued, "duration", time.Since(started), "err", err)
return
}
if err := d.store.MarkJobDone(context.WithoutCancel(ctx), job.ID, job.TenantID, job.DocumentID); err != nil {
d.logger.Error("job queue: marking job done failed", "job_id", job.ID, "err", err)
return
}
d.logger.Info("job queue: job done",
"worker", worker, "job_id", job.ID, "tenant_id", job.TenantID, "document_id", job.DocumentID,
"duration", time.Since(started))
}
// reapLoop setzt regelmäßig hängengebliebene 'processing'-Jobs zurück.
func (d *Dispatcher) reapLoop(ctx context.Context) {
defer d.wg.Done()
interval := d.cfg.ResolvedJobTimeout() / 2
if interval < 5*time.Second {
interval = 5 * time.Second
}
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-d.stopCh:
return
case <-ctx.Done():
return
case <-ticker.C:
n, err := d.store.ReapStaleJobs(ctx, d.cfg.ResolvedJobTimeout(), d.cfg.ResolvedMaxRetries())
if err != nil {
d.logger.Warn("job queue: reaper failed", "err", err)
continue
}
if n > 0 {
d.logger.Warn("job queue: reset stale processing jobs", "count", n, "timeout", d.cfg.ResolvedJobTimeout())
}
}
}
}
// String beschreibt die aktive Konfiguration (Diagnose-/Log-Hilfe).
func (d *Dispatcher) String() string {
return fmt.Sprintf("jobqueue(workers=%d poll=%s timeout=%s max_retries=%d)",
d.cfg.ResolvedWorkers(), d.cfg.ResolvedPollInterval(), d.cfg.ResolvedJobTimeout(), d.cfg.ResolvedMaxRetries())
}
+239
View File
@@ -0,0 +1,239 @@
// Package ldapauth implements the LDAP bind/search authentication flow against
// a per-tenant directory (config from internal/ldapstore). It uses
// github.com/go-ldap/ldap/v3 (pure Go, CGO_ENABLED=0 compatible).
//
// Flow (Authenticate):
// 1. Connect over LDAPS or StartTLS (cleartext LDAP is rejected).
// 2. Service-bind with bind_dn + decrypted bind password.
// 3. Search base_dn with user_filter, loginName escaped per RFC 4515 to
// prevent LDAP filter injection; expect exactly one entry.
// 4. Re-bind as the found user DN with the user-supplied password
// (this is the actual credential check — no fallback to a local password).
// 5. Optionally search the group tree to decide admin group membership,
// which the caller maps to a role.
package ldapauth
import (
"context"
"crypto/tls"
"fmt"
"net"
"strings"
"time"
"github.com/go-ldap/ldap/v3"
"archivdms/internal/ldapstore"
)
// Result is the outcome of a successful authentication.
type Result struct {
// Username is the attr_username value from the directory (used as the
// local username / ldap_uid on JIT provisioning).
Username string
// Email is the attr_email value.
Email string
// DisplayName is the attr_name value.
DisplayName string
// UserDN is the distinguished name the user bound with.
UserDN string
// IsAdmin is true when admin_group_dn is configured and the user is a
// member of it — mapped by the caller to domain_admin.
IsAdmin bool
}
// Authenticator performs LDAP authentication. It is stateless apart from a
// dial timeout, so a single instance can be shared across requests.
type Authenticator struct {
dialTimeout time.Duration
}
// New returns an Authenticator with the given dial/connect timeout (<=0 uses
// a 10s default).
func New(dialTimeout time.Duration) *Authenticator {
if dialTimeout <= 0 {
dialTimeout = 10 * time.Second
}
return &Authenticator{dialTimeout: dialTimeout}
}
// connect opens a TLS-protected LDAP connection according to cfg.UseTLS.
func (a *Authenticator) connect(cfg *ldapstore.Config) (*ldap.Conn, error) {
tlsCfg := &tls.Config{ServerName: cfg.Host, MinVersion: tls.VersionTLS12}
addr := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port)
dialer := &net.Dialer{Timeout: a.dialTimeout}
switch cfg.UseTLS {
case ldapstore.TLSModeLDAPS:
conn, err := ldap.DialURL("ldaps://"+addr, ldap.DialWithTLSConfig(tlsCfg), ldap.DialWithDialer(dialer))
if err != nil {
return nil, fmt.Errorf("ldapauth: dial ldaps: %w", err)
}
return conn, nil
case ldapstore.TLSModeStartTLS:
conn, err := ldap.DialURL("ldap://"+addr, ldap.DialWithDialer(dialer))
if err != nil {
return nil, fmt.Errorf("ldapauth: dial ldap: %w", err)
}
if err := conn.StartTLS(tlsCfg); err != nil {
conn.Close()
return nil, fmt.Errorf("ldapauth: starttls: %w", err)
}
return conn, nil
default:
return nil, fmt.Errorf("ldapauth: cleartext LDAP not permitted (use_tls=%q)", cfg.UseTLS)
}
}
// TestConnection performs only the service-bind and a base-DN search — it does
// NOT attempt a user login. Returns the round-trip latency.
func (a *Authenticator) TestConnection(ctx context.Context, cfg *ldapstore.Config, bindPassword string) (time.Duration, error) {
start := time.Now()
conn, err := a.connect(cfg)
if err != nil {
return 0, err
}
defer conn.Close()
conn.SetTimeout(a.dialTimeout)
if err := conn.Bind(cfg.BindDN, bindPassword); err != nil {
return 0, fmt.Errorf("ldapauth: service bind failed: %w", err)
}
// Minimal base-scope search to confirm base_dn is reachable/valid.
req := ldap.NewSearchRequest(
cfg.BaseDN, ldap.ScopeBaseObject, ldap.NeverDerefAliases, 1, int(a.dialTimeout.Seconds()), false,
"(objectClass=*)", []string{"dn"}, nil,
)
if _, err := conn.Search(req); err != nil {
return 0, fmt.Errorf("ldapauth: base search failed: %w", err)
}
return time.Since(start), nil
}
// Authenticate runs the full bind/search/re-bind flow.
func (a *Authenticator) Authenticate(ctx context.Context, cfg *ldapstore.Config, bindPassword, loginName, userPassword string) (*Result, error) {
if userPassword == "" {
// Prevent LDAP "unauthenticated bind" (empty password = anonymous success).
return nil, fmt.Errorf("ldapauth: empty password")
}
conn, err := a.connect(cfg)
if err != nil {
return nil, err
}
defer conn.Close()
conn.SetTimeout(a.dialTimeout)
// 1) Service bind.
if err := conn.Bind(cfg.BindDN, bindPassword); err != nil {
return nil, fmt.Errorf("ldapauth: service bind failed: %w", err)
}
// 2) Search for the user. loginName escaped against filter injection.
filter := strings.ReplaceAll(cfg.UserFilter, "%s", ldap.EscapeFilter(loginName))
attrs := []string{"dn", cfg.AttrUsername, cfg.AttrEmail, cfg.AttrName}
searchReq := ldap.NewSearchRequest(
cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases, 2, int(a.dialTimeout.Seconds()), false,
filter, attrs, nil,
)
sr, err := conn.Search(searchReq)
if err != nil {
return nil, fmt.Errorf("ldapauth: user search failed: %w", err)
}
if len(sr.Entries) == 0 {
return nil, fmt.Errorf("ldapauth: user not found")
}
if len(sr.Entries) > 1 {
return nil, fmt.Errorf("ldapauth: user filter not unique (%d entries)", len(sr.Entries))
}
entry := sr.Entries[0]
res := &Result{
UserDN: entry.DN,
Username: firstNonEmpty(entry.GetAttributeValue(cfg.AttrUsername), loginName),
Email: entry.GetAttributeValue(cfg.AttrEmail),
DisplayName: entry.GetAttributeValue(cfg.AttrName),
}
// 3) Re-bind as the user to verify the password.
if err := conn.Bind(entry.DN, userPassword); err != nil {
return nil, fmt.Errorf("ldapauth: invalid credentials")
}
// 4) Group membership for admin role mapping. Re-bind as service account
// first (the user account may lack read rights on the group tree).
if cfg.AdminGroupDN != "" {
if err := conn.Bind(cfg.BindDN, bindPassword); err != nil {
return nil, fmt.Errorf("ldapauth: re-bind for group search failed: %w", err)
}
isAdmin, err := a.isAdminMember(conn, cfg, res)
if err != nil {
return nil, err
}
res.IsAdmin = isAdmin
}
return res, nil
}
// isAdminMember checks whether the authenticated user belongs to admin_group_dn.
// Two strategies are supported:
// - group_base_dn + group_filter set: search the group tree with a filter
// where %s is replaced by the user DN (escaped), then check whether the
// admin_group_dn is among the returned group DNs.
// - otherwise: a base-scope search of admin_group_dn testing the standard
// member/uniqueMember/memberUid attributes against the user.
func (a *Authenticator) isAdminMember(conn *ldap.Conn, cfg *ldapstore.Config, res *Result) (bool, error) {
if cfg.GroupBaseDN != "" && cfg.GroupFilter != "" {
filter := cfg.GroupFilter
filter = strings.ReplaceAll(filter, "%d", ldap.EscapeFilter(res.UserDN))
filter = strings.ReplaceAll(filter, "%s", ldap.EscapeFilter(res.Username))
req := ldap.NewSearchRequest(
cfg.GroupBaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases, 0, int(a.dialTimeout.Seconds()), false,
filter, []string{"dn"}, nil,
)
sr, err := conn.Search(req)
if err != nil {
return false, fmt.Errorf("ldapauth: group search failed: %w", err)
}
for _, e := range sr.Entries {
if strings.EqualFold(strings.TrimSpace(e.DN), strings.TrimSpace(cfg.AdminGroupDN)) {
return true, nil
}
}
return false, nil
}
// Fallback: inspect the admin group entry directly.
req := ldap.NewSearchRequest(
cfg.AdminGroupDN, ldap.ScopeBaseObject, ldap.NeverDerefAliases, 0, int(a.dialTimeout.Seconds()), false,
"(objectClass=*)", []string{"member", "uniqueMember", "memberUid"}, nil,
)
sr, err := conn.Search(req)
if err != nil {
return false, fmt.Errorf("ldapauth: admin group lookup failed: %w", err)
}
if len(sr.Entries) == 0 {
return false, nil
}
e := sr.Entries[0]
for _, dn := range append(e.GetAttributeValues("member"), e.GetAttributeValues("uniqueMember")...) {
if strings.EqualFold(strings.TrimSpace(dn), strings.TrimSpace(res.UserDN)) {
return true, nil
}
}
for _, uid := range e.GetAttributeValues("memberUid") {
if strings.EqualFold(strings.TrimSpace(uid), strings.TrimSpace(res.Username)) {
return true, nil
}
}
return false, nil
}
func firstNonEmpty(vals ...string) string {
for _, v := range vals {
if strings.TrimSpace(v) != "" {
return v
}
}
return ""
}
+263
View File
@@ -0,0 +1,263 @@
// Package ldapstore is a PostgreSQL-backed CRUD store for per-tenant LDAP
// directory configuration (ldap_configs table). It follows the same
// Store-per-schema pattern as userstore/tenantstore: initSchema() is idempotent
// and called from New().
//
// The LDAP service-bind password is never stored in plaintext: it is encrypted
// with AES-256-GCM via internal/cryptutil (key derived from the application
// master secret) and stored as ciphertext + nonce. Get() returns the config
// WITHOUT the password (only HasBindPassword); GetWithSecret() decrypts it for
// the actual bind at login time.
package ldapstore
import (
"context"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"archivdms/internal/cryptutil"
)
// ErrNotFound is returned when no ldap_configs row exists for a tenant.
var ErrNotFound = errors.New("ldapstore: config not found")
// TLS mode values for Config.UseTLS.
const (
TLSModeLDAPS = "ldaps"
TLSModeStartTLS = "starttls"
)
// Config mirrors a row of ldap_configs, minus the encrypted password columns.
// HasBindPassword reports whether a bind password is stored (surfaced to the
// API as "is_set"); the plaintext is only ever available via GetWithSecret.
type Config struct {
ID int64 `json:"id"`
TenantID int64 `json:"tenant_id"`
Enabled bool `json:"enabled"`
Host string `json:"host"`
Port int `json:"port"`
UseTLS string `json:"use_tls"`
BindDN string `json:"bind_dn"`
HasBindPassword bool `json:"bind_password_set"`
BaseDN string `json:"base_dn"`
UserFilter string `json:"user_filter"`
AttrUsername string `json:"attr_username"`
AttrEmail string `json:"attr_email"`
AttrName string `json:"attr_name"`
GroupBaseDN string `json:"group_base_dn"`
GroupFilter string `json:"group_filter"`
AdminGroupDN string `json:"admin_group_dn"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// Store is a PostgreSQL-backed LDAP config store.
type Store struct {
pool *pgxpool.Pool
box *cryptutil.Box
}
// New connects to PostgreSQL, initialises the schema, and derives the
// password-encryption key from secret (the application master/JWT secret).
func New(dsn, secret string) (*Store, error) {
ctx := context.Background()
box, err := cryptutil.NewBox(secret)
if err != nil {
return nil, fmt.Errorf("ldapstore: crypto init: %w", err)
}
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("ldapstore: connect: %w", err)
}
s := &Store{pool: pool, box: box}
if err := s.initSchema(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ldapstore: init schema: %w", err)
}
return s, nil
}
// initSchema creates ldap_configs and adds the LDAP columns to users.
// Idempotent. Documented in migrations/011_ldap.sql.
//
// No FK on tenant_id: consistent with the rest of the schema (documents /
// permissions use a plain BIGINT tenant_id) and required because tenants/users
// are created by other stores whose init order relative to this one is not
// guaranteed (see cmd/archivdms/main.go).
func (s *Store) initSchema(ctx context.Context) error {
_, err := s.pool.Exec(ctx, `
CREATE TABLE IF NOT EXISTS ldap_configs (
id BIGSERIAL PRIMARY KEY,
tenant_id BIGINT NOT NULL UNIQUE,
enabled BOOLEAN NOT NULL DEFAULT false,
host VARCHAR(255) NOT NULL,
port INTEGER NOT NULL DEFAULT 636,
use_tls VARCHAR(20) NOT NULL DEFAULT 'ldaps',
bind_dn VARCHAR(500) NOT NULL,
bind_password_enc BYTEA NOT NULL,
bind_password_nonce BYTEA NOT NULL,
base_dn VARCHAR(500) NOT NULL,
user_filter VARCHAR(500) NOT NULL DEFAULT '(uid=%s)',
attr_username VARCHAR(100) NOT NULL DEFAULT 'uid',
attr_email VARCHAR(100) NOT NULL DEFAULT 'mail',
attr_name VARCHAR(100) NOT NULL DEFAULT 'cn',
group_base_dn VARCHAR(500),
group_filter VARCHAR(500),
admin_group_dn VARCHAR(500),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
ALTER TABLE users ADD COLUMN IF NOT EXISTS auth_source VARCHAR(20) NOT NULL DEFAULT 'local';
ALTER TABLE users ADD COLUMN IF NOT EXISTS ldap_uid VARCHAR(255);
ALTER TABLE users ADD COLUMN IF NOT EXISTS ldap_synced_at TIMESTAMPTZ;
CREATE UNIQUE INDEX IF NOT EXISTS idx_users_tenant_ldap_uid ON users (tenant_id, ldap_uid) WHERE ldap_uid IS NOT NULL;
`)
return err
}
// Close closes the underlying connection pool.
func (s *Store) Close() error {
s.pool.Close()
return nil
}
const selectCols = `id, tenant_id, enabled, host, port, use_tls, bind_dn,
(octet_length(bind_password_enc) > 0) AS has_pw,
base_dn, user_filter, attr_username, attr_email, attr_name,
COALESCE(group_base_dn, ''), COALESCE(group_filter, ''), COALESCE(admin_group_dn, ''),
created_at, updated_at`
func scanConfig(row pgx.Row) (*Config, error) {
var c Config
err := row.Scan(
&c.ID, &c.TenantID, &c.Enabled, &c.Host, &c.Port, &c.UseTLS, &c.BindDN,
&c.HasBindPassword, &c.BaseDN, &c.UserFilter, &c.AttrUsername, &c.AttrEmail, &c.AttrName,
&c.GroupBaseDN, &c.GroupFilter, &c.AdminGroupDN, &c.CreatedAt, &c.UpdatedAt,
)
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrNotFound
}
if err != nil {
return nil, fmt.Errorf("ldapstore: scan: %w", err)
}
return &c, nil
}
// Get returns the LDAP config for a tenant WITHOUT the bind password.
func (s *Store) Get(ctx context.Context, tenantID int64) (*Config, error) {
row := s.pool.QueryRow(ctx, `SELECT `+selectCols+` FROM ldap_configs WHERE tenant_id = $1`, tenantID)
return scanConfig(row)
}
// GetWithSecret returns the LDAP config together with the decrypted bind
// password. Only used at login/test time — never surfaced to API responses.
func (s *Store) GetWithSecret(ctx context.Context, tenantID int64) (*Config, string, error) {
cfg, err := s.Get(ctx, tenantID)
if err != nil {
return nil, "", err
}
var enc, nonce []byte
err = s.pool.QueryRow(ctx,
`SELECT bind_password_enc, bind_password_nonce FROM ldap_configs WHERE tenant_id = $1`, tenantID,
).Scan(&enc, &nonce)
if err != nil {
return nil, "", fmt.Errorf("ldapstore: read secret: %w", err)
}
pw, err := s.box.Decrypt(enc, nonce)
if err != nil {
return nil, "", fmt.Errorf("ldapstore: decrypt bind password: %w", err)
}
return cfg, string(pw), nil
}
// Upsert creates or updates the LDAP config for cfg.TenantID.
//
// newPassword semantics:
// - non-nil: the bind password is (re)encrypted and stored.
// - nil on an existing row: the stored password is kept unchanged.
// - nil on a new row: an error is returned (a bind password is mandatory).
func (s *Store) Upsert(ctx context.Context, cfg Config, newPassword *string) (*Config, error) {
if cfg.UseTLS != TLSModeLDAPS && cfg.UseTLS != TLSModeStartTLS {
return nil, fmt.Errorf("ldapstore: use_tls must be %q or %q (cleartext LDAP not permitted)", TLSModeLDAPS, TLSModeStartTLS)
}
if cfg.Port == 0 {
cfg.Port = 636
}
// Determine the password bytes to store.
var enc, nonce []byte
_, existing, existErr := s.GetWithSecret(ctx, cfg.TenantID)
switch {
case newPassword != nil:
var err error
enc, nonce, err = s.box.Encrypt([]byte(*newPassword))
if err != nil {
return nil, fmt.Errorf("ldapstore: encrypt bind password: %w", err)
}
case existErr == nil:
// Keep the existing password — re-encrypt to get fresh bytes.
var err error
enc, nonce, err = s.box.Encrypt([]byte(existing))
if err != nil {
return nil, fmt.Errorf("ldapstore: re-encrypt bind password: %w", err)
}
default:
return nil, fmt.Errorf("ldapstore: bind password required for new config")
}
nullable := func(s string) any {
if s == "" {
return nil
}
return s
}
_, err := s.pool.Exec(ctx, `
INSERT INTO ldap_configs
(tenant_id, enabled, host, port, use_tls, bind_dn, bind_password_enc, bind_password_nonce,
base_dn, user_filter, attr_username, attr_email, attr_name,
group_base_dn, group_filter, admin_group_dn, created_at, updated_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16, NOW(), NOW())
ON CONFLICT (tenant_id) DO UPDATE SET
enabled = EXCLUDED.enabled,
host = EXCLUDED.host,
port = EXCLUDED.port,
use_tls = EXCLUDED.use_tls,
bind_dn = EXCLUDED.bind_dn,
bind_password_enc = EXCLUDED.bind_password_enc,
bind_password_nonce = EXCLUDED.bind_password_nonce,
base_dn = EXCLUDED.base_dn,
user_filter = EXCLUDED.user_filter,
attr_username = EXCLUDED.attr_username,
attr_email = EXCLUDED.attr_email,
attr_name = EXCLUDED.attr_name,
group_base_dn = EXCLUDED.group_base_dn,
group_filter = EXCLUDED.group_filter,
admin_group_dn = EXCLUDED.admin_group_dn,
updated_at = NOW()`,
cfg.TenantID, cfg.Enabled, cfg.Host, cfg.Port, cfg.UseTLS, cfg.BindDN, enc, nonce,
cfg.BaseDN, cfg.UserFilter, cfg.AttrUsername, cfg.AttrEmail, cfg.AttrName,
nullable(cfg.GroupBaseDN), nullable(cfg.GroupFilter), nullable(cfg.AdminGroupDN),
)
if err != nil {
return nil, fmt.Errorf("ldapstore: upsert: %w", err)
}
return s.Get(ctx, cfg.TenantID)
}
// Delete removes the LDAP config for a tenant. Returns ErrNotFound if absent.
func (s *Store) Delete(ctx context.Context, tenantID int64) error {
tag, err := s.pool.Exec(ctx, `DELETE FROM ldap_configs WHERE tenant_id = $1`, tenantID)
if err != nil {
return fmt.Errorf("ldapstore: delete: %w", err)
}
if tag.RowsAffected() == 0 {
return ErrNotFound
}
return nil
}
+166
View File
@@ -0,0 +1,166 @@
// Package llm is a minimal HTTP client for an EXTERNAL, already-running Ollama
// server (never installed on the archivdms host — the base URL is provided per
// tenant, see internal/storage/ollama_config.go). It intentionally does no
// retrying, no connection pooling and no streaming: a single direct call to
// Ollama's /api/generate endpoint, CGO-free, net/http only.
package llm
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// generateRequest is the JSON body POSTed to <base_url>/api/generate. stream is
// always false (we want the whole answer at once) and format is "json" so the
// model is nudged to emit valid JSON in the response field.
type generateRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
Stream bool `json:"stream"`
Format string `json:"format"`
}
// generateResponse is the (non-streaming) envelope Ollama returns; the actual
// model output is the Response string, which — because we requested
// format=json — is itself a JSON document the caller parses structurally.
type generateResponse struct {
Response string `json:"response"`
Done bool `json:"done"`
Error string `json:"error"`
}
// tagsResponse is the JSON envelope Ollama returns from GET /api/tags: a list
// of the models installed on that server. Only the name is consumed here.
type tagsResponse struct {
Models []struct {
Name string `json:"name"`
} `json:"models"`
}
// ListModels performs a single blocking GET /api/tags call against the given
// Ollama base URL and returns the names of the models installed on that server,
// so the frontend can offer a picklist instead of a free-text model field. Any
// network error, timeout or non-200 status yields a clear error — there is NO
// silent fallback. The returned slice is always non-nil (make, never nil) so it
// JSON-encodes as [] rather than null.
func ListModels(ctx context.Context, baseURL string, timeout time.Duration) ([]string, error) {
baseURL = strings.TrimRight(strings.TrimSpace(baseURL), "/")
if baseURL == "" {
return nil, fmt.Errorf("llm: ollama base_url is empty")
}
if timeout <= 0 {
timeout = 10 * time.Second
}
reqCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
req, err := http.NewRequestWithContext(reqCtx, http.MethodGet, baseURL+"/api/tags", nil)
if err != nil {
return nil, fmt.Errorf("llm: build tags request: %w", err)
}
client := &http.Client{Timeout: timeout}
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("llm: ollama tags request failed: %w", err)
}
defer resp.Body.Close()
raw, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, fmt.Errorf("llm: read ollama tags response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("llm: ollama returned status %d: %s", resp.StatusCode, strings.TrimSpace(string(raw)))
}
var env tagsResponse
if err := json.Unmarshal(raw, &env); err != nil {
return nil, fmt.Errorf("llm: parse ollama tags envelope: %w", err)
}
out := make([]string, 0, len(env.Models))
for _, m := range env.Models {
if name := strings.TrimSpace(m.Name); name != "" {
out = append(out, name)
}
}
return out, nil
}
// GenerateJSON performs a single blocking /api/generate call against the given
// Ollama base URL and returns the model's inner `response` field as a
// json.RawMessage (the caller unmarshals it into its own schema). Any network
// error, timeout, non-200 status, Ollama-reported error or empty/invalid outer
// response yields a clear error — there is NO silent fallback, so the caller
// can report to the frontend exactly that Ollama did not answer.
func GenerateJSON(ctx context.Context, baseURL, model string, timeout time.Duration, prompt string) (json.RawMessage, error) {
baseURL = strings.TrimRight(strings.TrimSpace(baseURL), "/")
if baseURL == "" {
return nil, fmt.Errorf("llm: ollama base_url is empty")
}
if model == "" {
return nil, fmt.Errorf("llm: ollama model is empty")
}
if timeout <= 0 {
timeout = 30 * time.Second
}
body, err := json.Marshal(generateRequest{
Model: model,
Prompt: prompt,
Stream: false,
Format: "json",
})
if err != nil {
return nil, fmt.Errorf("llm: marshal request: %w", err)
}
reqCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
req, err := http.NewRequestWithContext(reqCtx, http.MethodPost, baseURL+"/api/generate", bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("llm: build request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: timeout}
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("llm: ollama request failed: %w", err)
}
defer resp.Body.Close()
// Cap the read so a misbehaving/unexpected endpoint cannot exhaust memory.
raw, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, fmt.Errorf("llm: read ollama response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("llm: ollama returned status %d: %s", resp.StatusCode, strings.TrimSpace(string(raw)))
}
var env generateResponse
if err := json.Unmarshal(raw, &env); err != nil {
return nil, fmt.Errorf("llm: parse ollama envelope: %w", err)
}
if env.Error != "" {
return nil, fmt.Errorf("llm: ollama error: %s", env.Error)
}
inner := strings.TrimSpace(env.Response)
if inner == "" {
return nil, fmt.Errorf("llm: ollama returned an empty response")
}
if !json.Valid([]byte(inner)) {
return nil, fmt.Errorf("llm: ollama response field is not valid JSON")
}
return json.RawMessage(inner), nil
}
+162
View File
@@ -0,0 +1,162 @@
// Package mailer sends transactional emails via an outbound SMTP relay,
// ported 1:1 from archivmail's internal/mailer (it has no mail-archiving
// specifics — it is a generic outbound SMTP client used for reminder
// notifications, invites, and password resets).
package mailer
import (
"crypto/tls"
"fmt"
"net"
"net/smtp"
"strings"
"sync"
"time"
"archivdms/config"
)
// Mailer sends transactional emails via the configured SMTP-Out relay.
type Mailer struct {
mu sync.RWMutex
cfg config.SMTPOutConfig
}
// New creates a Mailer from the smtp_out config section.
func New(cfg config.SMTPOutConfig) *Mailer {
return &Mailer{cfg: cfg}
}
// Reload replaces the runtime configuration without restarting the process.
func (m *Mailer) Reload(cfg config.SMTPOutConfig) {
m.mu.Lock()
m.cfg = cfg
m.mu.Unlock()
}
// IsConfigured returns true when the smtp_out config is usable.
func (m *Mailer) IsConfigured() bool {
m.mu.RLock()
defer m.mu.RUnlock()
return m.cfg.Host != "" && m.cfg.From != ""
}
// Send sends an HTML + plaintext email to a single recipient.
func (m *Mailer) Send(to, subject, htmlBody, textBody string) error {
m.mu.RLock()
cfg := m.cfg
m.mu.RUnlock()
if cfg.Host == "" || cfg.From == "" {
return fmt.Errorf("mailer: smtp_out not configured")
}
addr := fmt.Sprintf("%s:%d", cfg.Host, port(cfg.Port))
msg := buildMIME(cfg.From, to, subject, htmlBody, textBody)
var auth smtp.Auth
if cfg.User != "" {
auth = smtp.PlainAuth("", cfg.User, cfg.Password, cfg.Host)
}
if cfg.TLS {
return sendTLS(addr, cfg.Host, auth, cfg.From, to, msg)
}
return sendSTARTTLS(addr, auth, cfg.From, to, msg)
}
func port(p int) int {
if p == 0 {
return 587
}
return p
}
func sendTLS(addr, host string, auth smtp.Auth, from, to string, msg []byte) error {
tlsCfg := &tls.Config{ServerName: host, MinVersion: tls.VersionTLS12}
conn, err := tls.DialWithDialer(&net.Dialer{Timeout: 10 * time.Second}, "tcp", addr, tlsCfg)
if err != nil {
return fmt.Errorf("mailer: tls dial: %w", err)
}
defer conn.Close()
c, err := smtp.NewClient(conn, host)
if err != nil {
return fmt.Errorf("mailer: smtp client: %w", err)
}
defer c.Close()
if auth != nil {
if err := c.Auth(auth); err != nil {
return fmt.Errorf("mailer: auth: %w", err)
}
}
return send(c, from, to, msg)
}
func sendSTARTTLS(addr string, auth smtp.Auth, from, to string, msg []byte) error {
c, err := smtp.Dial(addr)
if err != nil {
return fmt.Errorf("mailer: dial: %w", err)
}
defer c.Close()
host, _, _ := net.SplitHostPort(addr)
if ok, _ := c.Extension("STARTTLS"); ok {
tlsCfg := &tls.Config{ServerName: host, MinVersion: tls.VersionTLS12}
if err := c.StartTLS(tlsCfg); err != nil {
return fmt.Errorf("mailer: starttls: %w", err)
}
}
if auth != nil {
if err := c.Auth(auth); err != nil {
return fmt.Errorf("mailer: auth: %w", err)
}
}
return send(c, from, to, msg)
}
func send(c *smtp.Client, from, to string, msg []byte) error {
if err := c.Mail(from); err != nil {
return fmt.Errorf("mailer: MAIL FROM: %w", err)
}
if err := c.Rcpt(to); err != nil {
return fmt.Errorf("mailer: RCPT TO: %w", err)
}
wc, err := c.Data()
if err != nil {
return fmt.Errorf("mailer: DATA: %w", err)
}
defer wc.Close()
if _, err := wc.Write(msg); err != nil {
return fmt.Errorf("mailer: write: %w", err)
}
return nil
}
func buildMIME(from, to, subject, htmlBody, textBody string) []byte {
boundary := "----=archivdms_boundary_20260101"
var b strings.Builder
b.WriteString("From: " + from + "\r\n")
b.WriteString("To: " + to + "\r\n")
b.WriteString("Subject: " + subject + "\r\n")
b.WriteString("MIME-Version: 1.0\r\n")
b.WriteString(`Content-Type: multipart/alternative; boundary="` + boundary + `"` + "\r\n")
b.WriteString("\r\n")
b.WriteString("--" + boundary + "\r\n")
b.WriteString("Content-Type: text/plain; charset=UTF-8\r\n")
b.WriteString("Content-Transfer-Encoding: quoted-printable\r\n")
b.WriteString("\r\n")
b.WriteString(textBody + "\r\n")
b.WriteString("--" + boundary + "\r\n")
b.WriteString("Content-Type: text/html; charset=UTF-8\r\n")
b.WriteString("Content-Transfer-Encoding: quoted-printable\r\n")
b.WriteString("\r\n")
b.WriteString(htmlBody + "\r\n")
b.WriteString("--" + boundary + "--\r\n")
return []byte(b.String())
}
+20
View File
@@ -0,0 +1,20 @@
package mailer
import "fmt"
// ReminderDueTemplate renders subject/HTML/text bodies for a reminder
// ("Wiedervorlage") that has reached its due date. Used by the
// `archivdms reminders notify` cron subcommand.
func ReminderDueTemplate(documentTitle, note, dueDate, appURL string) (subject, html, text string) {
subject = fmt.Sprintf("Wiedervorlage fällig: %s", documentTitle)
html = fmt.Sprintf(`<p>Eine Wiedervorlage ist fällig:</p>
<p><strong>Dokument:</strong> %s<br>
<strong>Fällig am:</strong> %s</p>
<p>%s</p>
<p><a href="%s">Zum Dokument</a></p>`, documentTitle, dueDate, note, appURL)
text = fmt.Sprintf("Eine Wiedervorlage ist fällig:\nDokument: %s\nFällig am: %s\n%s\n\nZum Dokument: %s",
documentTitle, dueDate, note, appURL)
return subject, html, text
}
+219
View File
@@ -0,0 +1,219 @@
// Package matching implements the classification-matching algorithms used
// by tags/document_types/correspondents (internal/storage/taxonomy.go) to
// auto-assign themselves to a newly ingested document based on its OCR text
// plus title. Deliberately dependency-free — no fuzzy-matching Go module is
// added (project style: avoid unnecessary deps, see internal/ocr package
// comment) — the fuzzy algorithm is a small self-contained normalized
// Levenshtein ratio.
package matching
import (
"regexp"
"strings"
)
// Supported algorithm names, mirrored by the match_algorithm CHECK
// constraint on tags/document_types/correspondents.
const (
AlgorithmNone = "none"
AlgorithmAny = "any"
AlgorithmAll = "all"
AlgorithmExact = "exact"
AlgorithmRegex = "regex"
AlgorithmFuzzy = "fuzzy"
)
// fuzzyThreshold is the minimum normalized similarity ratio (0..1) for a
// fuzzy match to count as a hit.
const fuzzyThreshold = 0.85
// Match reports whether pattern matches somewhere in text, using the given
// algorithm and case-sensitivity. Unknown algorithms and "none" always
// return false (never auto-assigned). Malformed regex patterns return
// false rather than panicking — callers are expected to log this
// separately if desired.
func Match(algorithm, pattern string, caseSensitive bool, text string) bool {
if strings.TrimSpace(pattern) == "" {
return false
}
if !caseSensitive {
text = strings.ToLower(text)
pattern = strings.ToLower(pattern)
}
switch algorithm {
case AlgorithmAny:
return matchTerms(pattern, text, false)
case AlgorithmAll:
return matchTerms(pattern, text, true)
case AlgorithmExact:
return strings.Contains(text, pattern)
case AlgorithmRegex:
re, err := regexp.Compile(pattern)
if err != nil {
return false
}
return re.MatchString(text)
case AlgorithmFuzzy:
return fuzzyContains(pattern, text)
case AlgorithmNone:
return false
default:
return false
}
}
// FuzzyThreshold is the minimum FuzzyScore at which the fuzzy Match
// algorithm counts as a hit. Exported so suggestion providers (see
// internal/storage/metadata_suggestions.go) can pick their own lower floor
// relative to the auto-assign threshold.
const FuzzyThreshold = fuzzyThreshold
// FuzzyScore returns the best normalized similarity ratio (0..1) between
// pattern and any whitespace-delimited window of text of pattern's own
// word-count length, using the same normalized Levenshtein ratio and
// word-window scan as the fuzzy Match algorithm. Unlike Match it returns the
// raw score instead of a bool threshold decision, so callers can surface
// near-miss candidates that scored below FuzzyThreshold. Returns 0 for an
// empty pattern. This does NOT change the fuzzy Match algorithm — it only
// exposes its underlying score.
func FuzzyScore(pattern string, caseSensitive bool, text string) float64 {
if strings.TrimSpace(pattern) == "" {
return 0
}
if !caseSensitive {
text = strings.ToLower(text)
pattern = strings.ToLower(pattern)
}
patternWords := strings.Fields(pattern)
if len(patternWords) == 0 {
return 0
}
textWords := strings.Fields(text)
n := len(patternWords)
if len(textWords) < n {
return levenshteinRatio(pattern, text)
}
best := 0.0
for i := 0; i+n <= len(textWords); i++ {
window := strings.Join(textWords[i:i+n], " ")
if r := levenshteinRatio(pattern, window); r > best {
best = r
}
}
return best
}
// tokenizePattern splits pattern into terms, honoring "quoted multi-word
// terms" as single tokens (e.g. `invoice "Muster GmbH" urgent`).
func tokenizePattern(pattern string) []string {
var terms []string
var cur strings.Builder
inQuotes := false
flush := func() {
if t := strings.TrimSpace(cur.String()); t != "" {
terms = append(terms, t)
}
cur.Reset()
}
for _, r := range pattern {
switch {
case r == '"':
inQuotes = !inQuotes
if !inQuotes {
flush()
}
case r == ' ' && !inQuotes:
flush()
default:
cur.WriteRune(r)
}
}
flush()
return terms
}
// matchTerms implements the any/all algorithms: pattern is tokenized into
// (possibly quoted, multi-word) terms; requireAll selects "all" vs "any".
func matchTerms(pattern, text string, requireAll bool) bool {
terms := tokenizePattern(pattern)
if len(terms) == 0 {
return false
}
for _, term := range terms {
hit := strings.Contains(text, term)
if requireAll && !hit {
return false
}
if !requireAll && hit {
return true
}
}
return requireAll
}
// fuzzyContains reports whether any whitespace-delimited window of text (of
// pattern's own word-count length) is within fuzzyThreshold similarity of
// pattern, using a normalized Levenshtein ratio. This is intentionally
// simple (word-window scan, not a full substring-alignment fuzzy search) —
// adequate for short tag/correspondent names against OCR text.
func fuzzyContains(pattern, text string) bool {
patternWords := strings.Fields(pattern)
if len(patternWords) == 0 {
return false
}
textWords := strings.Fields(text)
n := len(patternWords)
if len(textWords) < n {
return levenshteinRatio(pattern, text) >= fuzzyThreshold
}
for i := 0; i+n <= len(textWords); i++ {
window := strings.Join(textWords[i:i+n], " ")
if levenshteinRatio(pattern, window) >= fuzzyThreshold {
return true
}
}
return false
}
// levenshteinRatio returns a normalized similarity ratio in [0,1]: 1 means
// identical strings, 0 means completely dissimilar (edit distance equal to
// the longer string's length).
func levenshteinRatio(a, b string) float64 {
maxLen := max(len([]rune(a)), len([]rune(b)))
if maxLen == 0 {
return 1
}
dist := levenshteinDistance(a, b)
return 1 - float64(dist)/float64(maxLen)
}
// levenshteinDistance computes the classic edit distance between two
// strings (rune-aware), using a single-row dynamic-programming table to
// keep memory usage O(min(len(a),len(b))).
func levenshteinDistance(a, b string) int {
ra, rb := []rune(a), []rune(b)
if len(ra) < len(rb) {
ra, rb = rb, ra
}
prev := make([]int, len(rb)+1)
curr := make([]int, len(rb)+1)
for j := range prev {
prev[j] = j
}
for i := 1; i <= len(ra); i++ {
curr[0] = i
for j := 1; j <= len(rb); j++ {
cost := 1
if ra[i-1] == rb[j-1] {
cost = 0
}
del := prev[j] + 1
ins := curr[j-1] + 1
sub := prev[j-1] + cost
curr[j] = min(del, min(ins, sub))
}
prev, curr = curr, prev
}
return prev[len(rb)]
}
+230
View File
@@ -0,0 +1,230 @@
package ocr
import (
"bytes"
"context"
"encoding/base64"
"fmt"
"html"
"io"
"mime"
"log/slog"
"mime/multipart"
"mime/quotedprintable"
"net/mail"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
)
// officeMimeTypes are the document formats we route through LibreOffice
// (soffice --headless --convert-to pdf) before OCR. Mirrors the Paperless-ngx
// Gotenberg / Docspell LibreOffice conversion stage: the resulting PDF is then
// fed to the normal pdftotext / pdftoppm+tesseract pipeline (ocrPDF), so both
// text-layer PDFs and scanned-image content inside the office file are covered.
var officeMimeTypes = map[string]bool{
// Word / text processing
"application/msword": true,
"application/vnd.openxmlformats-officedocument.wordprocessingml.document": true,
"application/vnd.oasis.opendocument.text": true,
"application/rtf": true,
"text/rtf": true,
// Spreadsheets
"application/vnd.ms-excel": true,
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": true,
"application/vnd.oasis.opendocument.spreadsheet": true,
// Presentations
"application/vnd.ms-powerpoint": true,
"application/vnd.openxmlformats-officedocument.presentationml.presentation": true,
"application/vnd.oasis.opendocument.presentation": true,
}
// isOfficeMime reports whether mimeType names an Office document format that
// Extract routes through LibreOffice conversion.
func isOfficeMime(mimeType string) bool {
return officeMimeTypes[strings.ToLower(strings.TrimSpace(mimeType))]
}
// officeConvertTimeout bounds a single LibreOffice conversion. Deliberately
// more generous than the per-OCR-call timeout: a cold soffice start plus a
// large spreadsheet can legitimately take longer than a tesseract page.
func (e *Extractor) officeConvertTimeout() time.Duration {
base := e.timeout()
if base < 120*time.Second {
return 120 * time.Second
}
return base
}
// officeToPDF converts an Office document at filePath to a temporary PDF via
// LibreOffice headless, returning the PDF path and a cleanup func the caller
// must defer. LibreOffice needs a private user-profile dir to run reliably and
// concurrently (multiple soffice instances sharing the default profile clash),
// so each conversion gets its own scratch dir under TmpDir.
func (e *Extractor) officeToPDF(ctx context.Context, filePath string) (string, func(), error) {
bin := e.sofficePath()
if _, err := exec.LookPath(bin); err != nil {
return "", nil, fmt.Errorf("ocr: libreoffice (%s) not found in PATH: %w", bin, err)
}
workDir := filepath.Join(e.tmpDir(), "office-"+randomID())
if err := os.MkdirAll(workDir, 0o700); err != nil {
return "", nil, fmt.Errorf("ocr: create office convert dir: %w", err)
}
cleanup := func() { os.RemoveAll(workDir) }
profileDir := filepath.Join(workDir, "profile")
cctx, cancel := context.WithTimeout(ctx, e.officeConvertTimeout())
defer cancel()
args := []string{
"--headless", "--norestore", "--nologo", "--nolockcheck",
"-env:UserInstallation=file://" + profileDir,
"--convert-to", "pdf", "--outdir", workDir, filePath,
}
cmd := exec.CommandContext(cctx, bin, args...)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
cleanup()
if cctx.Err() == context.DeadlineExceeded {
return "", nil, fmt.Errorf("ocr: libreoffice conversion timed out after %s", e.officeConvertTimeout())
}
return "", nil, fmt.Errorf("ocr: libreoffice conversion failed: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
// soffice names the output <input-basename>.pdf in --outdir. Prefer that
// exact name, but fall back to the first *.pdf in the dir in case the base
// name was sanitized.
base := strings.TrimSuffix(filepath.Base(filePath), filepath.Ext(filePath))
pdfPath := filepath.Join(workDir, base+".pdf")
if _, err := os.Stat(pdfPath); err != nil {
matches, _ := filepath.Glob(filepath.Join(workDir, "*.pdf"))
if len(matches) == 0 {
cleanup()
return "", nil, fmt.Errorf("ocr: libreoffice produced no pdf for %s", filepath.Base(filePath))
}
pdfPath = matches[0]
}
e.log(slog.LevelInfo, "office document converted to pdf",
"src", filepath.Base(filePath), "pdf", filepath.Base(pdfPath))
return pdfPath, cleanup, nil
}
// extractEML parses an .eml file and returns its human-readable text: a short
// header block (Date/From/To/Cc/Subject, MIME-word-decoded) followed by the
// concatenated text of every text/plain and (tag-stripped) text/html body part.
// Binary attachments are ignored. Best-effort throughout — a malformed message
// yields whatever could be parsed rather than an error, so an e-mail is never
// silently dropped from full-text search.
func (e *Extractor) extractEML(filePath string) (*Result, error) {
data, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("ocr: read eml: %w", err)
}
msg, err := mail.ReadMessage(bytes.NewReader(data))
if err != nil {
return nil, fmt.Errorf("ocr: parse eml: %w", err)
}
var sb strings.Builder
dec := new(mime.WordDecoder)
for _, h := range []string{"Date", "From", "To", "Cc", "Subject"} {
v := msg.Header.Get(h)
if v == "" {
continue
}
if d, derr := dec.DecodeHeader(v); derr == nil {
v = d
}
sb.WriteString(h)
sb.WriteString(": ")
sb.WriteString(v)
sb.WriteString("\n")
}
sb.WriteString("\n")
body := mailPartText(msg.Header.Get("Content-Type"), msg.Header.Get("Content-Transfer-Encoding"), msg.Body)
sb.WriteString(body)
return &Result{Text: strings.TrimSpace(sb.String())}, nil
}
// mailPartText recursively extracts readable text from a MIME part. multipart/*
// containers are walked; text/plain is decoded verbatim, text/html is decoded
// and tag-stripped; everything else (attachments, images) is skipped.
func mailPartText(contentType, cte string, body io.Reader) string {
mediaType, params, err := mime.ParseMediaType(contentType)
if err != nil || mediaType == "" {
// No/invalid Content-Type: assume text/plain.
return decodeMailBody(body, cte)
}
if strings.HasPrefix(mediaType, "multipart/") {
boundary := params["boundary"]
if boundary == "" {
return ""
}
mr := multipart.NewReader(body, boundary)
var parts []string
for {
p, perr := mr.NextPart()
if perr != nil {
break
}
pt := mailPartText(p.Header.Get("Content-Type"), p.Header.Get("Content-Transfer-Encoding"), p)
p.Close()
if strings.TrimSpace(pt) != "" {
parts = append(parts, pt)
}
}
return strings.Join(parts, "\n")
}
switch {
case strings.HasPrefix(mediaType, "text/plain"):
return decodeMailBody(body, cte)
case strings.HasPrefix(mediaType, "text/html"):
return stripHTML(decodeMailBody(body, cte))
default:
return "" // attachment / binary part
}
}
// decodeMailBody reads a leaf MIME part, undoing base64 / quoted-printable
// transfer encoding.
func decodeMailBody(r io.Reader, cte string) string {
switch strings.ToLower(strings.TrimSpace(cte)) {
case "base64":
r = base64.NewDecoder(base64.StdEncoding, r)
case "quoted-printable":
r = quotedprintable.NewReader(r)
}
b, err := io.ReadAll(r)
if err != nil {
return string(b) // return whatever decoded before the error
}
return string(b)
}
var (
htmlTagRe = regexp.MustCompile(`(?s)<(script|style)[^>]*>.*?</(script|style)>`)
anyTagRe = regexp.MustCompile(`(?s)<[^>]*>`)
wsRe = regexp.MustCompile(`[ \t\f\v]+`)
blankRe = regexp.MustCompile(`\n{3,}`)
)
// stripHTML reduces an HTML body to readable plain text: drops script/style
// blocks and all tags, unescapes entities, and collapses runaway whitespace.
func stripHTML(s string) string {
s = htmlTagRe.ReplaceAllString(s, " ")
s = anyTagRe.ReplaceAllString(s, " ")
s = html.UnescapeString(s)
s = wsRe.ReplaceAllString(s, " ")
s = blankRe.ReplaceAllString(s, "\n\n")
return strings.TrimSpace(s)
}
+359
View File
@@ -0,0 +1,359 @@
// Word-level bounding boxes for OCR text-highlight/overlay (Phase 1 —
// datengrundlage only, see project memory
// project_ocr_textmarkierung_overlay.md). This file adds:
//
// - WordBox / TSV extraction (tesseract's `tsv` output mode)
// - a small geometry-transform mechanism to map word boxes from the
// coordinate space of the final, fully-preprocessed image tesseract
// actually recognized text on, back into the coordinate space of the
// file the frontend actually displays to the user.
//
// Koordinatenraum (why this file exists at all): runTesseract's
// preprocessing pipeline (clampImageSize -> deskewImage -> normalizeContrast
// -> rotateForOSD, see ocr.go) can resize and rotate the image before
// tesseract ever sees it. The frontend, however, always renders the
// untouched original upload (internal/api/document_handlers.go
// handleGetDocumentFile serves doc.StoragePath byte-for-byte; verified
// 2026-07-30 — no transformed copy is ever persisted or served). Word boxes
// from tesseract are therefore in the WRONG coordinate space for direct use
// against the displayed image unless mapped back.
//
// Full forward order in runTesseract (each step optional):
//
// clampImageSize -> deskewImage | deskewImageHough -> normalizeContrast
// -> rotateForOSD -> binarizeImage
//
// and, for image uploads only, one final forward step applied in ocrImage
// AFTER the inversion above: the file's EXIF Orientation (see exif.go), which
// moves the boxes from raw-pixel space into the space the browser actually
// renders. Inversion happens strictly last-forward-step-first
// (mapWordsToOriginal iterates the slice backwards), so any combination —
// e.g. clamp + hough-deskew + OSD 90 degrees + EXIF 6 — composes correctly:
// the geometric chain is undone in reverse, then EXIF is applied once on top.
//
// What is handled exactly vs. approximately:
// - clampImageSize: pure uniform scale -> inverted exactly (simple ratio).
// - normalizeContrast / binarizeImage: no geometry change; still measured
// (geomChain.recordScale) rather than assumed, and dropped as identity.
// - rotateForOSD: our own rotate90CW, always an exact multiple of 90
// degrees -> inverted with pixel-exact integer math (mirrors the forward
// loop in rotateImageFile step for step, no trig/rounding involved).
// - deskewImage (ImageMagick `-deskew`, arbitrary small angle + canvas
// resize to bound the rotated image): inverted via the standard
// rotate-about-center formula using the angle ImageMagick reports via
// `-print "%[deskew:angle]"` plus before/after pixel dimensions. This is
// geometrically the correct construction for a generic "rotate and
// expand canvas" operation. Sign convention reviewed 2026-07-30 against
// ImageMagick's source behaviour: DeskewImage derives the `deskew:angle`
// artifact from the same `degrees` it feeds into the affine matrix
// [[cos,-sin],[sin,cos]], and AffineTransformImage expands the canvas
// symmetrically about the centre (auto-crop off by default) — so the
// centre-to-centre inverse with rad = -angleDeg below is the exact
// transpose. Still not validated against a real deskewed sample's pixel
// output, so treat it as reviewed-but-not-field-verified.
// Per project memory (two prior deskew-angle tuning attempts were tested
// against the doc id 4-9 corpus and rejected — see
// project_deskew_border_trick_tested_negative.md and
// project_deskew_disable_for_photos_tested_negative.md), do NOT blindly
// adjust this formula's sign/rounding by trial and error; instead verify
// against a real deskewed sample (overlay the mapped word boxes on the
// original image) before touching it, and record the result either way.
// - deskewImageHough: the angle is detected by hough_deskew.py but APPLIED
// by `convert -rotate <angle>`, whose sign convention is documented and
// unambiguous (positive = clockwise). The inverse below therefore IS
// verified for this path — the unverified sign caveat above applies only
// to ImageMagick's own `-deskew`/%[deskew:angle] pair.
// - Steps whose geometry cannot be measured (image.DecodeConfig only knows
// the formats this package imports, i.e. JPEG and PNG — a TIFF/BMP/WebP
// upload fails every measurement while ImageMagick still processes it)
// invalidate the whole chain via geomChain, and the document then gets NO
// word boxes. Silently skipping such a step used to leave the remaining
// transforms mapping into a coordinate space that no longer existed.
//
// PDF scope note: for the pdftoppm raster-fallback OCR path, WordBox
// coordinates are mapped back to the *rasterized page PNG's* pixel space
// (post-preprocessing -> pre-preprocessing raster), not further back into
// PDF point/MediaBox coordinate space. The frontend currently renders PDFs
// via the browser's native PDF viewer (iframe over the original file), which
// uses PDF page-coordinate space, not raster pixels — mapping raster pixels
// into that space is a straightforward additional scale step (raster DPI vs.
// MediaBox size, both knowable via pdftoppm's -r 300 and `pdfinfo`) but is
// left for whoever builds the overlay UI in a later phase, since it depends
// on how that phase chooses to render PDF pages (canvas render at a chosen
// DPI vs. native iframe).
package ocr
import (
"image"
"log/slog"
"math"
"os"
)
// WordBox is a single OCR-recognized word with its bounding box, already
// mapped (best-effort — see package doc comment above) into the coordinate
// space of the file the frontend actually displays for the document this
// word was found in.
type WordBox struct {
Text string
Left int
Top int
Width int
Height int
Confidence float64
// Line, Block, Par come straight from tesseract's TSV line_num/block_num/
// par_num columns, useful for later grouping words into lines/paragraphs
// (e.g. for the eventual highlight-overlay UI) without re-deriving that
// from raw positions.
Line int
Block int
Par int
// Page is the 1-based PDF page number this word was found on. Always 1
// for image uploads (a single "page"; there is no page 0 in output).
Page int
}
// geomTransform describes one preprocessing step's effect on image geometry,
// used to invert tesseract's word bounding boxes back towards the originally
// displayed file. See the package doc comment for what is exact vs.
// best-effort here.
type geomTransform struct {
oldW, oldH int
newW, newH int
// angleDeg is the clockwise rotation applied around the image center, in
// degrees. Zero for a pure resize/no-op step.
angleDeg float64
// exact90 marks a rotation known to be an exact multiple of 90 degrees,
// produced by our own rotate90CW (rotateForOSD) — inverted with
// pixel-exact integer math rather than the trig formula used for
// deskew's arbitrary angle.
exact90 bool
}
// invert maps a point (x, y) from the "new" (post-step) image's pixel space
// back into the "old" (pre-step) image's pixel space.
func (t geomTransform) invert(x, y float64) (float64, float64) {
if t.angleDeg == 0 {
if t.newW == 0 || t.newH == 0 {
return x, y
}
scaleX := float64(t.oldW) / float64(t.newW)
scaleY := float64(t.oldH) / float64(t.newH)
return x * scaleX, y * scaleY
}
if t.exact90 {
steps := (int(math.Round(t.angleDeg)) / 90) % 4
if steps < 0 {
steps += 4
}
curW, curH := t.newW, t.newH
cx, cy := x, y
for i := 0; i < steps; i++ {
// Forward step (rotateImageFile/rotate90CW) was, on pixel
// INDICES: src(w,h) -> dst(h,w), src(x,y) -> dst(h-1-y, x).
// mapWordsToOriginal feeds box EDGE coordinates (left..left+width,
// i.e. a continuous [0,w] range, not indices [0,w-1]), so the
// continuous form of the same rotation is used here:
// dst(x,y) = (h - y, x) => src = (cy, curW - cx)
// (Identical convention to applyEXIFOrientation in exif.go; using
// the index form on edge coordinates would shift every box by one
// pixel per rotation step.)
nx := cy
ny := float64(curW) - cx
curW, curH = curH, curW
cx, cy = nx, ny
}
return cx, cy
}
// General case (ImageMagick -deskew): rotation about the image center
// with the canvas expanded to bound the rotated image. Sign convention:
// positive angleDeg == clockwise (ImageMagick `-rotate`), so the inverse
// rotates by -angleDeg about the new centre and re-centres on the old
// canvas. See package doc comment for how far this is verified per path
// (hough: yes; ImageMagick's own -deskew: source-reviewed only).
rad := -t.angleDeg * math.Pi / 180
cxNew, cyNew := float64(t.newW)/2, float64(t.newH)/2
cxOld, cyOld := float64(t.oldW)/2, float64(t.oldH)/2
dx, dy := x-cxNew, y-cyNew
cos, sin := math.Cos(rad), math.Sin(rad)
rx := dx*cos - dy*sin
ry := dx*sin + dy*cos
return rx + cxOld, ry + cyOld
}
// isIdentity reports whether this step changed no geometry at all (same
// dimensions, no rotation) and can therefore be dropped from the chain.
func (t geomTransform) isIdentity() bool {
return t.angleDeg == 0 && t.oldW == t.newW && t.oldH == t.newH
}
// mapWordsToOriginal applies transforms in reverse (last-applied-preprocessing-
// step-first) order, mutating words in place to convert their bounding boxes
// from final-tesseract-image space into the coordinate space of the file
// before any of these transforms ran.
//
// ALL FOUR corners are inverted, not just top-left/bottom-right. That matters
// as soon as a non-90-degree rotation (deskew) is in the chain: under a
// rotation the two opposite corners alone no longer span the rotated
// rectangle's axis-aligned bounding box — for a typical 2-3 degree deskew the
// resulting box is systematically too narrow/short and offset, and at angles
// approaching 45 degrees it collapses towards zero size. The result here is
// the true axis-aligned bounding box of the back-rotated word quad, which is
// what the frontend overlay draws.
func mapWordsToOriginal(words []WordBox, transforms []geomTransform) {
if len(transforms) == 0 {
return
}
for i := range words {
l, t := float64(words[i].Left), float64(words[i].Top)
r, b := float64(words[i].Left+words[i].Width), float64(words[i].Top+words[i].Height)
corners := [4][2]float64{{l, t}, {r, t}, {r, b}, {l, b}}
for c := range corners {
x, y := corners[c][0], corners[c][1]
for j := len(transforms) - 1; j >= 0; j-- {
x, y = transforms[j].invert(x, y)
}
corners[c][0], corners[c][1] = x, y
}
minX, maxX := corners[0][0], corners[0][0]
minY, maxY := corners[0][1], corners[0][1]
for c := 1; c < 4; c++ {
minX = math.Min(minX, corners[c][0])
maxX = math.Max(maxX, corners[c][0])
minY = math.Min(minY, corners[c][1])
maxY = math.Max(maxY, corners[c][1])
}
words[i].Left = int(math.Round(minX))
words[i].Top = int(math.Round(minY))
words[i].Width = int(math.Round(maxX - minX))
words[i].Height = int(math.Round(maxY - minY))
}
}
// geomChain collects the geometry-changing preprocessing steps of a single
// runTesseract pass, so word boxes can be inverted back into the source
// image's coordinate space afterwards.
//
// The important property it enforces (this was a real, silent bug before):
// a preprocessing step that DID change geometry but whose geometry could not
// be measured must invalidate the whole chain, not just be skipped. Skipping
// it leaves the remaining transforms mapping into a coordinate space that no
// longer exists, and the frontend then draws a confidently wrong overlay.
// The realistic trigger is an upload format image.DecodeConfig cannot read:
// this package only registers image/jpeg and image/png, so TIFF/BMP/WebP/GIF
// uploads (all accepted as image/*) fail every decodeImageDims call while
// ImageMagick happily processes them. Rather than misplace boxes we return
// none for those documents.
type geomChain struct {
steps []geomTransform
broken bool
log func(level slog.Level, msg string, args ...any)
}
// recordScale books a step that may only scale the image uniformly
// (clampImageSize) or must not change geometry at all (normalizeContrast,
// binarizeImage). Identity steps are dropped.
func (c *geomChain) recordScale(step, oldPath, newPath string) {
t, ok := buildScaleTransform(oldPath, newPath)
if !ok {
c.fail(step, "image dimensions unreadable (unsupported format for image.DecodeConfig?)")
return
}
if t.isIdentity() {
return
}
c.steps = append(c.steps, t)
}
// recordRotation books a rotation step (deskewImage/deskewImageHough/
// rotateForOSD). A reported angle of 0 combined with changed dimensions means
// the angle was lost (e.g. an ImageMagick build not populating
// %[deskew:angle]) while a rotation really was applied — unrecoverable, so
// the chain is invalidated instead of silently mapping with angle 0.
func (c *geomChain) recordRotation(step, oldPath, newPath string, angleDeg float64, exact90 bool) {
t, ok := buildRotationTransform(oldPath, newPath, angleDeg, exact90)
if !ok {
c.fail(step, "image dimensions unreadable (unsupported format for image.DecodeConfig?)")
return
}
if angleDeg == 0 && (t.oldW != t.newW || t.oldH != t.newH) {
c.fail(step, "rotation applied but angle unknown (0) — cannot invert")
return
}
if t.isIdentity() {
return
}
c.steps = append(c.steps, t)
}
func (c *geomChain) fail(step, reason string) {
c.broken = true
if c.log != nil {
c.log(slog.LevelWarn, "ocr word boxes disabled: preprocessing geometry not invertible",
"step", step, "reason", reason)
}
}
// transforms returns the collected chain; ok is false when any step could not
// be recorded reliably, in which case callers must not emit word boxes at all.
func (c *geomChain) transforms() ([]geomTransform, bool) {
if c.broken {
return nil, false
}
return c.steps, true
}
// decodeImageDims returns the pixel width/height of the image at path
// without decoding full pixel data (image.DecodeConfig only reads the
// header).
func decodeImageDims(path string) (w, h int, err error) {
f, err := os.Open(path)
if err != nil {
return 0, 0, err
}
defer f.Close()
cfg, _, err := image.DecodeConfig(f)
if err != nil {
return 0, 0, err
}
return cfg.Width, cfg.Height, nil
}
// buildScaleTransform records a pure-resize geometry step (clampImageSize)
// by decoding both images' dimensions. ok is false if either image's
// dimensions cannot be read, in which case the caller should skip recording
// a transform (best-effort, same tolerance as the rest of this package).
func buildScaleTransform(oldPath, newPath string) (geomTransform, bool) {
oldW, oldH, err := decodeImageDims(oldPath)
if err != nil {
return geomTransform{}, false
}
newW, newH, err := decodeImageDims(newPath)
if err != nil {
return geomTransform{}, false
}
return geomTransform{oldW: oldW, oldH: oldH, newW: newW, newH: newH}, true
}
// buildRotationTransform records a rotation geometry step (deskewImage or
// rotateForOSD) by decoding both images' dimensions plus the rotation angle
// applied. exact90 distinguishes rotateForOSD's pixel-exact 90-degree
// rotations from deskewImage's arbitrary-angle, best-effort inverse.
func buildRotationTransform(oldPath, newPath string, angleDeg float64, exact90 bool) (geomTransform, bool) {
oldW, oldH, err := decodeImageDims(oldPath)
if err != nil {
return geomTransform{}, false
}
newW, newH, err := decodeImageDims(newPath)
if err != nil {
return geomTransform{}, false
}
return geomTransform{
oldW: oldW, oldH: oldH,
newW: newW, newH: newH,
angleDeg: angleDeg,
exact90: exact90,
}, true
}
+224
View File
@@ -0,0 +1,224 @@
package ocr
// EXIF-Orientierung für den OCR-Wortbox-Koordinatenraum.
//
// Warum diese Datei existiert (Root Cause des Overlay-Versatzes, 2026-07-30):
// Die Vorverarbeitungskette in runTesseract arbeitet ausschließlich auf ROHEN
// Pixeln — weder tesseract noch ImageMagick `convert` wenden das EXIF-Tag
// `Orientation` von selbst an (dafür bräuchte es explizit `-auto-orient`).
// mapWordsToOriginal rechnet die Wortboxen folglich in den ROH-Pixelraum der
// gespeicherten Datei zurück.
//
// Der Browser tut aber genau das Gegenteil: seit der Vereinheitlichung von
// `image-orientation: from-image` als Default (Chrome 81+, Firefox 26+,
// Safari 13.1+) rendert er ein <img> IMMER EXIF-orientiert und meldet auch
// naturalWidth/naturalHeight bereits gedreht. Bei einem Handyfoto mit
// Orientation 6/8 (Hochkant aufgenommen, Sensor liefert Querformat-Pixel)
// zeigt das Frontend also ein 3000x4000-Bild, während jede Wortbox in
// 4000x3000-Rohkoordinaten vorliegt: das Overlay ist um 90 Grad verdreht und
// liegt zum Teil komplett außerhalb des Bildes. Genau das ist das gemeldete
// "passt nicht mit den OCR-Feldern" — kein Subpixel-/Deskew-Problem, sondern
// ein kompletter Raumwechsel.
//
// Lösung: nach der Rücktransformation in den Rohraum wird hier EINMAL die
// EXIF-Orientierung vorwärts angewandt, damit die gespeicherten Koordinaten im
// tatsächlich DARGESTELLTEN Raum liegen (das ist auch die dokumentierte
// Semantik der ocr_words-Spalten und der API — "Koordinatenraum der
// angezeigten Datei"). Orientation 1 (bzw. kein EXIF, PNG, PDF-Raster) ist ein
// No-Op, betrifft also nur genau die Fotos, bei denen der Browser dreht.
//
// Der EXIF-Parser ist bewusst minimal und dependency-frei (nur stdlib): er
// sucht den APP1/"Exif\0\0"-Marker, liest den TIFF-Header und die IFD0-Einträge
// und gibt Tag 0x0112 zurück. Alles andere (XMP, MakerNotes, Thumbnails) wird
// nicht angefasst.
import (
"encoding/binary"
"errors"
"io"
"math"
"os"
)
// errNoEXIFOrientation signalisiert "kein verwertbares Orientation-Tag" —
// Aufrufer behandeln das wie Orientation 1.
var errNoEXIFOrientation = errors.New("ocr: no exif orientation")
// maxEXIFScan begrenzt, wie weit wir im JPEG nach dem APP1-Segment suchen.
// EXIF steht per Spezifikation direkt hinter SOI; die Grenze verhindert nur,
// dass eine kaputte Datei uns durch das ganze Bild laufen lässt.
const maxEXIFScan = 1 << 20 // 1 MiB
// jpegEXIFOrientation liefert den Wert des EXIF-Tags Orientation (1..8) der
// Datei an path. Für Nicht-JPEGs, JPEGs ohne EXIF, unlesbare oder unplausible
// Werte wird 1 (= keine Drehung) zurückgegeben; ein Fehler wird nur zur
// optionalen Diagnose mitgegeben und ist für Aufrufer nicht fatal.
func jpegEXIFOrientation(path string) (int, error) {
f, err := os.Open(path)
if err != nil {
return 1, err
}
defer f.Close()
var soi [2]byte
if _, err := io.ReadFull(f, soi[:]); err != nil {
return 1, err
}
if soi[0] != 0xFF || soi[1] != 0xD8 { // kein JPEG (PNG/TIFF/…): kein EXIF-Handling
return 1, errNoEXIFOrientation
}
scanned := 0
var hdr [4]byte
for scanned < maxEXIFScan {
// Marker suchen: beliebig viele 0xFF-Füllbytes, dann der Markercode.
var b [1]byte
if _, err := io.ReadFull(f, b[:]); err != nil {
return 1, errNoEXIFOrientation
}
scanned++
if b[0] != 0xFF {
continue
}
for {
if _, err := io.ReadFull(f, b[:]); err != nil {
return 1, errNoEXIFOrientation
}
scanned++
if b[0] != 0xFF {
break
}
}
marker := b[0]
switch {
case marker == 0xDA || marker == 0xD9:
// SOS (Bilddaten) bzw. EOI (Dateiende) erreicht: ab hier kann kein
// APP1/EXIF-Segment mehr kommen. Muss VOR der Prüfung auf
// längenlose Marker stehen — 0xD9 fällt sonst in den
// RST/D0..D7-Bereich und wir würden durch die Bilddaten weiterlaufen.
return 1, errNoEXIFOrientation
case marker == 0x00 || marker == 0xFF:
continue // Byte-Stuffing/Füllbyte, kein echter Marker
case marker == 0xD8 || marker == 0x01 || (marker >= 0xD0 && marker <= 0xD7):
continue // SOI/TEM/RSTn: längenlose Marker
}
if _, err := io.ReadFull(f, hdr[:2]); err != nil {
return 1, errNoEXIFOrientation
}
segLen := int(binary.BigEndian.Uint16(hdr[:2]))
if segLen < 2 {
return 1, errNoEXIFOrientation
}
payload := make([]byte, segLen-2)
if _, err := io.ReadFull(f, payload); err != nil {
return 1, errNoEXIFOrientation
}
scanned += segLen
if marker != 0xE1 || len(payload) < 6 || string(payload[:6]) != "Exif\x00\x00" {
continue
}
return orientationFromTIFF(payload[6:])
}
return 1, errNoEXIFOrientation
}
// orientationFromTIFF liest Tag 0x0112 aus dem IFD0 eines TIFF-Headers (der
// Nutzlast eines EXIF-APP1-Segments ohne "Exif\0\0"-Präfix).
func orientationFromTIFF(tiff []byte) (int, error) {
if len(tiff) < 8 {
return 1, errNoEXIFOrientation
}
var bo binary.ByteOrder
switch {
case tiff[0] == 'I' && tiff[1] == 'I':
bo = binary.LittleEndian
case tiff[0] == 'M' && tiff[1] == 'M':
bo = binary.BigEndian
default:
return 1, errNoEXIFOrientation
}
if bo.Uint16(tiff[2:4]) != 42 {
return 1, errNoEXIFOrientation
}
offset := int(bo.Uint32(tiff[4:8]))
if offset < 8 || offset+2 > len(tiff) {
return 1, errNoEXIFOrientation
}
count := int(bo.Uint16(tiff[offset : offset+2]))
entry := offset + 2
for i := 0; i < count; i++ {
if entry+12 > len(tiff) {
break
}
tag := bo.Uint16(tiff[entry : entry+2])
typ := bo.Uint16(tiff[entry+2 : entry+4])
if tag == 0x0112 && typ == 3 /* SHORT */ {
v := int(bo.Uint16(tiff[entry+8 : entry+10]))
if v >= 1 && v <= 8 {
return v, nil
}
return 1, errNoEXIFOrientation
}
entry += 12
}
return 1, errNoEXIFOrientation
}
// applyEXIFOrientation überführt Wortboxen aus dem ROH-Pixelraum eines Bildes
// (Breite rawW, Höhe rawH) in den vom Browser DARGESTELLTEN Raum, indem die
// EXIF-Orientierung orientation (1..8) vorwärts angewandt wird. orientation 1
// sowie ungültige Werte/Dimensionen sind ein No-Op.
//
// Die acht EXIF-Fälle entsprechen den üblichen Definitionen (2/4/5/7 enthalten
// eine Spiegelung; sie kommen bei Kameras praktisch nicht vor, werden aber der
// Vollständigkeit halber korrekt behandelt, damit hier nie stillschweigend ein
// falscher Raum entsteht):
//
// 1 (x, y) 2 (W-x, y) 3 (W-x, H-y) 4 (x, H-y)
// 5 (y, x) 6 (H-y, x) 7 (H-y, W-x) 8 (y, W-x)
//
// Bei 5..8 tauschen Breite und Höhe die Rollen — genau der Fall, in dem das
// Overlay ohne diese Korrektur komplett neben dem Bild landet.
func applyEXIFOrientation(words []WordBox, orientation, rawW, rawH int) {
if orientation <= 1 || orientation > 8 || rawW <= 0 || rawH <= 0 {
return
}
w, h := float64(rawW), float64(rawH)
mapPoint := func(x, y float64) (float64, float64) {
switch orientation {
case 2:
return w - x, y
case 3:
return w - x, h - y
case 4:
return x, h - y
case 5:
return y, x
case 6:
return h - y, x
case 7:
return h - y, w - x
case 8:
return y, w - x
default:
return x, y
}
}
for i := range words {
x0, y0 := mapPoint(float64(words[i].Left), float64(words[i].Top))
x1, y1 := mapPoint(float64(words[i].Left+words[i].Width), float64(words[i].Top+words[i].Height))
if x0 > x1 {
x0, x1 = x1, x0
}
if y0 > y1 {
y0, y1 = y1, y0
}
// math.Round statt int(v+0.5): Wortboxen können nach der
// Rücktransformation aus einem Deskew-Schritt knapp negative
// Randkoordinaten haben, dort rundet int(v+0.5) in die falsche Richtung.
words[i].Left = int(math.Round(x0))
words[i].Top = int(math.Round(y0))
words[i].Width = int(math.Round(x1 - x0))
words[i].Height = int(math.Round(y1 - y0))
}
}
+1060
View File
File diff suppressed because it is too large Load Diff
+134
View File
@@ -0,0 +1,134 @@
#!/usr/bin/env python3
"""hough_deskew.py — fine-skew angle detector for the archivdms OCR pipeline.
Sidecar script for internal/ocr/ocr.go's `hough` deskew method
(config.OCRConfig.DeskewMethod == "hough"), an ALTERNATIVE to the default
ImageMagick `-deskew` peak/valley text-line projection analysis
(deskewImage() in ocr.go). ImageMagick's approach needs surrounding
background/margin to find the page's background rows/columns and fails on
tightly-cropped phone photos of receipts (no margin context) see
project_deskew_disable_for_photos_tested_negative and
project_deskew_border_trick_tested_negative in agent memory for two
previously-tried and rejected workarounds. This script separates ANGLE
DETECTION (via OpenCV, this file) from angle APPLICATION (plain `convert
-rotate <deg>` in ocr.go) per the recommendation that produced this rewrite.
Usage:
python3 hough_deskew.py <image-path>
Behavior:
- Reads the image with OpenCV, grayscale + Otsu threshold.
- Finds the largest contour by area and takes cv2.minAreaRect() of it.
This is deliberately NOT text-line-projection-based (that is exactly
what ImageMagick already does and what fails on cropped photos)
minAreaRect degrades gracefully to "the boundary of whatever content is
in frame" even when that content fills the whole image, which is
normally the case for a tightly-cropped phone photo.
- Falls back to cv2.HoughLinesP() long-line-angle voting if no usable
contour is found (e.g. near-blank background, no single dominant
shape) takes the median angle of detected line segments within
+/-45 degrees of horizontal.
- Prints exactly one float (the skew angle in degrees, ImageMagick
`-rotate` sign convention: positive = clockwise) to stdout and exits 0
on success.
- On any failure (bad path, unreadable image, no contours/lines found),
prints nothing to stdout, writes a one-line reason to stderr, and
exits non-zero. ocr.go's houghDeskewAngle treats this as "angle 0,
keep going" — never a fatal OCR error.
Dependencies: opencv-python (or the Debian python3-opencv apt package, which
pulls in numpy as a transitive dependency) no other third-party packages.
Deliberately not using the `deskew` PyPI package: it wraps a very similar
Radon/Hough approach but pulls in scikit-image, a much heavier dependency
tree, for no accuracy benefit found in testing.
"""
import sys
try:
import cv2
import numpy as np
except ImportError as exc: # pragma: no cover - environment/dependency issue
print(f"hough_deskew: missing dependency: {exc}", file=sys.stderr)
sys.exit(2)
def _angle_from_min_area_rect(gray: "np.ndarray"):
"""Return a skew angle in degrees via Otsu threshold + largest contour's
minAreaRect, or None if no usable contour was found."""
_, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
contours, _ = cv2.findContours(thresh, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
if not contours:
return None
largest = max(contours, key=cv2.contourArea)
# Ignore contours covering too little of the frame — noise/artifacts, not
# the document itself.
img_area = gray.shape[0] * gray.shape[1]
if cv2.contourArea(largest) < 0.05 * img_area:
return None
rect = cv2.minAreaRect(largest)
angle = rect[2] # OpenCV: angle in (-90, 0] for cv2.minAreaRect
# Normalize to the smallest rotation that would make the rect's long side
# horizontal (matches ImageMagick -deskew / -rotate's small-angle
# convention rather than cv2's raw (-90, 0] range).
w, h = rect[1]
if w < h:
angle = angle + 90
if angle > 45:
angle -= 90
elif angle < -45:
angle += 90
return angle
def _angle_from_hough_lines(gray: "np.ndarray"):
"""Fallback: median angle of long line segments detected via
HoughLinesP, restricted to +/-45 degrees of horizontal. Returns None if
no usable lines were found."""
edges = cv2.Canny(gray, 50, 150, apertureSize=3)
lines = cv2.HoughLinesP(
edges, 1, np.pi / 180, threshold=100, minLineLength=gray.shape[1] // 4, maxLineGap=20
)
if lines is None or len(lines) == 0:
return None
angles = []
for line in lines:
x1, y1, x2, y2 = line[0]
dx, dy = x2 - x1, y2 - y1
if dx == 0:
continue
angle = np.degrees(np.arctan2(dy, dx))
if -45 <= angle <= 45:
angles.append(angle)
if not angles:
return None
return float(np.median(angles))
def main() -> int:
if len(sys.argv) != 2:
print("hough_deskew: usage: hough_deskew.py <image-path>", file=sys.stderr)
return 2
path = sys.argv[1]
img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
if img is None:
print(f"hough_deskew: could not read image: {path}", file=sys.stderr)
return 1
angle = _angle_from_min_area_rect(img)
if angle is None:
angle = _angle_from_hough_lines(img)
if angle is None:
print("hough_deskew: no usable contour or line angle found", file=sys.stderr)
return 1
print(f"{angle:.4f}")
return 0
if __name__ == "__main__":
sys.exit(main())
+514
View File
@@ -0,0 +1,514 @@
// Package pagesplit implements barcode separator-page splitting for
// multi-page PDF ingest ("Trennseiten-Split", inspired by Paperless-ngx's
// ASN/separator barcode feature, adapted to archivdms's ingest pipeline).
//
// Idea: a scanner operator interleaves printed separator sheets carrying a
// well-known barcode (default value "ARCHIVDMS-SPLIT") between the individual
// receipts of one long scan run. At ingest the PDF is checked page by page for
// that barcode; where it is found, the document is cut, and the separator page
// itself is dropped (it is a control sheet, not content — same behaviour as
// Paperless-ngx). Each resulting part then runs through the completely normal
// staging path (own WORM file, own hash/duplicate check, own processing job).
//
// Design constraints this package follows, all inherited from the existing
// codebase:
//
// - No CGO, no PDF library: everything is done by shelling out to the
// poppler-utils binaries that are already a service dependency of the OCR
// pipeline (pdfinfo, pdftoppm, pdfseparate, pdfunite) plus zbarimg via
// internal/barcode. Deliberately NOT qpdf/pdftk — those would be a new
// package dependency for something poppler already covers.
// - Best-effort, fail-safe: every error path returns "no split" rather than
// failing the upload. A scanner run that cannot be analysed must still be
// archived, unsplit, rather than rejected. The one thing that is never
// silently swallowed is a *partially* produced split — Split either yields
// a complete set of parts or nothing at all.
// - Off by default (Detector.Enabled), per the project's conservative rule
// for new preprocessing behaviour (cf. the Otsu binarize switch in
// internal/ocr).
//
// Scope note: only application/pdf is handled. Multi-page TIFF is a
// theoretically possible scanner output but is not currently produced by any
// archivdms ingest path (HTTP upload and the SFTP watcher both hand single
// images or PDFs to the pipeline), so it is intentionally out of scope here
// rather than half-supported.
package pagesplit
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
"archivdms/internal/barcode"
)
// DefaultMarker is the barcode payload that marks a separator page when no
// other value is configured. Chosen to be unambiguous and unlikely to collide
// with a taxonomy barcode (internal/storage/taxonomy.go barcode_value) or with
// anything printed on a real invoice.
const DefaultMarker = "ARCHIVDMS-SPLIT"
// defaultRasterDPI is the resolution separator detection rasterizes at. Much
// lower than the 300 dpi the OCR pipeline uses: a separator sheet carries one
// large, high-contrast barcode, and 150 dpi decodes those reliably while
// keeping the extra pdftoppm pass cheap on long scan runs.
const defaultRasterDPI = 150
// defaultTimeout bounds each individual poppler subprocess call.
const defaultTimeout = 120 * time.Second
// defaultMaxPages caps how many pages are analysed. A scan run beyond this is
// treated as "not analysable" (no split, archived as one document) instead of
// spending unbounded time rasterizing — the same bounded-worst-case reasoning
// as internal/ocr's maxImagePixels clamp.
const defaultMaxPages = 200
// Detector performs separator-page detection and PDF splitting.
//
// Construct via New and set the optional fields afterwards; the zero value is
// disabled and therefore a safe no-op.
type Detector struct {
// Enabled turns the whole feature on. False (zero value) => Split always
// reports "no split".
Enabled bool
// Marker is the barcode payload identifying a separator page. Empty
// defaults to DefaultMarker. Compared case-insensitively after trimming.
Marker string
// MarkerPrefix switches the comparison from "equals Marker" to "starts
// with Marker", so operators can encode extra data on the separator sheet
// (e.g. "ARCHIVDMS-SPLIT-2026-INVOICES") with a single configured value.
MarkerPrefix bool
// PdftoppmPath/PdfinfoPath/PdfseparatePath/PdfunitePath name the poppler
// binaries. Empty values fall back to the plain command names.
PdftoppmPath string
PdfinfoPath string
PdfseparatePath string
PdfunitePath string
// TmpDir is the scratch base directory (config.StorageConfig.OCRTmpPath()).
// Every Split call gets its own subdirectory, removed by Result.Cleanup.
TmpDir string
// RasterDPI overrides defaultRasterDPI. MaxPages overrides defaultMaxPages.
RasterDPI int
MaxPages int
// Timeout bounds each subprocess call. Zero => defaultTimeout.
Timeout time.Duration
// Logger receives best-effort diagnostics. Optional (nil = silent).
Logger *slog.Logger
}
// New builds a Detector from the resolved config values.
func New(enabled bool, marker string, markerPrefix bool, pdftoppmPath, tmpDir string) *Detector {
return &Detector{
Enabled: enabled,
Marker: marker,
MarkerPrefix: markerPrefix,
PdftoppmPath: pdftoppmPath,
TmpDir: tmpDir,
}
}
// Result describes a completed split.
type Result struct {
// Parts holds the absolute paths of the produced part PDFs, in original
// page order. Always at least one entry when Split reports split == true.
Parts []string
// PartPageRanges[i] holds the 1-based [first,last] page numbers of Parts[i]
// within the original document — audit-log material, so the aggregation of
// pages into parts stays reconstructible after the original is gone.
PartPageRanges [][2]int
// SeparatorPages holds the 1-based page numbers that carried the marker
// barcode and were therefore dropped.
SeparatorPages []int
// PageCount is the original document's total page count.
PageCount int
// Cleanup removes the scratch directory holding Parts. Never nil when
// Split returned split == true; callers must defer it.
Cleanup func()
}
// Split analyses pdfPath for separator pages and, if any are found, produces
// one part PDF per content segment.
//
// Returns split == false (with a nil Result) for every "carry on normally"
// outcome: detector disabled, poppler/zbarimg missing, fewer than two pages,
// page count above MaxPages, no separator barcode found, or every page being a
// separator page. Only genuinely unexpected failures return an error, and even
// those are meant to be treated by the caller as "archive unsplit" plus an
// audit entry — never as an upload failure.
//
// pdfPath must be a scratch/inbox file: it is only ever read, but the whole
// point of this function is that it runs BEFORE the file becomes a WORM
// archive object, so it must never be pointed at store/.
func (d *Detector) Split(ctx context.Context, pdfPath string) (res *Result, split bool, err error) {
if d == nil || !d.Enabled {
return nil, false, nil
}
for _, bin := range []string{d.pdfinfoPath(), d.pdftoppmPath(), d.pdfseparatePath(), d.pdfunitePath()} {
if _, lookErr := exec.LookPath(bin); lookErr != nil {
d.log(slog.LevelWarn, "pagesplit skipped: poppler binary not found in PATH",
"binary", bin, "err", lookErr)
return nil, false, nil
}
}
if _, lookErr := exec.LookPath("zbarimg"); lookErr != nil {
d.log(slog.LevelWarn, "pagesplit skipped: zbarimg not found in PATH", "err", lookErr)
return nil, false, nil
}
pageCount, err := d.pageCount(ctx, pdfPath)
if err != nil {
return nil, false, fmt.Errorf("pagesplit: page count: %w", err)
}
if pageCount < 2 {
return nil, false, nil
}
if pageCount > d.maxPages() {
d.log(slog.LevelWarn, "pagesplit skipped: page count above limit",
"file", pdfPath, "pages", pageCount, "max_pages", d.maxPages())
return nil, false, nil
}
jobDir := filepath.Join(d.tmpDir(), "split-"+randomID())
if mkErr := os.MkdirAll(jobDir, 0o750); mkErr != nil {
return nil, false, fmt.Errorf("pagesplit: create scratch dir: %w", mkErr)
}
cleanup := func() { os.RemoveAll(jobDir) }
// Anything below that returns without a successful split must not leak the
// scratch directory; the success path hands cleanup to the caller instead.
ok := false
defer func() {
if !ok {
cleanup()
}
}()
sepPages, err := d.detectSeparatorPages(ctx, pdfPath, jobDir, pageCount)
if err != nil {
return nil, false, fmt.Errorf("pagesplit: separator detection: %w", err)
}
if len(sepPages) == 0 {
return nil, false, nil
}
ranges := contentRanges(pageCount, sepPages)
if len(ranges) == 0 {
// Pathological upload: only separator sheets, no content at all. Do not
// silently discard it — archive the original unsplit so the operator
// sees what was scanned.
d.log(slog.LevelWarn, "pagesplit skipped: document consists of separator pages only",
"file", pdfPath, "pages", pageCount)
return nil, false, nil
}
partsDir := filepath.Join(jobDir, "parts")
if mkErr := os.MkdirAll(partsDir, 0o750); mkErr != nil {
return nil, false, fmt.Errorf("pagesplit: create parts dir: %w", mkErr)
}
var parts []string
for i, rg := range ranges {
partPath, perr := d.extractRange(ctx, pdfPath, partsDir, i+1, rg[0], rg[1])
if perr != nil {
// Partial split is never handed out — the caller falls back to
// archiving the unsplit original.
return nil, false, fmt.Errorf("pagesplit: extract pages %d-%d: %w", rg[0], rg[1], perr)
}
parts = append(parts, partPath)
}
d.log(slog.LevelInfo, "pagesplit produced parts",
"file", pdfPath, "pages", pageCount, "separator_pages", sepPages, "parts", len(parts))
ok = true
return &Result{
Parts: parts,
PartPageRanges: ranges,
SeparatorPages: sepPages,
PageCount: pageCount,
Cleanup: cleanup,
}, true, nil
}
// IsSeparatorValue reports whether a decoded barcode payload marks a separator
// page under this detector's marker configuration.
func (d *Detector) IsSeparatorValue(value string) bool {
v := strings.ToUpper(strings.TrimSpace(value))
m := strings.ToUpper(strings.TrimSpace(d.marker()))
if v == "" || m == "" {
return false
}
if d.MarkerPrefix {
return strings.HasPrefix(v, m)
}
return v == m
}
var pdfinfoPagesRegex = regexp.MustCompile(`(?m)^Pages:\s+(\d+)`)
// pageCount reads the page count via `pdfinfo`.
func (d *Detector) pageCount(ctx context.Context, pdfPath string) (int, error) {
cctx, cancel := context.WithTimeout(ctx, d.timeout())
defer cancel()
cmd := exec.CommandContext(cctx, d.pdfinfoPath(), pdfPath)
var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return 0, fmt.Errorf("pdfinfo failed: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
m := pdfinfoPagesRegex.FindStringSubmatch(out.String())
if m == nil {
return 0, fmt.Errorf("pdfinfo output had no Pages line")
}
n, err := strconv.Atoi(m[1])
if err != nil {
return 0, fmt.Errorf("pdfinfo page count unparseable: %w", err)
}
return n, nil
}
// pageNumRegex pulls the page number out of the filenames pdftoppm/pdfseparate
// generate (page-01.png, page-1.png, seg-12.pdf, ...). Sorting on that number
// rather than lexically matters as soon as a run crosses 9 or 99 pages.
var pageNumRegex = regexp.MustCompile(`(\d+)\D*$`)
// detectSeparatorPages rasterizes every page once and decodes barcodes on it,
// returning the 1-based page numbers that carry the marker.
func (d *Detector) detectSeparatorPages(ctx context.Context, pdfPath, jobDir string, pageCount int) ([]int, error) {
rasterDir := filepath.Join(jobDir, "raster")
if err := os.MkdirAll(rasterDir, 0o750); err != nil {
return nil, fmt.Errorf("create raster dir: %w", err)
}
cctx, cancel := context.WithTimeout(ctx, d.timeout())
defer cancel()
prefix := filepath.Join(rasterDir, "page")
cmd := exec.CommandContext(cctx, d.pdftoppmPath(),
"-r", strconv.Itoa(d.rasterDPI()), "-png", pdfPath, prefix)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return nil, fmt.Errorf("pdftoppm failed: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
pages, err := sortedNumberedFiles(rasterDir, ".png")
if err != nil {
return nil, err
}
if len(pages) != pageCount {
// Mismatch means the page-number mapping below cannot be trusted, and a
// wrong mapping would cut the document in the wrong place — refuse.
return nil, fmt.Errorf("rasterized %d pages but pdfinfo reported %d", len(pages), pageCount)
}
var sep []int
for i, page := range pages {
codes, decErr := barcode.DecodeBarcodes(ctx, page)
if decErr != nil {
// Best-effort per page, exactly as in internal/ocr: a page whose
// barcode pass errored is simply treated as a content page.
continue
}
for _, code := range codes {
if d.IsSeparatorValue(code) {
sep = append(sep, i+1)
break
}
}
}
return sep, nil
}
// contentRanges turns a page count plus the separator page numbers into the
// 1-based inclusive page ranges of the content segments, dropping the
// separator pages themselves and any empty segment (two adjacent separator
// sheets, or one at the very start/end).
func contentRanges(pageCount int, sepPages []int) [][2]int {
isSep := make(map[int]bool, len(sepPages))
for _, p := range sepPages {
isSep[p] = true
}
var ranges [][2]int
start := 0
for p := 1; p <= pageCount; p++ {
if isSep[p] {
if start != 0 {
ranges = append(ranges, [2]int{start, p - 1})
start = 0
}
continue
}
if start == 0 {
start = p
}
}
if start != 0 {
ranges = append(ranges, [2]int{start, pageCount})
}
return ranges
}
// extractRange writes pages [first,last] of pdfPath into one PDF under
// partsDir, using pdfseparate (per-page extraction) plus pdfunite (re-merge)
// — the poppler-only equivalent of `qpdf --pages`.
func (d *Detector) extractRange(ctx context.Context, pdfPath, partsDir string, index, first, last int) (string, error) {
segDir := filepath.Join(partsDir, fmt.Sprintf("seg-%03d", index))
if err := os.MkdirAll(segDir, 0o750); err != nil {
return "", fmt.Errorf("create segment dir: %w", err)
}
sepCtx, cancelSep := context.WithTimeout(ctx, d.timeout())
defer cancelSep()
pattern := filepath.Join(segDir, "p-%d.pdf")
cmd := exec.CommandContext(sepCtx, d.pdfseparatePath(),
"-f", strconv.Itoa(first), "-l", strconv.Itoa(last), pdfPath, pattern)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return "", fmt.Errorf("pdfseparate failed: %w (%s)", err, strings.TrimSpace(stderr.String()))
}
pageFiles, err := sortedNumberedFiles(segDir, ".pdf")
if err != nil {
return "", err
}
want := last - first + 1
if len(pageFiles) != want {
return "", fmt.Errorf("pdfseparate produced %d pages, expected %d", len(pageFiles), want)
}
if len(pageFiles) == 1 {
// Single-page segment: the extracted page already IS the part.
return pageFiles[0], nil
}
uniteCtx, cancelUnite := context.WithTimeout(ctx, d.timeout())
defer cancelUnite()
outPath := filepath.Join(partsDir, fmt.Sprintf("part-%03d.pdf", index))
args := append(append([]string{}, pageFiles...), outPath)
uniteCmd := exec.CommandContext(uniteCtx, d.pdfunitePath(), args...)
var uniteErr bytes.Buffer
uniteCmd.Stderr = &uniteErr
if err := uniteCmd.Run(); err != nil {
os.Remove(outPath)
return "", fmt.Errorf("pdfunite failed: %w (%s)", err, strings.TrimSpace(uniteErr.String()))
}
if fi, statErr := os.Stat(outPath); statErr != nil || fi.Size() == 0 {
os.Remove(outPath)
return "", fmt.Errorf("pdfunite produced empty/missing output: %v", statErr)
}
return outPath, nil
}
// sortedNumberedFiles lists dir's files with the given extension, sorted by
// the trailing number in their name (numeric, not lexical).
func sortedNumberedFiles(dir, ext string) ([]string, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return nil, fmt.Errorf("read dir %s: %w", dir, err)
}
type numbered struct {
path string
num int
}
var found []numbered
for _, entry := range entries {
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ext) {
continue
}
num := 0
base := strings.TrimSuffix(entry.Name(), ext)
if m := pageNumRegex.FindStringSubmatch(base); m != nil {
num, _ = strconv.Atoi(m[1])
}
found = append(found, numbered{path: filepath.Join(dir, entry.Name()), num: num})
}
sort.Slice(found, func(i, j int) bool {
if found[i].num != found[j].num {
return found[i].num < found[j].num
}
return found[i].path < found[j].path
})
paths := make([]string, 0, len(found))
for _, f := range found {
paths = append(paths, f.path)
}
return paths, nil
}
// randomID returns a random hex string for scratch directory names. Kept
// dependency-free, same approach as internal/ocr.randomID.
func randomID() string {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return fmt.Sprintf("job-%d", time.Now().UnixNano())
}
return hex.EncodeToString(b)
}
func (d *Detector) log(level slog.Level, msg string, args ...any) {
if d == nil || d.Logger == nil {
return
}
d.Logger.Log(context.Background(), level, msg, args...)
}
func (d *Detector) marker() string {
if strings.TrimSpace(d.Marker) == "" {
return DefaultMarker
}
return d.Marker
}
func (d *Detector) pdftoppmPath() string { return orDefault(d.PdftoppmPath, "pdftoppm") }
func (d *Detector) pdfinfoPath() string { return orDefault(d.PdfinfoPath, "pdfinfo") }
func (d *Detector) pdfseparatePath() string { return orDefault(d.PdfseparatePath, "pdfseparate") }
func (d *Detector) pdfunitePath() string { return orDefault(d.PdfunitePath, "pdfunite") }
func orDefault(v, def string) string {
if strings.TrimSpace(v) == "" {
return def
}
return v
}
func (d *Detector) tmpDir() string {
if strings.TrimSpace(d.TmpDir) == "" {
return os.TempDir()
}
return d.TmpDir
}
func (d *Detector) rasterDPI() int {
if d.RasterDPI <= 0 {
return defaultRasterDPI
}
return d.RasterDPI
}
func (d *Detector) maxPages() int {
if d.MaxPages <= 0 {
return defaultMaxPages
}
return d.MaxPages
}
func (d *Detector) timeout() time.Duration {
if d.Timeout <= 0 {
return defaultTimeout
}
return d.Timeout
}
+398
View File
@@ -0,0 +1,398 @@
// Package sftpserver implements an embedded, per-tenant SFTP server for
// archivdms. Instead of provisioning real OS users + OpenSSH
// ChrootDirectory per tenant, the server runs inside the archivdms binary
// and enforces tenant isolation entirely in software:
//
// - Authentication is checked against the `sftp_credentials` table
// (internal/storage/sftp_credentials.go), a narrow, independently
// revocable credential — not a full user login.
// - Once authenticated, a tenant is virtually "locked" into
// `<storage.base_path>/inbox/<tenant_id>/`: the SFTP handlers only ever
// resolve paths relative to that directory and reject any path that
// would escape it (no OS-level chroot, no setuid, no real filesystem
// jail — just careful path handling).
// - A polling watcher goroutine picks up files dropped into that
// directory and feeds them through the exact same
// inbox->hash->store->OCR->DB pipeline as the HTTP upload endpoint
// (see internal/api/document_handlers.go storeUploadedFile, exposed
// here via the UploadFunc callback to avoid an import cycle between
// internal/api and internal/sftpserver).
package sftpserver
import (
"context"
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"encoding/pem"
"errors"
"fmt"
"io"
"log/slog"
"net"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"github.com/pkg/sftp"
"golang.org/x/crypto/ssh"
"archivdms/config"
"archivdms/internal/audit"
"archivdms/internal/storage"
)
// pollInterval is how often the watcher scans inbox directories for new
// files dropped over SFTP. No fsnotify dependency — a simple polling loop is
// good enough for this volume/latency profile (analogous to the project's
// existing cron-style background jobs).
const pollInterval = 5 * time.Second
// UploadFunc is the shared upload-pipeline entry point, implemented by
// internal/api.Server.StoreUploadedFile. Kept as a function value (rather
// than importing internal/api directly) to avoid an import cycle:
// internal/api already imports internal/storage and internal/audit, and
// wiring happens the other way around in cmd/archivdms/main.go.
type UploadFunc func(ctx context.Context, tenantID int64, title, docType, correspondent string, file io.Reader, filename, contentType string) (*storage.Document, string, error)
// Server is the embedded per-tenant SFTP server.
type Server struct {
cfg config.SFTPConfig
storageCfg config.StorageConfig
store *storage.Store
audlog *audit.Logger
logger *slog.Logger
upload UploadFunc
listener net.Listener
sshCfg *ssh.ServerConfig
stopOnce sync.Once
stopCh chan struct{}
}
// New constructs an SFTP server. Call Start to begin listening and Stop to
// shut down.
func New(cfg config.SFTPConfig, storageCfg config.StorageConfig, store *storage.Store, audlog *audit.Logger, logger *slog.Logger, upload UploadFunc) *Server {
return &Server{
cfg: cfg,
storageCfg: storageCfg,
store: store,
audlog: audlog,
logger: logger,
upload: upload,
stopCh: make(chan struct{}),
}
}
// Start loads/generates the host key, opens the listener, and launches the
// accept loop plus the inbox watcher as background goroutines. It returns
// once the listener is up (or an error occurred setting it up); the accept
// loop itself keeps running in the background.
func (s *Server) Start(ctx context.Context) error {
signer, err := s.loadOrCreateHostKey()
if err != nil {
return fmt.Errorf("sftpserver: host key: %w", err)
}
s.sshCfg = &ssh.ServerConfig{
PasswordCallback: s.passwordCallback,
}
s.sshCfg.AddHostKey(signer)
bind := s.cfg.ResolvedBind()
ln, err := net.Listen("tcp", bind)
if err != nil {
return fmt.Errorf("sftpserver: listen %s: %w", bind, err)
}
s.listener = ln
s.logger.Info("sftp server listening", "addr", bind)
go s.acceptLoop()
go s.watchLoop(ctx)
return nil
}
// Stop closes the listener, ending the accept loop, and signals the watcher
// to exit.
func (s *Server) Stop() {
s.stopOnce.Do(func() {
close(s.stopCh)
if s.listener != nil {
_ = s.listener.Close()
}
})
}
// --- host key bootstrap ---
func (s *Server) loadOrCreateHostKey() (ssh.Signer, error) {
path := s.cfg.ResolvedHostKeyPath(s.storageCfg.BasePath)
if data, err := os.ReadFile(path); err == nil {
return ssh.ParsePrivateKey(data)
} else if !os.IsNotExist(err) {
return nil, err
}
s.logger.Info("sftp host key not found, generating a new one", "path", path)
key, err := rsa.GenerateKey(rand.Reader, 4096)
if err != nil {
return nil, fmt.Errorf("generate host key: %w", err)
}
der := x509.MarshalPKCS1PrivateKey(key)
block := &pem.Block{Type: "RSA PRIVATE KEY", Bytes: der}
if dir := filepath.Dir(path); dir != "" && dir != "." {
if err := os.MkdirAll(dir, 0o700); err != nil {
return nil, fmt.Errorf("create host key dir: %w", err)
}
}
if err := os.WriteFile(path, pem.EncodeToMemory(block), 0o600); err != nil {
return nil, fmt.Errorf("write host key: %w", err)
}
return ssh.NewSignerFromKey(key)
}
// --- authentication ---
func (s *Server) passwordCallback(meta ssh.ConnMetadata, password []byte) (*ssh.Permissions, error) {
username := meta.User()
ctx := context.Background()
cred, err := s.store.VerifySFTPLogin(ctx, username, string(password))
success := err == nil
detail := ""
if err != nil {
detail = err.Error()
}
var tenantID *int64
if cred != nil {
tenantID = &cred.TenantID
}
if s.audlog != nil {
s.audlog.Log(audit.Entry{
EventType: audit.EventSFTPLogin,
Username: username,
IPAddress: remoteIPFromConn(meta.RemoteAddr()),
TenantID: tenantID,
Success: success,
Detail: detail,
})
}
if !success {
return nil, fmt.Errorf("sftpserver: authentication failed")
}
_ = s.store.TouchSFTPLastLogin(ctx, cred.ID)
return &ssh.Permissions{
Extensions: map[string]string{
"tenant_id": strconv.FormatInt(cred.TenantID, 10),
"username": username,
},
}, nil
}
func remoteIPFromConn(addr net.Addr) string {
if addr == nil {
return ""
}
host, _, err := net.SplitHostPort(addr.String())
if err != nil {
return addr.String()
}
return host
}
// --- accept loop ---
func (s *Server) acceptLoop() {
for {
conn, err := s.listener.Accept()
if err != nil {
select {
case <-s.stopCh:
return
default:
s.logger.Warn("sftp accept error", "err", err)
continue
}
}
go s.handleConn(conn)
}
}
func (s *Server) handleConn(conn net.Conn) {
sshConn, chans, reqs, err := ssh.NewServerConn(conn, s.sshCfg)
if err != nil {
s.logger.Warn("sftp handshake failed", "err", err, "remote", conn.RemoteAddr())
return
}
defer sshConn.Close()
tenantIDStr := sshConn.Permissions.Extensions["tenant_id"]
tenantID, err := strconv.ParseInt(tenantIDStr, 10, 64)
if err != nil {
s.logger.Error("sftp connection missing tenant_id extension", "err", err)
return
}
go ssh.DiscardRequests(reqs)
for newChan := range chans {
if newChan.ChannelType() != "session" {
_ = newChan.Reject(ssh.UnknownChannelType, "unsupported channel type")
continue
}
channel, requests, err := newChan.Accept()
if err != nil {
s.logger.Warn("sftp channel accept failed", "err", err)
continue
}
go s.handleSession(channel, requests, tenantID)
}
}
func (s *Server) handleSession(channel ssh.Channel, requests <-chan *ssh.Request, tenantID int64) {
defer channel.Close()
for req := range requests {
ok := req.Type == "subsystem" && string(req.Payload[4:]) == "sftp"
if req.WantReply {
_ = req.Reply(ok, nil)
}
if !ok {
continue
}
root := filepath.Join(s.storageCfg.InboxPath(), strconv.FormatInt(tenantID, 10))
if err := os.MkdirAll(root, 0o750); err != nil {
s.logger.Error("sftp: create tenant inbox dir failed", "tenant_id", tenantID, "err", err)
return
}
fs := &tenantFS{root: root}
handlers := sftp.Handlers{
FileGet: fs,
FilePut: fs,
FileCmd: fs,
FileList: fs,
}
server := sftp.NewRequestServer(channel, handlers)
if err := server.Serve(); err != nil && err != io.EOF {
s.logger.Warn("sftp session ended with error", "tenant_id", tenantID, "err", err)
}
_ = server.Close()
return
}
}
// --- watcher: picks up files dropped into inbox/<tenant_id>/ and feeds them
// through the shared upload pipeline ---
func (s *Server) watchLoop(ctx context.Context) {
ticker := time.NewTicker(pollInterval)
defer ticker.Stop()
for {
select {
case <-s.stopCh:
return
case <-ctx.Done():
return
case <-ticker.C:
s.scanInbox(ctx)
}
}
}
func (s *Server) scanInbox(ctx context.Context) {
base := s.storageCfg.InboxPath()
tenantDirs, err := os.ReadDir(base)
if err != nil {
if !os.IsNotExist(err) {
s.logger.Warn("sftp watcher: read inbox root failed", "err", err)
}
return
}
for _, td := range tenantDirs {
if !td.IsDir() {
continue
}
tenantID, err := strconv.ParseInt(td.Name(), 10, 64)
if err != nil {
continue // not a tenant directory (e.g. stray file), skip
}
s.scanTenantInbox(ctx, tenantID, filepath.Join(base, td.Name()))
}
}
func (s *Server) scanTenantInbox(ctx context.Context, tenantID int64, dir string) {
entries, err := os.ReadDir(dir)
if err != nil {
s.logger.Warn("sftp watcher: read tenant inbox failed", "tenant_id", tenantID, "err", err)
return
}
for _, e := range entries {
if e.IsDir() {
continue
}
path := filepath.Join(dir, e.Name())
s.processInboxFile(ctx, tenantID, path, e.Name())
}
}
func (s *Server) processInboxFile(ctx context.Context, tenantID int64, path, filename string) {
// Skip files still being written (e.g. an in-progress SFTP PUT). A
// simple heuristic: if the file's mtime is very recent, give the next
// poll cycle a chance to see it settle instead of processing a partial
// upload.
info, err := os.Stat(path)
if err != nil {
return // vanished since ReadDir, e.g. concurrent processing
}
if time.Since(info.ModTime()) < pollInterval {
return
}
f, err := os.Open(path)
if err != nil {
s.logger.Warn("sftp watcher: open inbox file failed", "path", path, "err", err)
return
}
title := strings.TrimSuffix(filename, filepath.Ext(filename))
doc, warn, err := s.upload(ctx, tenantID, title, "", "", f, filename, "")
f.Close()
if err != nil {
if errors.Is(err, storage.ErrDuplicateContentHash) {
s.logger.Info("sftp watcher: duplicate content, discarding", "path", path)
} else {
s.logger.Error("sftp watcher: upload pipeline failed", "path", path, "err", err)
if s.audlog != nil {
s.audlog.Log(audit.Entry{EventType: audit.EventDocumentCreate, Username: "sftp:tenant-" + strconv.FormatInt(tenantID, 10), TenantID: &tenantID, Success: false, Detail: err.Error()})
}
return // leave the file in place for a retry on the next cycle
}
} else {
s.logger.Info("sftp watcher: document created", "document_id", doc.ID, "path", path)
if warn != "" && s.logger != nil {
s.logger.Warn("sftp watcher: upload succeeded with warning", "document_id", doc.ID, "warn", warn)
}
}
// Remove the original SFTP-dropped file: storeUploadedFile writes its own
// copy into inbox/<tenant>/<random>.<ext> and moves *that* into store/, so
// this original drop file is no longer needed either way (processed or
// confirmed duplicate).
if err := os.Remove(path); err != nil {
s.logger.Warn("sftp watcher: cleanup of inbox file failed", "path", path, "err", err)
}
}
+133
View File
@@ -0,0 +1,133 @@
package sftpserver
import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"github.com/pkg/sftp"
)
// tenantFS implements the four github.com/pkg/sftp request-server
// interfaces (FileReader/FileWriter/FileCmder/FileLister) on top of a single
// real directory (root) — the authenticated tenant's
// inbox/<tenant_id>/ folder.
//
// This is the "virtual chroot": every incoming SFTP path is resolved
// relative to root and validated to never escape it (no "..", no absolute
// paths pointing elsewhere). v1 intentionally supports only a flat
// directory — no subfolder create/navigate/delete — which keeps the path
// validation trivial: a request path may only name a direct child of root.
type tenantFS struct {
root string
}
// resolve maps a virtual SFTP path ("/", "/foo.pdf", ...) onto a real path
// under fs.root, rejecting anything that isn't a direct child of the root
// (blocks path traversal and subfolder use in one check).
func (fs *tenantFS) resolve(virtual string) (string, error) {
clean := filepath.Clean("/" + virtual)
if clean == "/" {
return fs.root, nil
}
clean = strings.TrimPrefix(clean, "/")
if strings.Contains(clean, "/") || clean == ".." || clean == "." {
return "", errors.New("sftpserver: path escapes tenant root or is not a direct child")
}
return filepath.Join(fs.root, clean), nil
}
// Fileread implements sftp.FileReader (GET). Reading back an already
// uploaded-but-not-yet-processed file is allowed (harmless), but there is
// nothing to read once the watcher has moved the file into store/ (by
// design — inbox/ is a transient staging area, not a browsable archive).
func (fs *tenantFS) Fileread(r *sftp.Request) (io.ReaderAt, error) {
path, err := fs.resolve(r.Filepath)
if err != nil {
return nil, err
}
f, err := os.Open(path)
if err != nil {
return nil, err
}
return f, nil
}
// Filewrite implements sftp.FileWriter (PUT). New files are created at the
// root of the tenant's inbox only; existing files may not be overwritten
// (O_EXCL) to avoid a client silently clobbering a file the watcher hasn't
// picked up yet.
func (fs *tenantFS) Filewrite(r *sftp.Request) (io.WriterAt, error) {
path, err := fs.resolve(r.Filepath)
if err != nil {
return nil, err
}
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o640)
if err != nil {
return nil, err
}
return f, nil
}
// Filecmd implements sftp.FileCmder for out-of-band filesystem operations
// (Remove, Rename, Mkdir, Setstat, ...). v1 deliberately supports none of
// these beyond what's needed for a plain "put a file" workflow — clients
// get a clean permission error rather than silently succeeding.
func (fs *tenantFS) Filecmd(r *sftp.Request) error {
return errors.New("sftpserver: operation not permitted (only uploading new files is supported)")
}
// Filelist implements sftp.FileLister (LIST/STAT). Listing the root shows
// the tenant's pending (not-yet-watched) inbox files; anything else is
// rejected by resolve.
func (fs *tenantFS) Filelist(r *sftp.Request) (sftp.ListerAt, error) {
switch r.Method {
case "List":
path, err := fs.resolve(r.Filepath)
if err != nil {
return nil, err
}
entries, err := os.ReadDir(path)
if err != nil {
return nil, err
}
infos := make([]os.FileInfo, 0, len(entries))
for _, e := range entries {
info, err := e.Info()
if err != nil {
continue
}
infos = append(infos, info)
}
return listerAt(infos), nil
case "Stat", "Lstat":
path, err := fs.resolve(r.Filepath)
if err != nil {
return nil, err
}
info, err := os.Stat(path)
if err != nil {
return nil, err
}
return listerAt([]os.FileInfo{info}), nil
default:
return nil, errors.New("sftpserver: unsupported list method " + r.Method)
}
}
// listerAt is the minimal []os.FileInfo -> sftp.ListerAt adapter expected by
// github.com/pkg/sftp's request server.
type listerAt []os.FileInfo
func (l listerAt) ListAt(dst []os.FileInfo, offset int64) (int, error) {
if offset >= int64(len(l)) {
return 0, io.EOF
}
n := copy(dst, l[offset:])
if n < len(dst) {
return n, io.EOF
}
return n, nil
}
+173
View File
@@ -0,0 +1,173 @@
// Per-tenant API keys for the read-only Buchhaltungs-Pull-API (see
// migrations/026_accounting_api_keys.sql and
// internal/api/accounting_handlers.go).
//
// Token handling follows exactly the share-link pattern in shares.go: the raw
// key is generated once (32 bytes crypto/rand, base64url, with a fixed
// "adms_" prefix so it is recognisable in logs/config files), returned to the
// caller EXACTLY once at creation time, and only its hex SHA-256 hash is ever
// persisted. Lookup for authentication is always by key_hash, never by id.
//
// Keys are never hard-deleted: revoking only sets revoked_at, so the audit
// trail of which key pulled which documents stays resolvable (GoBD
// Nachvollziehbarkeit).
package storage
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
)
// accountingKeyPrefix marks a raw accounting API key as such. It is part of
// the hashed value (the whole string is hashed), it is NOT a separate column.
const accountingKeyPrefix = "adms_"
// ErrAccountingKeyNotFound is returned when a key lookup (by id+tenant or by
// key_hash) matches no usable row — unknown key, wrong tenant, or revoked.
var ErrAccountingKeyNotFound = errors.New("storage: accounting api key not found")
// AccountingAPIKey is the safe view of an accounting_api_keys row. The hash is
// deliberately NOT part of this struct so it can never be serialised into an
// API response, and the plaintext key exists only as the second return value
// of CreateAccountingAPIKey.
type AccountingAPIKey struct {
ID int64 `json:"id"`
TenantID int64 `json:"tenant_id"`
Label string `json:"label"`
CreatedBy *int64 `json:"created_by,omitempty"`
CreatedAt time.Time `json:"created_at"`
RevokedAt *time.Time `json:"revoked_at,omitempty"`
LastUsedAt *time.Time `json:"last_used_at,omitempty"`
}
func (s *Store) initAccountingAPIKeysSchema(ctx context.Context) error {
_, err := s.db.Exec(ctx, `
-- No FK on tenant_id / created_by: consistent with the rest of the
-- schema (plain BIGINT), because tenants/users are owned by other
-- stores that initialise after storage.New().
CREATE TABLE IF NOT EXISTS accounting_api_keys (
id BIGSERIAL PRIMARY KEY,
tenant_id BIGINT NOT NULL,
key_hash TEXT NOT NULL UNIQUE,
label TEXT NOT NULL DEFAULT '',
created_by BIGINT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
revoked_at TIMESTAMPTZ,
last_used_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_accounting_api_keys_tenant ON accounting_api_keys(tenant_id);
`)
if err != nil {
return fmt.Errorf("storage: create accounting_api_keys table: %w", err)
}
return nil
}
// hashAccountingKey returns the hex-encoded SHA-256 of a raw accounting API
// key, the value persisted in / looked up from accounting_api_keys.key_hash.
func hashAccountingKey(key string) string {
sum := sha256.Sum256([]byte(key))
return hex.EncodeToString(sum[:])
}
// CreateAccountingAPIKey inserts a new key for a tenant and returns the stored
// row plus the raw (plaintext) key. The plaintext is returned ONLY here and
// never again — only its SHA-256 hash is persisted.
func (s *Store) CreateAccountingAPIKey(ctx context.Context, tenantID int64, label string, createdBy *int64) (*AccountingAPIKey, string, error) {
rawBytes := make([]byte, 32)
if _, err := rand.Read(rawBytes); err != nil {
return nil, "", fmt.Errorf("storage: generate accounting api key: %w", err)
}
key := accountingKeyPrefix + base64.RawURLEncoding.EncodeToString(rawBytes)
var k AccountingAPIKey
err := s.db.QueryRow(ctx, `
INSERT INTO accounting_api_keys (tenant_id, key_hash, label, created_by)
VALUES ($1, $2, $3, $4)
RETURNING id, tenant_id, label, created_by, created_at, revoked_at, last_used_at
`, tenantID, hashAccountingKey(key), label, createdBy,
).Scan(&k.ID, &k.TenantID, &k.Label, &k.CreatedBy, &k.CreatedAt, &k.RevokedAt, &k.LastUsedAt)
if err != nil {
return nil, "", fmt.Errorf("storage: create accounting api key: %w", err)
}
return &k, key, nil
}
// ResolveAccountingAPIKey authenticates a raw key: it hashes the key, looks the
// row up by key_hash, rejects revoked keys, refreshes last_used_at and returns
// the owning tenant id plus the key id.
//
// The returned tenantID is THE ONLY trusted tenant source for the pull
// endpoints — no caller may take a tenant_id from the request itself.
// Unknown and revoked keys both yield ErrAccountingKeyNotFound so the caller
// cannot distinguish them.
func (s *Store) ResolveAccountingAPIKey(ctx context.Context, rawKey string) (tenantID int64, keyID int64, err error) {
if rawKey == "" {
return 0, 0, ErrAccountingKeyNotFound
}
// Single statement: authenticate + touch last_used_at atomically. The
// revoked_at IS NULL guard lives in the WHERE clause, so a revoked key can
// never return a tenant id.
err = s.db.QueryRow(ctx, `
UPDATE accounting_api_keys
SET last_used_at = now()
WHERE key_hash = $1 AND revoked_at IS NULL
RETURNING tenant_id, id
`, hashAccountingKey(rawKey)).Scan(&tenantID, &keyID)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return 0, 0, ErrAccountingKeyNotFound
}
return 0, 0, fmt.Errorf("storage: resolve accounting api key: %w", err)
}
return tenantID, keyID, nil
}
// ListAccountingAPIKeys returns all keys of a tenant (including revoked ones —
// no hard delete), newest first. Never returns the hash or the plaintext.
func (s *Store) ListAccountingAPIKeys(ctx context.Context, tenantID int64) ([]AccountingAPIKey, error) {
rows, err := s.db.Query(ctx, `
SELECT id, tenant_id, label, created_by, created_at, revoked_at, last_used_at
FROM accounting_api_keys
WHERE tenant_id = $1
ORDER BY created_at DESC
`, tenantID)
if err != nil {
return nil, fmt.Errorf("storage: list accounting api keys: %w", err)
}
defer rows.Close()
out := make([]AccountingAPIKey, 0)
for rows.Next() {
var k AccountingAPIKey
if err := rows.Scan(&k.ID, &k.TenantID, &k.Label, &k.CreatedBy, &k.CreatedAt, &k.RevokedAt, &k.LastUsedAt); err != nil {
return nil, fmt.Errorf("storage: scan accounting api key: %w", err)
}
out = append(out, k)
}
return out, rows.Err()
}
// RevokeAccountingAPIKey marks a key as revoked (never hard-deleted), scoped to
// tenant ownership (IDOR guard: id AND tenant_id). Returns
// ErrAccountingKeyNotFound when no key of that id belongs to the tenant.
func (s *Store) RevokeAccountingAPIKey(ctx context.Context, id, tenantID int64) error {
tag, err := s.db.Exec(ctx, `
UPDATE accounting_api_keys SET revoked_at = now()
WHERE id = $1 AND tenant_id = $2 AND revoked_at IS NULL
`, id, tenantID)
if err != nil {
return fmt.Errorf("storage: revoke accounting api key: %w", err)
}
if tag.RowsAffected() == 0 {
return ErrAccountingKeyNotFound
}
return nil
}
+218
View File
@@ -0,0 +1,218 @@
// Read-only query layer for the Buchhaltungs-Pull-API (see
// internal/api/accounting_handlers.go). Deliberately separate from
// documents.go's ListDocuments: the accounting view is a machine-to-machine
// export with its own reduced projection (no ocr_text, no storage_path, no
// content_hash) and keyset pagination over (created_at, id).
//
// Tenant isolation: every query here takes tenantID as its FIRST parameter and
// filters `WHERE d.tenant_id = $1` — there is no variant without it. The
// caller (the Bearer-auth middleware) derives that id solely from the resolved
// API key, never from the request.
//
// No permission-group ACL filter is applied: an accounting API key is a
// tenant-level machine credential (like the SFTP inbox account), not a user
// session. That is why creating one requires domain_admin.
package storage
import (
"context"
"encoding/base64"
"errors"
"fmt"
"strconv"
"strings"
"time"
"github.com/jackc/pgx/v5"
)
// ErrInvalidAccountingCursor is returned when a client-supplied cursor cannot
// be decoded. The handler maps this to HTTP 400.
var ErrInvalidAccountingCursor = errors.New("storage: invalid accounting cursor")
// AccountingDocument is the reduced, export-safe projection of a document for
// the pull API. storage_path / content_hash / ocr_text are intentionally
// absent (GoBD/security: the WORM location is never exposed; the file is only
// reachable through the streaming endpoint).
type AccountingDocument struct {
ID int64 `json:"id"`
Title string `json:"title"`
DocumentDate *time.Time `json:"document_date,omitempty"`
DocumentDateScore *float64 `json:"document_date_score,omitempty"`
DocTypeID *int64 `json:"doc_type_id,omitempty"`
DocType string `json:"doc_type,omitempty"`
CorrespondentID *int64 `json:"correspondent_id,omitempty"`
Correspondent string `json:"correspondent,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// AccountingDocumentFilter holds the (already validated) query parameters of
// GET /api/v1/accounting/documents. TenantID is NOT part of it on purpose — it
// is passed separately from the API-key context so it can never be overwritten
// by a decoded request body/query.
type AccountingDocumentFilter struct {
// Since/Until bound document_date (inclusive/exclusive respectively).
Since *time.Time
Until *time.Time
// DocTypeID restricts to one document type.
DocTypeID *int64
// MinDateScore is the confidence quality gate (e.g. 0.75); documents with a
// NULL score are excluded as soon as this is set.
MinDateScore *float64
// Cursor is the opaque keyset cursor from a previous page ("" = first page).
Cursor string
// Limit is the page size (already clamped by the handler).
Limit int
}
// AccountingPage is one page of pull results plus the cursor for the next one.
type AccountingPage struct {
Documents []AccountingDocument `json:"documents"`
NextCursor string `json:"next_cursor,omitempty"`
HasMore bool `json:"has_more"`
}
// encodeAccountingCursor builds the opaque keyset cursor from the last row of a
// page. Format (base64url of) "<unix_nanos>:<id>" — the exact tuple the ORDER
// BY / WHERE comparison uses.
func encodeAccountingCursor(createdAt time.Time, id int64) string {
raw := strconv.FormatInt(createdAt.UTC().UnixNano(), 10) + ":" + strconv.FormatInt(id, 10)
return base64.RawURLEncoding.EncodeToString([]byte(raw))
}
// decodeAccountingCursor parses a cursor produced by encodeAccountingCursor.
func decodeAccountingCursor(cursor string) (time.Time, int64, error) {
b, err := base64.RawURLEncoding.DecodeString(cursor)
if err != nil {
return time.Time{}, 0, ErrInvalidAccountingCursor
}
parts := strings.SplitN(string(b), ":", 2)
if len(parts) != 2 {
return time.Time{}, 0, ErrInvalidAccountingCursor
}
nanos, err := strconv.ParseInt(parts[0], 10, 64)
if err != nil {
return time.Time{}, 0, ErrInvalidAccountingCursor
}
id, err := strconv.ParseInt(parts[1], 10, 64)
if err != nil {
return time.Time{}, 0, ErrInvalidAccountingCursor
}
return time.Unix(0, nanos).UTC(), id, nil
}
// ListAccountingDocuments returns one keyset-paginated page of a tenant's
// documents for the pull API, ordered by (created_at, id) ascending so a
// consumer can poll incrementally without ever re-reading or skipping rows.
// Soft-deleted (trashed) documents are excluded.
func (s *Store) ListAccountingDocuments(ctx context.Context, tenantID int64, f AccountingDocumentFilter) (*AccountingPage, error) {
limit := f.Limit
if limit <= 0 {
limit = 100
}
// $1 is always the tenant id — the isolation predicate is not optional.
args := []any{tenantID}
where := []string{"d.tenant_id = $1", "d.deleted_at IS NULL"}
if f.Since != nil {
args = append(args, *f.Since)
where = append(where, fmt.Sprintf("d.document_date >= $%d", len(args)))
}
if f.Until != nil {
args = append(args, *f.Until)
where = append(where, fmt.Sprintf("d.document_date < $%d", len(args)))
}
if f.DocTypeID != nil {
args = append(args, *f.DocTypeID)
where = append(where, fmt.Sprintf("d.doc_type_id = $%d", len(args)))
}
if f.MinDateScore != nil {
args = append(args, *f.MinDateScore)
where = append(where, fmt.Sprintf("d.document_date_score IS NOT NULL AND d.document_date_score >= $%d", len(args)))
}
if f.Cursor != "" {
curTS, curID, err := decodeAccountingCursor(f.Cursor)
if err != nil {
return nil, err
}
args = append(args, curTS, curID)
where = append(where, fmt.Sprintf("(d.created_at, d.id) > ($%d, $%d)", len(args)-1, len(args)))
}
// Fetch one extra row to detect whether a further page exists.
args = append(args, limit+1)
query := `
SELECT d.id, d.title, d.document_date, d.document_date_score,
d.doc_type_id, COALESCE(dt.name, COALESCE(d.doc_type, '')),
d.correspondent_id, COALESCE(c.name, COALESCE(d.correspondent, '')),
d.created_at
FROM documents d
LEFT JOIN document_types dt ON dt.id = d.doc_type_id AND dt.tenant_id = d.tenant_id
LEFT JOIN correspondents c ON c.id = d.correspondent_id AND c.tenant_id = d.tenant_id
WHERE ` + strings.Join(where, " AND ") + `
ORDER BY d.created_at ASC, d.id ASC
LIMIT $` + strconv.Itoa(len(args))
rows, err := s.db.Query(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("storage: list accounting documents: %w", err)
}
defer rows.Close()
out := make([]AccountingDocument, 0, limit)
for rows.Next() {
var d AccountingDocument
if err := rows.Scan(&d.ID, &d.Title, &d.DocumentDate, &d.DocumentDateScore,
&d.DocTypeID, &d.DocType, &d.CorrespondentID, &d.Correspondent, &d.CreatedAt); err != nil {
return nil, fmt.Errorf("storage: scan accounting document: %w", err)
}
out = append(out, d)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("storage: list accounting documents: %w", err)
}
page := &AccountingPage{Documents: out}
if len(out) > limit {
page.Documents = out[:limit]
page.HasMore = true
last := page.Documents[limit-1]
page.NextCursor = encodeAccountingCursor(last.CreatedAt, last.ID)
}
return page, nil
}
// AccountingFileRef carries the server-side-only information needed to stream a
// document's WORM file. The storage path is unexported and reachable only via
// StoragePath(), mirroring storage.ResolvedShare, so a handler cannot
// accidentally serialise it into a response.
type AccountingFileRef struct {
DocumentID int64
Title string
storagePath string
}
// StoragePath returns the WORM path of the file (server-side only).
func (r *AccountingFileRef) StoragePath() string { return r.storagePath }
// GetAccountingDocumentFile resolves a document's WORM file location, scoped to
// the tenant of the API key (id AND tenant_id — IDOR guard). Returns
// ErrDocumentNotFound for unknown id, foreign tenant and trashed document
// alike, so the endpoint never reveals whether a document exists outside the
// caller's tenant.
func (s *Store) GetAccountingDocumentFile(ctx context.Context, id, tenantID int64) (*AccountingFileRef, error) {
var ref AccountingFileRef
err := s.db.QueryRow(ctx, `
SELECT id, title, storage_path
FROM documents
WHERE id = $1 AND tenant_id = $2 AND deleted_at IS NULL
`, id, tenantID).Scan(&ref.DocumentID, &ref.Title, &ref.storagePath)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrDocumentNotFound
}
return nil, fmt.Errorf("storage: get accounting document file: %w", err)
}
return &ref, nil
}

Some files were not shown because too many files have changed in this diff Show More