docs: vollständige README, PROJ-2 Web-Upload, PROJ-19 Mailpiler-Migration
README.md:
- Vollständige Dokumentation aller implementierten Funktionen
- Konfigurationsreferenz, Installation, Systemd, REST-API-Übersicht
- In-Progress-Features klar gekennzeichnet
PROJ-2 (EML/MBOX Web-Upload):
- POST /api/admin/upload – Multipart-Upload mit Hintergrund-Job
- GET /api/admin/upload/{jobID}/progress – Polling
- Admin-Tab "Import" mit Drag-and-Drop, Fortschrittsbalken, Abschlussbericht
PROJ-19 (Mailpiler Migration):
- archivmail import-piler mit Methoden: pilerexport | direct | auto
- Direct: AES-256-CBC + zlib mit defensiven Fallbacks
- pilerexport: Wrapper um mailpilers Export-Tool
Status-Updates: PROJ-3, PROJ-4, PROJ-6, PROJ-7, PROJ-10, PROJ-11 → Deployed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
sysops
@@ -1,94 +1,835 @@
|
|||||||
# archivmail
|
# archivmail
|
||||||
|
|
||||||
Selbst gehostetes Mail-Archiv-System für Unternehmen. E-Mails werden aus IMAP, SMTP und EML/MBOX-Quellen importiert, volltext-indexiert (Xapian) und verschlüsselt archiviert.
|
Ein selbst gehostetes, unternehmenstaugliches Mail-Archiv-System. Empfängt E-Mails über SMTP (BCC-Journaling), IMAP oder Datei-Upload, speichert sie verschlüsselt, indexiert sie für Volltextsuche und stellt sie über eine Web-Oberfläche zur Verfügung.
|
||||||
|
|
||||||
## Features
|
---
|
||||||
|
|
||||||
- Import via IMAP, SMTP-Journaling und EML/MBOX-Upload
|
## Inhaltsverzeichnis
|
||||||
- Volltext-Suche über Xapian
|
|
||||||
- AES-256-GCM-Verschlüsselung der archivierten Mails
|
- [Funktionsübersicht](#funktionsübersicht)
|
||||||
- Anhang-Deduplizierung
|
- [Architektur](#architektur)
|
||||||
- Rollenmodell: `user`, `auditor`, `admin`
|
- [Voraussetzungen](#voraussetzungen)
|
||||||
- Audit-Log (PostgreSQL + Append-only Logdatei)
|
- [Installation](#installation)
|
||||||
|
- [Konfiguration](#konfiguration)
|
||||||
|
- [Systemd-Dienste](#systemd-dienste)
|
||||||
|
- [Funktionen im Detail](#funktionen-im-detail)
|
||||||
|
- [Authentifizierung & Rollen](#authentifizierung--rollen)
|
||||||
|
- [SMTP-Eingang (BCC-Journaling)](#smtp-eingang-bcc-journaling)
|
||||||
|
- [IMAP-Import & Auto-Sync](#imap-import--auto-sync)
|
||||||
|
- [EML/MBOX Web-Upload](#emlmbox-web-upload)
|
||||||
|
- [Speicherung & Verschlüsselung](#speicherung--verschlüsselung)
|
||||||
|
- [Volltext-Suche](#volltext-suche)
|
||||||
|
- [E-Mail-Ansicht](#e-mail-ansicht)
|
||||||
|
- [E-Mail-Export](#e-mail-export)
|
||||||
|
- [Admin-Dashboard](#admin-dashboard)
|
||||||
|
- [Audit-Log](#audit-log)
|
||||||
|
- [Integritätsprüfung](#integritätsprüfung)
|
||||||
|
- [CLI Import & Export](#cli-import--export)
|
||||||
|
- [Mailpiler Migration](#mailpiler-migration)
|
||||||
|
- [REST API](#rest-api)
|
||||||
|
- [In Entwicklung](#in-entwicklung)
|
||||||
|
- [Update](#update)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Funktionsübersicht
|
||||||
|
|
||||||
|
| Funktion | Status |
|
||||||
|
|----------|--------|
|
||||||
|
| Authentifizierung & Rollen (Admin / Auditor / User) | ✅ Deployed |
|
||||||
|
| SMTP-Eingang via BCC-Journaling | ✅ Deployed |
|
||||||
|
| IMAP-Import manuell & automatischer Sync | ✅ Deployed |
|
||||||
|
| EML/MBOX Web-Upload (Admin) | ✅ Deployed |
|
||||||
|
| AES-256-GCM verschlüsselte Speicherung | ✅ Deployed |
|
||||||
|
| Xapian Volltext-Indexierung (async) | ✅ Deployed |
|
||||||
|
| Volltext-Suche mit Filtern & Sortierung | ✅ Deployed |
|
||||||
|
| E-Mail-Ansicht mit HTML-Sandbox | ✅ Deployed |
|
||||||
|
| E-Mail-Export als EML / PDF / ZIP | ✅ Deployed |
|
||||||
|
| Admin-Dashboard (CPU, RAM, Disk, Archiv-Stats) | ✅ Deployed |
|
||||||
|
| Audit-Log (Suche, Login, Import, Export) | ✅ Deployed |
|
||||||
|
| SHA-256 Integritätsprüfung | ✅ Deployed |
|
||||||
|
| CLI: `archivmail import` / `export` | ✅ Deployed |
|
||||||
|
| Mailpiler → archivmail Migration | ✅ Deployed |
|
||||||
|
| Dienste-Verwaltung im Admin-Bereich | ✅ Deployed |
|
||||||
|
| Ordner- & Label-Verwaltung | 🔄 In Progress |
|
||||||
|
| POP3-Import | 🔄 In Progress |
|
||||||
|
| REST API für externe Anbindung (CRM) | 🔄 In Progress |
|
||||||
|
| LDAP / Active Directory | 🔄 In Progress |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architektur
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────┐
|
||||||
|
│ archivmail (Go-Binary) │
|
||||||
|
│ │
|
||||||
|
Postfix BCC ──┤── SMTP-Daemon (Port 2525) │
|
||||||
|
IMAP-Server ──┤── IMAP-Importer + Scheduler │──► PostgreSQL
|
||||||
|
Web-Upload ──┤── HTTP API (Port 8080) │ (Metadaten)
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │──► /var/archivmail/store
|
||||||
|
│ Async Index Worker │ (AES-256-GCM)
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │──► /var/archivmail/xapian
|
||||||
|
│ Xapian Index │ (Volltext-Index)
|
||||||
|
└─────────────────────────────────────┘
|
||||||
|
▲
|
||||||
|
│ /api/*
|
||||||
|
┌─────────────────────────────────────┐
|
||||||
|
│ Next.js Frontend (Port 3000) │
|
||||||
|
│ / /search /mail/[id] /admin │
|
||||||
|
│ /imap │
|
||||||
|
└─────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Komponenten:**
|
||||||
|
|
||||||
|
| Komponente | Technologie | Beschreibung |
|
||||||
|
|------------|-------------|--------------|
|
||||||
|
| Backend | Go 1.23 | REST API, SMTP-Daemon, IMAP-Importer, Storage, Indexierung |
|
||||||
|
| Frontend | Next.js 16 (TypeScript) | Web-Oberfläche, Tailwind CSS, shadcn/ui |
|
||||||
|
| Datenbank | PostgreSQL | Metadaten, Benutzer, Audit-Log, Session-Blacklist |
|
||||||
|
| Volltextsuche | Xapian | BM25-Ranking, Wildcards, Feldpräfixe |
|
||||||
|
| Speicherung | Dateisystem | AES-256-GCM verschlüsselt, SHA-256 Integrität |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Voraussetzungen
|
||||||
|
|
||||||
|
**Server:**
|
||||||
|
- Debian 12 / Ubuntu 22.04 oder neuer
|
||||||
|
- Go ≥ 1.23 (mit CGO für Xapian)
|
||||||
|
- Node.js ≥ 20
|
||||||
|
- PostgreSQL ≥ 14
|
||||||
|
- libxapian-dev
|
||||||
|
|
||||||
|
**Ports (Standard):**
|
||||||
|
- `8080` – HTTP API (Backend)
|
||||||
|
- `3000` – Web-Frontend
|
||||||
|
- `2525` – SMTP-Eingang (BCC-Journaling)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
|
### Automatisch (empfohlen)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Systembenutzer anlegen
|
curl -fsSL https://gitea.perlbach24.de/scripte/archivmail/raw/branch/main/update.sh | bash
|
||||||
useradd -r -s /sbin/nologin archivmail
|
|
||||||
|
|
||||||
# Verzeichnisse anlegen
|
|
||||||
mkdir -p /etc/archivmail
|
|
||||||
mkdir -p /var/archivmail/store
|
|
||||||
mkdir -p /var/archivmail/astore
|
|
||||||
mkdir -p /var/lib/archivmail/xapian
|
|
||||||
mkdir -p /var/log/archivmail
|
|
||||||
|
|
||||||
# Berechtigungen setzen
|
|
||||||
chown -R archivmail:archivmail /var/archivmail /var/lib/archivmail /var/log/archivmail
|
|
||||||
|
|
||||||
# Encryption Key generieren
|
|
||||||
openssl rand -base64 32 > /etc/archivmail/keyfile
|
|
||||||
chmod 400 /etc/archivmail/keyfile
|
|
||||||
chown archivmail:archivmail /etc/archivmail/keyfile
|
|
||||||
|
|
||||||
# Konfiguration
|
|
||||||
cp config.example.yml /etc/archivmail/config.yml
|
|
||||||
# config.yml anpassen (Datenbank, SMTP-Port, etc.)
|
|
||||||
|
|
||||||
# Systemd-Service aktivieren
|
|
||||||
cp archivmail.service /lib/systemd/system/
|
|
||||||
systemctl daemon-reload
|
|
||||||
systemctl enable --now archivmail
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Verzeichnisstruktur
|
Das Skript:
|
||||||
|
1. Klont/aktualisiert den Quellcode
|
||||||
|
2. Baut Backend (Go + CGO/Xapian) und Frontend (Next.js)
|
||||||
|
3. Stoppt Dienste, spielt Binaries ein, startet Dienste neu
|
||||||
|
4. Führt Datenbankmigrationen durch
|
||||||
|
|
||||||
```
|
### Manuell
|
||||||
/etc/archivmail/
|
|
||||||
├── config.yml # Hauptkonfiguration
|
|
||||||
└── keyfile # AES-256-GCM Key (chmod 400)
|
|
||||||
|
|
||||||
/var/archivmail/
|
```bash
|
||||||
├── store/ # Mailkörper (.m, verschlüsselt)
|
# Quellcode
|
||||||
│ └── <server_id>/<customer_id>/<hash>/xxxxx.m
|
git clone https://gitea.perlbach24.de/scripte/archivmail.git /opt/archivmail/_build
|
||||||
└── astore/ # Anhänge dedupliziert (verschlüsselt)
|
cd /opt/archivmail/_build
|
||||||
└── <hash>
|
|
||||||
|
|
||||||
/var/lib/archivmail/
|
# Backend bauen
|
||||||
└── xapian/ # Volltext-Index
|
CGO_ENABLED=1 go build -tags xapian -buildvcs=false \
|
||||||
|
-o /opt/archivmail/bin/archivmail ./cmd/archivmail/
|
||||||
|
|
||||||
/var/log/archivmail/
|
# Frontend bauen
|
||||||
└── audit.log # Audit-Log (JSON Lines, append-only)
|
npm ci && npm run build
|
||||||
|
rsync -a --delete .next/standalone/ /opt/archivmail/web/
|
||||||
|
rsync -a --delete .next/static/ /opt/archivmail/web/.next/static/
|
||||||
```
|
```
|
||||||
|
|
||||||
## Standard-Zugangsdaten
|
### Konfigurationsdatei anlegen
|
||||||
|
|
||||||
> **Wichtig:** Passwörter nach dem ersten Login ändern!
|
```bash
|
||||||
|
mkdir -p /etc/archivmail /var/archivmail/{store,astore,xapian} /var/log/archivmail
|
||||||
|
|
||||||
| Benutzer | Passwort | Rolle |
|
cat > /etc/archivmail/config.yml << 'EOF'
|
||||||
|----------|----------|-------|
|
server:
|
||||||
| `admin@archivmail` | `archivmailrockz` | Admin (Konfiguration, Nutzerverwaltung) |
|
api_port: 8080
|
||||||
| `auditor@archivmail` | `archivmailrockz` | Auditor (alle E-Mails + Audit-Log) |
|
smtp_port: 2525
|
||||||
|
|
||||||
### Rollenübersicht
|
database:
|
||||||
|
host: 127.0.0.1
|
||||||
|
port: 5432
|
||||||
|
name: archivmail
|
||||||
|
user: archivmail
|
||||||
|
password: SICHERES_PASSWORT
|
||||||
|
sslmode: disable
|
||||||
|
|
||||||
| Rolle | E-Mails (eigene) | E-Mails (alle) | Audit-Log | Konfiguration |
|
storage:
|
||||||
|-------|:-:|:-:|:-:|:-:|
|
store_path: /var/archivmail/store
|
||||||
| `user` | ✅ | ❌ | ❌ | ❌ |
|
astore_path: /var/archivmail/astore
|
||||||
| `auditor` | ✅ | ✅ | ✅ | ❌ |
|
xapian_path: /var/archivmail/xapian
|
||||||
| `admin` | ❌ | ❌ | ❌ | ✅ |
|
keyfile: /etc/archivmail/keyfile
|
||||||
|
|
||||||
## Technologie
|
audit:
|
||||||
|
log_path: /var/log/archivmail/audit.log
|
||||||
|
retention_days: 0
|
||||||
|
|
||||||
| Komponente | Technologie |
|
smtp:
|
||||||
|-----------|-------------|
|
enabled: true
|
||||||
| Backend | Go |
|
bind: ":2525"
|
||||||
| Datenbank | PostgreSQL |
|
domain: "archivmail.firma.de"
|
||||||
| Volltext-Index | Xapian |
|
allowed_ips:
|
||||||
| Verschlüsselung | AES-256-GCM |
|
- 127.0.0.1
|
||||||
| Webserver | eingebettet (Go net/http) |
|
- 192.168.1.0/24 # Postfix-Server-IP(s)
|
||||||
|
|
||||||
## Lizenz
|
api:
|
||||||
|
bind: ":8080"
|
||||||
|
secret: ZUFAELLIGER_JWT_SECRET_MINDESTENS_32_ZEICHEN
|
||||||
|
|
||||||
Proprietär
|
index:
|
||||||
|
path: /var/archivmail/xapian
|
||||||
|
backend: xapian
|
||||||
|
batch_size: 100
|
||||||
|
async_queue_size: 1000
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# AES-256 Schlüssel generieren
|
||||||
|
dd if=/dev/urandom bs=32 count=1 > /etc/archivmail/keyfile
|
||||||
|
chmod 600 /etc/archivmail/keyfile
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Konfiguration
|
||||||
|
|
||||||
|
### Vollständige Konfigurationsreferenz
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
server:
|
||||||
|
api_port: 8080 # HTTP-API Port
|
||||||
|
smtp_port: 2525 # SMTP-Eingang Port (Fallback wenn smtp.bind nicht gesetzt)
|
||||||
|
|
||||||
|
database:
|
||||||
|
host: 127.0.0.1
|
||||||
|
port: 5432
|
||||||
|
name: archivmail
|
||||||
|
user: archivmail
|
||||||
|
password: ""
|
||||||
|
sslmode: disable # disable | require | verify-full
|
||||||
|
|
||||||
|
storage:
|
||||||
|
store_path: /var/archivmail/store # Haupt-Mailspeicher (AES-256-GCM)
|
||||||
|
astore_path: /var/archivmail/astore # Anhang-Speicher
|
||||||
|
xapian_path: /var/archivmail/xapian # Xapian-Datenbankpfad (auch in index.path)
|
||||||
|
keyfile: /etc/archivmail/keyfile # 32-Byte AES-Schlüsseldatei
|
||||||
|
|
||||||
|
smtp:
|
||||||
|
enabled: true
|
||||||
|
bind: ":2525" # TCP-Adresse zum Lauschen
|
||||||
|
domain: "archivmail" # EHLO-Domainname
|
||||||
|
tls_cert: "" # Pfad zum TLS-Zertifikat (leer = kein TLS)
|
||||||
|
tls_key: "" # Pfad zum TLS-Schlüssel
|
||||||
|
max_size_mb: 50 # Maximale E-Mail-Größe in MB
|
||||||
|
allowed_ips: # Nur diese IPs dürfen zustellen
|
||||||
|
- 127.0.0.1
|
||||||
|
|
||||||
|
api:
|
||||||
|
bind: ":8080"
|
||||||
|
secret: "" # JWT-Signaturschlüssel (mind. 32 Zeichen, zufällig)
|
||||||
|
|
||||||
|
index:
|
||||||
|
path: /var/archivmail/xapian
|
||||||
|
backend: xapian # xapian (einziger unterstützter Backend)
|
||||||
|
batch_size: 100 # Dokumente pro Index-Batch
|
||||||
|
async_queue_size: 1000 # Größe der asynchronen Index-Queue
|
||||||
|
|
||||||
|
audit:
|
||||||
|
log_path: /var/log/archivmail/audit.log
|
||||||
|
retention_days: 0 # 0 = unbegrenzt
|
||||||
|
|
||||||
|
logging:
|
||||||
|
path: "" # leer = stdout
|
||||||
|
level: info # debug | info | warn | error
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Systemd-Dienste
|
||||||
|
|
||||||
|
Das System läuft mit zwei Systemd-Units:
|
||||||
|
|
||||||
|
| Dienst | Beschreibung | Port |
|
||||||
|
|--------|--------------|------|
|
||||||
|
| `archivmail` | Go-Backend (API + SMTP + IMAP-Scheduler) | 8080, 2525 |
|
||||||
|
| `archivmail-web` | Next.js-Frontend | 3000 |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Status prüfen
|
||||||
|
systemctl status archivmail archivmail-web
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
journalctl -u archivmail -f
|
||||||
|
journalctl -u archivmail-web -f
|
||||||
|
|
||||||
|
# Neustart
|
||||||
|
systemctl restart archivmail archivmail-web
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Funktionen im Detail
|
||||||
|
|
||||||
|
### Authentifizierung & Rollen
|
||||||
|
|
||||||
|
**Rollen:**
|
||||||
|
|
||||||
|
| Rolle | Rechte |
|
||||||
|
|-------|--------|
|
||||||
|
| `admin` | Vollzugriff: Benutzer verwalten, IMAP einrichten, Dienste steuern, alles suchen/exportieren |
|
||||||
|
| `auditor` | Audit-Log lesen, alle E-Mails suchen und exportieren |
|
||||||
|
| `user` | Nur eigene E-Mails suchen und lesen (Matching auf E-Mail-Adresse) |
|
||||||
|
|
||||||
|
**Sicherheit:**
|
||||||
|
- Passwörter mit bcrypt (Cost 12)
|
||||||
|
- Sessions als httpOnly SameSite=Strict Cookie (`archivmail_session`)
|
||||||
|
- JWT mit zufälligem JTI (kryptografische Entropie, 16 Byte)
|
||||||
|
- Token-Blacklist in PostgreSQL (Logout invalidiert Token sofort)
|
||||||
|
- Rate-Limiting: max. 5 Fehlversuche in 15 Minuten → HTTP 429
|
||||||
|
- Login-Versuche in `login_attempts`-Tabelle protokolliert
|
||||||
|
- Letzter Login wird gespeichert (`last_login_at`)
|
||||||
|
|
||||||
|
**Erstmalige Einrichtung:**
|
||||||
|
|
||||||
|
Beim ersten Start werden automatisch zwei Benutzer mit zufälligen Passwörtern angelegt:
|
||||||
|
|
||||||
|
```
|
||||||
|
╔══════════════════════════════════════════════════════════════╗
|
||||||
|
║ ARCHIVMAIL — ERSTMALIGE EINRICHTUNG ║
|
||||||
|
║ Initiale Zugangsdaten (NUR EINMAL ANGEZEIGT): ║
|
||||||
|
║ admin : <zufälliges Passwort> ║
|
||||||
|
║ auditor : <zufälliges Passwort> ║
|
||||||
|
║ Passwörter sofort nach dem ersten Login ändern! ║
|
||||||
|
╚══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### SMTP-Eingang (BCC-Journaling)
|
||||||
|
|
||||||
|
Der eingebettete SMTP-Daemon empfängt E-Mails von Postfix (oder anderem MTA) über BCC-Weiterleitung.
|
||||||
|
|
||||||
|
**Funktionsweise:**
|
||||||
|
```
|
||||||
|
Absender → Postfix → Empfänger
|
||||||
|
│
|
||||||
|
└── always_bcc → archivmail SMTP (Port 2525)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Speicherung + Indexierung
|
||||||
|
```
|
||||||
|
|
||||||
|
**Postfix-Konfiguration (`/etc/postfix/main.cf`):**
|
||||||
|
```
|
||||||
|
# Alle ausgehenden Mails archivieren:
|
||||||
|
always_bcc = archiv@archivmail-host
|
||||||
|
|
||||||
|
# Oder granular per Nutzer:
|
||||||
|
sender_bcc_maps = hash:/etc/postfix/sender_bcc
|
||||||
|
```
|
||||||
|
|
||||||
|
**Sicherheit:**
|
||||||
|
- Kein SMTP AUTH – Vertrauen ausschließlich über IP-Allowlist
|
||||||
|
- IP-Allowlist in `smtp.allowed_ips` konfigurieren
|
||||||
|
- 250 OK erst nach erfolgreicher Speicherung (kein Datenverlust)
|
||||||
|
- 250 OK auch bei Duplikaten (Postfix stellt nicht erneut zu)
|
||||||
|
- Maximale E-Mail-Größe konfigurierbar (`smtp.max_size_mb`, Standard: 50 MB)
|
||||||
|
- Optionale TLS/STARTTLS-Unterstützung (`smtp.tls_cert` / `smtp.tls_key`)
|
||||||
|
|
||||||
|
**API-Endpunkt:**
|
||||||
|
```
|
||||||
|
GET /api/admin/smtp/status
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### IMAP-Import & Auto-Sync
|
||||||
|
|
||||||
|
E-Mails von IMAP-Postfächern importieren – einmalig (Altbestände) oder automatisch als Hintergrundjob.
|
||||||
|
|
||||||
|
**Verbindung einrichten (Web-UI unter `/imap`):**
|
||||||
|
|
||||||
|
| Feld | Beschreibung |
|
||||||
|
|------|--------------|
|
||||||
|
| Name | Bezeichnung der Verbindung |
|
||||||
|
| Host / Port | IMAP-Server-Adresse |
|
||||||
|
| TLS | `ssl` (IMAPS), `starttls`, `none` |
|
||||||
|
| Benutzername / Passwort | IMAP-Zugangsdaten (Passwort AES-256-GCM verschlüsselt in DB) |
|
||||||
|
| Ausgeschlossene Ordner | Ordner die nicht importiert werden |
|
||||||
|
|
||||||
|
**Sync-Intervall:**
|
||||||
|
|
||||||
|
| Wert | Verhalten |
|
||||||
|
|------|-----------|
|
||||||
|
| 0 | Automatischer Sync deaktiviert |
|
||||||
|
| 5–1440 | Sync alle N Minuten (Hintergrund-Goroutine) |
|
||||||
|
|
||||||
|
**Inkrementeller Sync:**
|
||||||
|
- UID-basiert: Nur neue Nachrichten seit letztem Sync werden heruntergeladen
|
||||||
|
- Exponentielles Backoff bei Fehlern: 1s → 60s → 300s (3 Versuche)
|
||||||
|
- Duplikat-Erkennung via Content-Hash
|
||||||
|
|
||||||
|
**API-Endpunkte:**
|
||||||
|
```
|
||||||
|
GET /api/imap # Verbindungen auflisten
|
||||||
|
POST /api/imap # Neue Verbindung anlegen
|
||||||
|
DELETE /api/imap/{id} # Verbindung löschen
|
||||||
|
PATCH /api/imap/{id} # Sync-Intervall ändern
|
||||||
|
POST /api/imap/test # Verbindung testen + Ordner auflisten
|
||||||
|
POST /api/imap/{id}/import # Vollständigen Import starten
|
||||||
|
GET /api/imap/{id}/progress # Import-Fortschritt abfragen
|
||||||
|
POST /api/imap/{id}/sync # Manuellen inkrementellen Sync auslösen
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### EML/MBOX Web-Upload
|
||||||
|
|
||||||
|
E-Mails direkt über die Admin-Oberfläche hochladen (Tab „Import" unter `/admin`).
|
||||||
|
|
||||||
|
**Unterstützte Formate:**
|
||||||
|
- `.eml` – einzelne RFC-2822 E-Mail
|
||||||
|
- `.mbox` – MBOX-Datei mit beliebig vielen E-Mails
|
||||||
|
|
||||||
|
**Ablauf:**
|
||||||
|
1. Dateien per Drag-and-Drop oder Datei-Dialog hochladen
|
||||||
|
2. Backend startet Hintergrund-Import-Job, gibt sofort `job_id` zurück
|
||||||
|
3. Frontend pollt alle 1,5 Sekunden den Fortschritt
|
||||||
|
4. Abschlussbericht: Importiert / Übersprungen (Duplikate) / Fehler
|
||||||
|
|
||||||
|
**API-Endpunkte:**
|
||||||
|
```
|
||||||
|
POST /api/admin/upload # Multipart-Upload starten
|
||||||
|
GET /api/admin/upload/{jobID}/progress # Fortschritt abfragen
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Speicherung & Verschlüsselung
|
||||||
|
|
||||||
|
Alle E-Mails werden verschlüsselt auf dem Dateisystem gespeichert.
|
||||||
|
|
||||||
|
**Verschlüsselung:**
|
||||||
|
- Algorithmus: AES-256-GCM (authentifizierte Verschlüsselung)
|
||||||
|
- Schlüssel: 32-Byte-Datei (`keyfile` in Konfiguration)
|
||||||
|
- Nonce: 12 Byte, kryptografisch zufällig, pro E-Mail neu generiert
|
||||||
|
- Dateiformat: `[12-Byte Nonce][verschlüsselte Daten]`
|
||||||
|
|
||||||
|
**Datei-ID / Integrität:**
|
||||||
|
- Jede E-Mail erhält als ID den SHA-256-Hash des Klartexts
|
||||||
|
- Duplikat-Erkennung: gleicher Hash → gleiche Datei, Speicherung übersprungen
|
||||||
|
- ID dient gleichzeitig als Integritätsnachweis
|
||||||
|
|
||||||
|
**PostgreSQL-Metadaten (`emails`-Tabelle):**
|
||||||
|
|
||||||
|
| Spalte | Typ | Beschreibung |
|
||||||
|
|--------|-----|--------------|
|
||||||
|
| `id` | TEXT PK | SHA-256 Hash des E-Mail-Rohtexts |
|
||||||
|
| `from_addr` | TEXT | Absender |
|
||||||
|
| `to_addrs` | TEXT[] | Empfänger |
|
||||||
|
| `subject` | TEXT | Betreff |
|
||||||
|
| `message_id` | TEXT | RFC-2822 Message-ID |
|
||||||
|
| `mail_date` | TIMESTAMPTZ | Sendedatum |
|
||||||
|
| `size_bytes` | INTEGER | Größe in Bytes |
|
||||||
|
| `has_attachments` | BOOLEAN | Hat Anhänge |
|
||||||
|
| `stored_at` | TIMESTAMPTZ | Archivierungszeitpunkt |
|
||||||
|
| `indexed_at` | TIMESTAMPTZ | Indexierungszeitpunkt (NULL = ausstehend) |
|
||||||
|
| `verify_ok` | BOOLEAN | Letztes Integritätsprüfungsergebnis |
|
||||||
|
| `verified_at` | TIMESTAMPTZ | Zeitpunkt der letzten Prüfung |
|
||||||
|
|
||||||
|
**Async Index Worker:**
|
||||||
|
- Eingehende E-Mails werden sofort gespeichert
|
||||||
|
- Indexierung erfolgt asynchron über einen Go-Channel (Queue-Größe: 1000)
|
||||||
|
- Backfill beim Start: Nicht indexierte E-Mails werden nachindexiert
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Volltext-Suche
|
||||||
|
|
||||||
|
Suche über alle archivierten E-Mails per Xapian-Index unter `/search`.
|
||||||
|
|
||||||
|
**Suchfelder:**
|
||||||
|
- Freitext (durchsucht Betreff, Body, Absender, Empfänger, Anhangsnamen)
|
||||||
|
- Absender-Filter (`from`)
|
||||||
|
- Empfänger-Filter (`to`)
|
||||||
|
- Zeitraum (`date_from` / `date_to`, ISO 8601: `2024-01-01`)
|
||||||
|
- Nur Mails mit Anhängen (`has_attachment`)
|
||||||
|
- Sortierung: `date_desc` (Standard), `date_asc`, `relevance`
|
||||||
|
|
||||||
|
**Xapian-Suchsyntax:**
|
||||||
|
```
|
||||||
|
Rechnung AND 2024 # UND-Verknüpfung
|
||||||
|
"Angebot Projekt X" # Phrase
|
||||||
|
Rechn* # Wildcard
|
||||||
|
from:chef@firma.de # Feldpräfix
|
||||||
|
subject:Urlaubsantrag # Feldpräfix
|
||||||
|
```
|
||||||
|
|
||||||
|
**Paginierung:** 25 Ergebnisse pro Seite (konfigurierbar via `page_size`)
|
||||||
|
|
||||||
|
**Rollenfilter:**
|
||||||
|
- `user`: sieht nur E-Mails an/von der eigenen E-Mail-Adresse
|
||||||
|
- `admin` / `auditor`: sieht alle E-Mails
|
||||||
|
|
||||||
|
**API-Endpunkt:**
|
||||||
|
```
|
||||||
|
GET /api/search?q=...&from=...&to=...&date_from=...&date_to=...&has_attachment=true&sort=date_desc&page=1&page_size=25
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### E-Mail-Ansicht
|
||||||
|
|
||||||
|
Vollständige Darstellung einer archivierten E-Mail unter `/mail/[id]`.
|
||||||
|
|
||||||
|
**Funktionen:**
|
||||||
|
- HTML-Body original dargestellt in `<iframe sandbox="allow-same-origin">` – JavaScript aus der Mail wird blockiert
|
||||||
|
- Externe Inhalte (Bilder, Tracking-Pixel) standardmäßig blockiert, per Klick freischaltbar
|
||||||
|
- Fallback auf Plaintext wenn kein HTML vorhanden
|
||||||
|
- Anhänge einzeln herunterladbar
|
||||||
|
- Originale E-Mail-Header aufklappbar
|
||||||
|
- Download als `.eml` (Rohdatei)
|
||||||
|
- Export als `.pdf`
|
||||||
|
- Integrity-Badge: ✅ verifiziert OK / ⬜ noch nicht geprüft / ❌ Integritätsfehler
|
||||||
|
|
||||||
|
**API-Endpunkte:**
|
||||||
|
```
|
||||||
|
GET /api/mails/{id} # E-Mail-Details inkl. Body
|
||||||
|
GET /api/mails/{id}/attachments/{index} # Anhang herunterladen
|
||||||
|
GET /api/mails/{id}/raw # Rohe EML-Datei
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### E-Mail-Export
|
||||||
|
|
||||||
|
**Formate:**
|
||||||
|
|
||||||
|
| Format | Endpunkt | Beschreibung |
|
||||||
|
|--------|----------|--------------|
|
||||||
|
| PDF | `GET /api/export/pdf/{id}` | Einzelne Mail als PDF |
|
||||||
|
| EML | `GET /api/mails/{id}/raw` | Einzelne Mail als .eml |
|
||||||
|
| ZIP | `POST /api/export/zip` | Mehrere Mails + optional Anhänge |
|
||||||
|
|
||||||
|
**ZIP-Export:**
|
||||||
|
```json
|
||||||
|
POST /api/export/zip
|
||||||
|
{
|
||||||
|
"ids": ["sha256hash1", "sha256hash2"],
|
||||||
|
"attachments": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
ZIP enthält:
|
||||||
|
- Eine `.eml`-Datei pro E-Mail
|
||||||
|
- `manifest.csv` mit Metadaten (ID, Absender, Empfänger, Betreff, Datum, Größe)
|
||||||
|
- Anhänge wenn `attachments: true`
|
||||||
|
|
||||||
|
**Limits:**
|
||||||
|
- Max. 500 Mails pro ZIP-Export
|
||||||
|
- Jeder Export wird im Audit-Log erfasst
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Admin-Dashboard
|
||||||
|
|
||||||
|
Erreichbar unter `/admin` (nur Admins).
|
||||||
|
|
||||||
|
**Tabs:**
|
||||||
|
|
||||||
|
| Tab | Inhalt |
|
||||||
|
|-----|--------|
|
||||||
|
| **Dashboard** | CPU-Last (1/5/15 min), RAM, Festplatten, API-Status, SMTP-Status, Archiv-Statistiken, erste/letzte archivierte Mail |
|
||||||
|
| **Dienste** | Systemd-Dienste starten / stoppen / neustarten, externe Verbindungen sperren/freigeben |
|
||||||
|
| **Benutzer** | Benutzer anlegen, Rollen ändern, Passwort zurücksetzen, sperren/freischalten, löschen |
|
||||||
|
| **Audit-Log** | Alle Audit-Ereignisse mit Paginierung (25/Seite) |
|
||||||
|
| **Import** | EML/MBOX-Dateien hochladen, Fortschritt und Abschlussbericht |
|
||||||
|
| **Module** | Übersicht aller Features mit Deployment-Status |
|
||||||
|
|
||||||
|
**Dashboard API-Endpunkte:**
|
||||||
|
```
|
||||||
|
GET /api/admin/system/stats # CPU, RAM, Disks, erste/letzte Mail
|
||||||
|
GET /api/admin/smtp/status # SMTP-Daemon Status und Statistiken
|
||||||
|
GET /api/admin/storage/stats # Anzahl archivierter Mails und Gesamtgröße
|
||||||
|
GET /api/admin/services # Systemd-Dienste
|
||||||
|
POST /api/admin/services/{name}/action # start | stop | restart | enable | disable
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Audit-Log
|
||||||
|
|
||||||
|
Alle sicherheitsrelevanten Ereignisse werden lückenlos protokolliert.
|
||||||
|
|
||||||
|
**Erfasste Ereignisse:**
|
||||||
|
|
||||||
|
| Ereignis | Beschreibung |
|
||||||
|
|----------|--------------|
|
||||||
|
| `login_ok` | Erfolgreicher Login (Benutzer, IP) |
|
||||||
|
| `login_fail` | Fehlgeschlagener Login (Benutzer, IP) |
|
||||||
|
| `logout` | Logout |
|
||||||
|
| `search` | Suchanfrage (Suchbegriff, Benutzer) |
|
||||||
|
| `export_pdf` | PDF-Export einer E-Mail |
|
||||||
|
| `export_zip` | ZIP-Export (Anzahl Mails) |
|
||||||
|
| `user_create` | Neuer Benutzer angelegt |
|
||||||
|
| `user_update` | Benutzerdaten geändert |
|
||||||
|
| `user_delete` | Benutzer gelöscht |
|
||||||
|
| `imap_import` | IMAP-Import gestartet/abgeschlossen |
|
||||||
|
|
||||||
|
**Speicherung:** Parallel in PostgreSQL-Tabelle und Flat-File (`audit.log_path`)
|
||||||
|
|
||||||
|
**API:**
|
||||||
|
```
|
||||||
|
GET /api/audit?page=1&page_size=25&username=admin&event_type=login_ok
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Integritätsprüfung
|
||||||
|
|
||||||
|
Jede archivierte E-Mail wird regelmäßig auf Unverändertheit geprüft.
|
||||||
|
|
||||||
|
**Funktionsweise:**
|
||||||
|
1. Beim Speichern: SHA-256-Hash des Klartexts = Datei-ID
|
||||||
|
2. Beim Prüfen: Datei laden → entschlüsseln → SHA-256 berechnen → mit ID vergleichen
|
||||||
|
3. Bei Abweichung: Warnung im Log + `verify_ok = false` in DB
|
||||||
|
|
||||||
|
**Hintergrundprüfung:**
|
||||||
|
- Läuft beim Start einmal sofort durch
|
||||||
|
- Danach alle 5 Minuten wiederholt
|
||||||
|
- Ergebnisse in `emails`-Tabelle: `verify_ok`, `verified_at`
|
||||||
|
|
||||||
|
**Ergebnis in der E-Mail-Ansicht:**
|
||||||
|
```
|
||||||
|
✅ Integrität OK (geprüft: 2024-03-15 14:22:00)
|
||||||
|
❌ INTEGRITÄTSFEHLER – E-Mail wurde möglicherweise verändert
|
||||||
|
⬜ Noch nicht geprüft
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### CLI Import & Export
|
||||||
|
|
||||||
|
Das `archivmail`-Binary kann auch ohne laufenden Daemon als CLI-Tool eingesetzt werden.
|
||||||
|
|
||||||
|
**Import:**
|
||||||
|
```bash
|
||||||
|
# Einzelne EML-Datei
|
||||||
|
archivmail import --file /pfad/zur/datei.eml
|
||||||
|
|
||||||
|
# MBOX-Datei
|
||||||
|
archivmail import --file /pfad/zur/datei.mbox
|
||||||
|
|
||||||
|
# Verzeichnis (alle EML/MBOX-Dateien)
|
||||||
|
archivmail import --dir /pfad/zum/verzeichnis
|
||||||
|
|
||||||
|
# Rekursiv
|
||||||
|
archivmail import --dir /pfad/ --recursive
|
||||||
|
|
||||||
|
# Simulation (kein Speichern)
|
||||||
|
archivmail import --file datei.eml --dry-run
|
||||||
|
|
||||||
|
# JSON-Ausgabe für Skripting
|
||||||
|
archivmail import --dir /pfad/ --json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Export:**
|
||||||
|
```bash
|
||||||
|
# Alle Mails als EML in Verzeichnis
|
||||||
|
archivmail export --out /export/
|
||||||
|
|
||||||
|
# Als MBOX
|
||||||
|
archivmail export --out /export/archiv.mbox --format mbox
|
||||||
|
|
||||||
|
# Mit Filtern
|
||||||
|
archivmail export --out /export/ \
|
||||||
|
--from absender@firma.de \
|
||||||
|
--date-from 2024-01-01 \
|
||||||
|
--date-to 2024-12-31 \
|
||||||
|
--query "Rechnung"
|
||||||
|
|
||||||
|
# Vorhandene Dateien überschreiben
|
||||||
|
archivmail export --out /export/ --force
|
||||||
|
|
||||||
|
# JSON-Ausgabe
|
||||||
|
archivmail export --out /export/ --json
|
||||||
|
```
|
||||||
|
|
||||||
|
**Alle Befehle:**
|
||||||
|
```
|
||||||
|
archivmail serve Daemon starten (Standard)
|
||||||
|
archivmail import EML/MBOX importieren
|
||||||
|
archivmail import-piler Aus mailpiler migrieren
|
||||||
|
archivmail export Exportieren
|
||||||
|
archivmail version Version anzeigen
|
||||||
|
archivmail help Hilfe
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Mailpiler Migration
|
||||||
|
|
||||||
|
Vollständige Migration eines bestehenden mailpiler-Archivs nach archivmail.
|
||||||
|
|
||||||
|
**Methode 1: `pilerexport` (empfohlen)**
|
||||||
|
|
||||||
|
Erfordert das mailpiler-Tool `pilerexport` auf dem Server. Verarbeitet Entschlüsselung und Dekomprimierung automatisch.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Alle Mails migrieren
|
||||||
|
archivmail import-piler --config /etc/archivmail/config.yml
|
||||||
|
|
||||||
|
# Mit Datumsfilter
|
||||||
|
archivmail import-piler --date-from 2020-01-01 --date-to 2024-12-31
|
||||||
|
|
||||||
|
# Benutzerdefinierter Export-Ordner (bleibt erhalten)
|
||||||
|
archivmail import-piler --export-dir /mnt/migration/export
|
||||||
|
```
|
||||||
|
|
||||||
|
**Methode 2: Direkt (ohne laufendes mailpiler)**
|
||||||
|
|
||||||
|
Liest `.m`-Dateien direkt aus dem mailpiler Store-Verzeichnis und verarbeitet sie ohne dass mailpiler installiert sein muss.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
archivmail import-piler \
|
||||||
|
--method direct \
|
||||||
|
--store-dir /var/piler/store \
|
||||||
|
--key-file /var/piler/store/piler.key
|
||||||
|
```
|
||||||
|
|
||||||
|
Unterstützt:
|
||||||
|
- AES-256-CBC Entschlüsselung (IV in ersten 16 Bytes der Datei)
|
||||||
|
- zlib Dekomprimierung
|
||||||
|
- Mehrfache Fallback-Versuche für verschiedene mailpiler-Versionen
|
||||||
|
- Unkryptierte/unkomprimierte Dateien als letzter Fallback
|
||||||
|
|
||||||
|
**Auto-Modus (Standard):**
|
||||||
|
|
||||||
|
`--method auto` versucht zuerst `pilerexport`, fällt auf `direct` zurück:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
archivmail import-piler --config /etc/archivmail/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
**Alle Optionen:**
|
||||||
|
```
|
||||||
|
--config archivmail Konfigurationsdatei
|
||||||
|
--method auto | pilerexport | direct (Standard: auto)
|
||||||
|
--pilerexport Pfad zum pilerexport Binary (auto-erkennung)
|
||||||
|
--export-dir Ausgabeverzeichnis für pilerexport
|
||||||
|
--store-dir mailpiler Store-Verzeichnis (Standard: /var/piler/store)
|
||||||
|
--key-file mailpiler AES-Schlüsseldatei (Standard: /var/piler/store/piler.key)
|
||||||
|
--date-from Export ab Datum YYYY-MM-DD (pilerexport-Methode)
|
||||||
|
--date-to Export bis Datum YYYY-MM-DD (pilerexport-Methode)
|
||||||
|
--dry-run Simulation ohne Speichern
|
||||||
|
--json JSON-Ausgabe
|
||||||
|
```
|
||||||
|
|
||||||
|
**Typischer Migrationsablauf:**
|
||||||
|
```bash
|
||||||
|
# 1. Probe-Lauf
|
||||||
|
archivmail import-piler --dry-run --json
|
||||||
|
|
||||||
|
# 2. Migration mit Protokoll
|
||||||
|
archivmail import-piler 2>&1 | tee /var/log/archivmail/migration.log
|
||||||
|
|
||||||
|
# 3. Ergebnis prüfen
|
||||||
|
tail -20 /var/log/archivmail/migration.log
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## REST API
|
||||||
|
|
||||||
|
Alle Endpunkte erfordern eine gültige Session (Cookie `archivmail_session` oder `Authorization: Bearer <token>`).
|
||||||
|
|
||||||
|
### Authentifizierung
|
||||||
|
|
||||||
|
| Methode | Endpunkt | Beschreibung |
|
||||||
|
|---------|----------|--------------|
|
||||||
|
| POST | `/api/auth/login` | Login: `{"username":"...","password":"..."}` |
|
||||||
|
| GET | `/api/auth/me` | Eingeloggter Benutzer |
|
||||||
|
| POST | `/api/auth/logout` | Session invalidieren |
|
||||||
|
|
||||||
|
### Suche & Mails
|
||||||
|
|
||||||
|
| Methode | Endpunkt | Beschreibung |
|
||||||
|
|---------|----------|--------------|
|
||||||
|
| GET | `/api/search` | Volltext-Suche |
|
||||||
|
| GET | `/api/mails/{id}` | E-Mail-Details |
|
||||||
|
| GET | `/api/mails/{id}/raw` | Rohe EML |
|
||||||
|
| GET | `/api/mails/{id}/attachments/{n}` | Anhang herunterladen |
|
||||||
|
|
||||||
|
### Export
|
||||||
|
|
||||||
|
| Methode | Endpunkt | Beschreibung |
|
||||||
|
|---------|----------|--------------|
|
||||||
|
| GET | `/api/export/pdf/{id}` | PDF-Export |
|
||||||
|
| POST | `/api/export/zip` | ZIP-Export |
|
||||||
|
|
||||||
|
### Admin
|
||||||
|
|
||||||
|
| Methode | Endpunkt | Beschreibung |
|
||||||
|
|---------|----------|--------------|
|
||||||
|
| GET | `/api/users` | Benutzerliste |
|
||||||
|
| POST | `/api/users` | Benutzer anlegen |
|
||||||
|
| PATCH | `/api/users/{id}` | Benutzer ändern |
|
||||||
|
| DELETE | `/api/users/{id}` | Benutzer löschen |
|
||||||
|
| GET | `/api/audit` | Audit-Log |
|
||||||
|
| GET | `/api/admin/smtp/status` | SMTP-Status |
|
||||||
|
| GET | `/api/admin/storage/stats` | Speicher-Statistiken |
|
||||||
|
| GET | `/api/admin/system/stats` | System-Ressourcen |
|
||||||
|
| GET | `/api/admin/services` | Dienste-Status |
|
||||||
|
| POST | `/api/admin/services/{name}/action` | Dienst steuern |
|
||||||
|
| POST | `/api/admin/upload` | EML/MBOX hochladen |
|
||||||
|
| GET | `/api/admin/upload/{jobID}/progress` | Upload-Fortschritt |
|
||||||
|
|
||||||
|
### IMAP
|
||||||
|
|
||||||
|
| Methode | Endpunkt | Beschreibung |
|
||||||
|
|---------|----------|--------------|
|
||||||
|
| GET | `/api/imap` | Verbindungen auflisten |
|
||||||
|
| POST | `/api/imap` | Verbindung anlegen |
|
||||||
|
| DELETE | `/api/imap/{id}` | Verbindung löschen |
|
||||||
|
| PATCH | `/api/imap/{id}` | Sync-Intervall setzen |
|
||||||
|
| POST | `/api/imap/test` | Verbindung testen |
|
||||||
|
| POST | `/api/imap/{id}/import` | Vollständigen Import starten |
|
||||||
|
| GET | `/api/imap/{id}/progress` | Import-Fortschritt |
|
||||||
|
| POST | `/api/imap/{id}/sync` | Manuellen Sync auslösen |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## In Entwicklung
|
||||||
|
|
||||||
|
| Funktion | Beschreibung |
|
||||||
|
|----------|--------------|
|
||||||
|
| **Ordner- & Label-Verwaltung** | E-Mails in Ordner ablegen und mit Labels versehen |
|
||||||
|
| **POP3-Import** | E-Mails von POP3-Servern importieren |
|
||||||
|
| **REST API (extern)** | Vollständige API für CRM-Anbindung und externe Systeme |
|
||||||
|
| **LDAP / Active Directory** | Benutzer-Authentifizierung über LDAP/AD statt lokaler Accounts |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Update
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Auf dem Server als root:
|
||||||
|
bash /opt/archivmail/update.sh
|
||||||
|
|
||||||
|
# Oder direkt von Gitea:
|
||||||
|
curl -fsSL https://gitea.perlbach24.de/scripte/archivmail/raw/branch/main/update.sh | bash
|
||||||
|
```
|
||||||
|
|
||||||
|
Das Update-Skript führt automatisch durch:
|
||||||
|
1. Quellcode aktualisieren (git pull)
|
||||||
|
2. Backend neu bauen (CGO + Xapian)
|
||||||
|
3. Frontend neu bauen (Next.js standalone)
|
||||||
|
4. Dienste stoppen
|
||||||
|
5. Binaries und Frontend einspielen
|
||||||
|
6. Datenbankmigrationen durchführen
|
||||||
|
7. Dienste starten und Status prüfen
|
||||||
|
|||||||
@@ -264,6 +264,7 @@ func printHelp() {
|
|||||||
Commands:
|
Commands:
|
||||||
serve Daemon starten (Standard wenn kein Befehl angegeben)
|
serve Daemon starten (Standard wenn kein Befehl angegeben)
|
||||||
import E-Mails importieren (EML, MBOX, Verzeichnis)
|
import E-Mails importieren (EML, MBOX, Verzeichnis)
|
||||||
|
import-piler Aus mailpiler migrieren (pilerexport oder direkte Store-Methode)
|
||||||
export E-Mails exportieren (EML, MBOX)
|
export E-Mails exportieren (EML, MBOX)
|
||||||
version Version anzeigen
|
version Version anzeigen
|
||||||
help Diese Hilfe anzeigen
|
help Diese Hilfe anzeigen
|
||||||
@@ -276,6 +277,18 @@ archivmail import [flags]
|
|||||||
--dry-run Simulation ohne Speichern
|
--dry-run Simulation ohne Speichern
|
||||||
--json Maschinenlesbare JSON-Ausgabe
|
--json Maschinenlesbare JSON-Ausgabe
|
||||||
|
|
||||||
|
archivmail import-piler [flags]
|
||||||
|
--config Pfad zur Konfigurationsdatei (Standard: /etc/archivmail/config.yml)
|
||||||
|
--method auto | pilerexport | direct (Standard: auto)
|
||||||
|
--pilerexport Pfad zum pilerexport Binary (auto-erkennung)
|
||||||
|
--export-dir Ausgabeverzeichnis für pilerexport (temp wenn leer)
|
||||||
|
--store-dir mailpiler Store-Verzeichnis (Standard: /var/piler/store)
|
||||||
|
--key-file mailpiler AES-Schlüsseldatei (Standard: /var/piler/store/piler.key)
|
||||||
|
--date-from Export ab Datum YYYY-MM-DD (pilerexport-Methode)
|
||||||
|
--date-to Export bis Datum YYYY-MM-DD (pilerexport-Methode)
|
||||||
|
--dry-run Simulation ohne Speichern
|
||||||
|
--json Maschinenlesbare JSON-Ausgabe
|
||||||
|
|
||||||
archivmail export [flags]
|
archivmail export [flags]
|
||||||
--config Pfad zur Konfigurationsdatei (Standard: /etc/archivmail/config.yml)
|
--config Pfad zur Konfigurationsdatei (Standard: /etc/archivmail/config.yml)
|
||||||
--out Zielverzeichnis oder Zieldatei (Pflicht)
|
--out Zielverzeichnis oder Zieldatei (Pflicht)
|
||||||
|
|||||||
@@ -0,0 +1,426 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"compress/zlib"
|
||||||
|
"crypto/aes"
|
||||||
|
"crypto/cipher"
|
||||||
|
"encoding/json"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/archivmail/config"
|
||||||
|
"github.com/archivmail/internal/index"
|
||||||
|
"github.com/archivmail/internal/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
type pilerImportResult struct {
|
||||||
|
Status string `json:"status"`
|
||||||
|
Method string `json:"method"`
|
||||||
|
Imported int `json:"imported"`
|
||||||
|
Skipped int `json:"skipped"`
|
||||||
|
Errors int `json:"errors"`
|
||||||
|
DurationSec float64 `json:"duration_sec"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func runImportPiler(args []string) {
|
||||||
|
fs := flag.NewFlagSet("import-piler", flag.ExitOnError)
|
||||||
|
configPath := fs.String("config", "/etc/archivmail/config.yml", "archivmail config path")
|
||||||
|
method := fs.String("method", "auto", "import method: auto | pilerexport | direct")
|
||||||
|
pilerexpBin := fs.String("pilerexport", "", "path to pilerexport binary (auto-detect)")
|
||||||
|
exportDir := fs.String("export-dir", "", "output dir for pilerexport (temp dir if empty)")
|
||||||
|
storeDir := fs.String("store-dir", "/var/piler/store", "mailpiler store directory (direct method)")
|
||||||
|
keyFile := fs.String("key-file", "/var/piler/store/piler.key", "mailpiler AES key file (direct method)")
|
||||||
|
dateFrom := fs.String("date-from", "", "export from date YYYY-MM-DD (pilerexport method)")
|
||||||
|
dateTo := fs.String("date-to", "", "export to date YYYY-MM-DD (pilerexport method)")
|
||||||
|
dryRun := fs.Bool("dry-run", false, "simulate without saving")
|
||||||
|
jsonOut := fs.Bool("json", false, "machine-readable JSON output")
|
||||||
|
fs.Usage = func() {
|
||||||
|
fmt.Fprintln(os.Stderr, "Usage: archivmail import-piler [flags]")
|
||||||
|
fmt.Fprintln(os.Stderr, "")
|
||||||
|
fmt.Fprintln(os.Stderr, "Flags:")
|
||||||
|
fs.PrintDefaults()
|
||||||
|
fmt.Fprintln(os.Stderr, "")
|
||||||
|
fmt.Fprintln(os.Stderr, "Methods:")
|
||||||
|
fmt.Fprintln(os.Stderr, " pilerexport Calls pilerexport, imports resulting EML files (recommended)")
|
||||||
|
fmt.Fprintln(os.Stderr, " direct Reads .m files from mailpiler store, decrypts+decompresses")
|
||||||
|
fmt.Fprintln(os.Stderr, " auto Tries pilerexport first, falls back to direct")
|
||||||
|
}
|
||||||
|
fs.Parse(args)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
cfg, err := config.Load(*configPath)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: load config: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
storeCfg := storage.Config{
|
||||||
|
Dir: cfg.Storage.StorePath,
|
||||||
|
Keyfile: cfg.Storage.Keyfile,
|
||||||
|
DSN: cfg.Database.DSN(),
|
||||||
|
}
|
||||||
|
mailStore, err := storage.New(storeCfg)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: storage init: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer mailStore.Close()
|
||||||
|
|
||||||
|
batchSize := cfg.Index.BatchSize
|
||||||
|
if batchSize <= 0 {
|
||||||
|
batchSize = 100
|
||||||
|
}
|
||||||
|
backend := cfg.Index.Backend
|
||||||
|
if backend == "" {
|
||||||
|
backend = "xapian"
|
||||||
|
}
|
||||||
|
idx, err := index.New(cfg.Index.Path, batchSize, backend)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: index init: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
defer idx.Close()
|
||||||
|
|
||||||
|
// Resolve method
|
||||||
|
resolvedMethod := *method
|
||||||
|
if resolvedMethod == "auto" {
|
||||||
|
if bin := resolvePilerexport(*pilerexpBin); bin != "" {
|
||||||
|
resolvedMethod = "pilerexport"
|
||||||
|
} else {
|
||||||
|
resolvedMethod = "direct"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !*jsonOut {
|
||||||
|
fmt.Printf("Mailpiler → archivmail Migration\n")
|
||||||
|
fmt.Printf("Methode: %s\n\n", resolvedMethod)
|
||||||
|
}
|
||||||
|
|
||||||
|
var imported, skipped, errors int
|
||||||
|
|
||||||
|
switch resolvedMethod {
|
||||||
|
case "pilerexport":
|
||||||
|
imported, skipped, errors = runPilerexportMethod(
|
||||||
|
mailStore, idx,
|
||||||
|
*pilerexpBin, *exportDir, *dateFrom, *dateTo,
|
||||||
|
*dryRun, *jsonOut,
|
||||||
|
)
|
||||||
|
case "direct":
|
||||||
|
imported, skipped, errors = runDirectMethod(
|
||||||
|
mailStore, idx,
|
||||||
|
*storeDir, *keyFile,
|
||||||
|
*dryRun, *jsonOut,
|
||||||
|
)
|
||||||
|
default:
|
||||||
|
fmt.Fprintf(os.Stderr, "error: unknown method %q (use: auto, pilerexport, direct)\n", resolvedMethod)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
result := pilerImportResult{
|
||||||
|
Status: "done",
|
||||||
|
Method: resolvedMethod,
|
||||||
|
Imported: imported,
|
||||||
|
Skipped: skipped,
|
||||||
|
Errors: errors,
|
||||||
|
DurationSec: time.Since(start).Seconds(),
|
||||||
|
}
|
||||||
|
|
||||||
|
if *jsonOut {
|
||||||
|
enc := json.NewEncoder(os.Stdout)
|
||||||
|
enc.SetIndent("", " ")
|
||||||
|
enc.Encode(result)
|
||||||
|
} else {
|
||||||
|
fmt.Printf("\n╔══════════════════════════════════════╗\n")
|
||||||
|
fmt.Printf("║ Migration abgeschlossen ║\n")
|
||||||
|
fmt.Printf("╠══════════════════════════════════════╣\n")
|
||||||
|
fmt.Printf("║ Importiert: %-22d║\n", imported)
|
||||||
|
fmt.Printf("║ Übersprungen: %-22d║\n", skipped)
|
||||||
|
fmt.Printf("║ Fehler: %-22d║\n", errors)
|
||||||
|
fmt.Printf("║ Dauer: %-19.1fs║\n", result.DurationSec)
|
||||||
|
if *dryRun {
|
||||||
|
fmt.Printf("║ [dry-run] Keine Daten gespeichert ║\n")
|
||||||
|
}
|
||||||
|
fmt.Printf("╚══════════════════════════════════════╝\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
if errors > 0 {
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── pilerexport method ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
func runPilerexportMethod(mailStore *storage.Store, idx index.Indexer, binPath, exportDir, dateFrom, dateTo string, dryRun, jsonOut bool) (imported, skipped, errors int) {
|
||||||
|
bin := resolvePilerexport(binPath)
|
||||||
|
if bin == "" {
|
||||||
|
fmt.Fprintln(os.Stderr, "error: pilerexport binary not found")
|
||||||
|
fmt.Fprintln(os.Stderr, " Install mailpiler tools or use --method direct")
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create temp dir if needed
|
||||||
|
cleanupDir := false
|
||||||
|
if exportDir == "" {
|
||||||
|
tmp, err := os.MkdirTemp("", "archivmail-piler-*")
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: create temp dir: %v\n", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
exportDir = tmp
|
||||||
|
cleanupDir = true
|
||||||
|
} else {
|
||||||
|
if err := os.MkdirAll(exportDir, 0750); err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: create export dir %s: %v\n", exportDir, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !jsonOut {
|
||||||
|
fmt.Printf("Exportiere aus mailpiler nach %s ...\n", exportDir)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build pilerexport command
|
||||||
|
cmdArgs := []string{"-D", exportDir}
|
||||||
|
if dateFrom != "" {
|
||||||
|
cmdArgs = append(cmdArgs, "-f", dateFrom)
|
||||||
|
}
|
||||||
|
if dateTo != "" {
|
||||||
|
cmdArgs = append(cmdArgs, "-t", dateTo)
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd := exec.Command(bin, cmdArgs...)
|
||||||
|
cmd.Stdout = os.Stdout
|
||||||
|
cmd.Stderr = os.Stderr
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: pilerexport failed: %v\n", err)
|
||||||
|
fmt.Fprintln(os.Stderr, " Ensure pilerexport has access to mailpiler database and store.")
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !jsonOut {
|
||||||
|
fmt.Println("pilerexport abgeschlossen. Importiere EML-Dateien...")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Import all EML files from the export directory
|
||||||
|
total := 0
|
||||||
|
filepath.WalkDir(exportDir, func(path string, d os.DirEntry, werr error) error {
|
||||||
|
if werr != nil || d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
lower := strings.ToLower(d.Name())
|
||||||
|
if !strings.HasSuffix(lower, ".eml") && !strings.HasSuffix(lower, ".m") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
errors++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
total++
|
||||||
|
result := importMessage(mailStore, idx, raw, dryRun)
|
||||||
|
switch result {
|
||||||
|
case "imported":
|
||||||
|
imported++
|
||||||
|
case "skipped":
|
||||||
|
skipped++
|
||||||
|
case "error":
|
||||||
|
errors++
|
||||||
|
}
|
||||||
|
if !jsonOut && total%100 == 0 {
|
||||||
|
fmt.Printf(" Fortschritt: %d (importiert: %d, übersprungen: %d, fehler: %d)\n",
|
||||||
|
total, imported, skipped, errors)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
if cleanupDir {
|
||||||
|
os.RemoveAll(exportDir)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── direct method ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
func runDirectMethod(mailStore *storage.Store, idx index.Indexer, storeDir, keyFilePath string, dryRun, jsonOut bool) (imported, skipped, errors int) {
|
||||||
|
if _, err := os.Stat(storeDir); os.IsNotExist(err) {
|
||||||
|
fmt.Fprintf(os.Stderr, "error: store dir not found: %s\n", storeDir)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try to read AES key
|
||||||
|
var aesKey []byte
|
||||||
|
if data, err := os.ReadFile(keyFilePath); err == nil && len(data) >= 32 {
|
||||||
|
aesKey = data[:32]
|
||||||
|
if !jsonOut {
|
||||||
|
fmt.Printf("AES-Schlüssel geladen: %s\n", keyFilePath)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if !jsonOut {
|
||||||
|
fmt.Printf("Hinweis: Kein AES-Schlüssel geladen (%s) – versuche unkomprimiert/unkryptiert\n", keyFilePath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !jsonOut {
|
||||||
|
fmt.Printf("Lese .m Dateien aus %s ...\n", storeDir)
|
||||||
|
}
|
||||||
|
|
||||||
|
total := 0
|
||||||
|
filepath.WalkDir(storeDir, func(path string, d os.DirEntry, werr error) error {
|
||||||
|
if werr != nil || d.IsDir() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(d.Name(), ".m") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
errors++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode the piler file format
|
||||||
|
emailRaw, err := decodePilerFile(raw, aesKey)
|
||||||
|
if err != nil || len(emailRaw) == 0 {
|
||||||
|
errors++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
total++
|
||||||
|
result := importMessage(mailStore, idx, emailRaw, dryRun)
|
||||||
|
switch result {
|
||||||
|
case "imported":
|
||||||
|
imported++
|
||||||
|
case "skipped":
|
||||||
|
skipped++
|
||||||
|
case "error":
|
||||||
|
errors++
|
||||||
|
}
|
||||||
|
if !jsonOut && total%100 == 0 {
|
||||||
|
fmt.Printf(" Fortschritt: %d (importiert: %d, übersprungen: %d, fehler: %d)\n",
|
||||||
|
total, imported, skipped, errors)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// decodePilerFile attempts to decode a mailpiler .m file.
|
||||||
|
// mailpiler stores files as: [16-byte IV][AES-256-CBC encrypted zlib data]
|
||||||
|
// If no key is provided, zlib decompression alone is attempted.
|
||||||
|
func decodePilerFile(data []byte, aesKey []byte) ([]byte, error) {
|
||||||
|
// If key is available, try AES-256-CBC decrypt first, then zlib decompress
|
||||||
|
if len(aesKey) == 32 && len(data) > 16 {
|
||||||
|
decrypted, err := aes256CBCDecrypt(data, aesKey)
|
||||||
|
if err == nil {
|
||||||
|
if decompressed, err := zlibDecompress(decrypted); err == nil {
|
||||||
|
return decompressed, nil
|
||||||
|
}
|
||||||
|
// Maybe the decrypted data is already a raw email (no zlib)
|
||||||
|
if looksLikeEmail(decrypted) {
|
||||||
|
return decrypted, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try zlib decompression at various offsets (no encryption, or wrong key)
|
||||||
|
for _, skip := range []int{0, 4, 8, 12, 16} {
|
||||||
|
if skip >= len(data) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if out, err := zlibDecompress(data[skip:]); err == nil && looksLikeEmail(out) {
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try raw (uncompressed, unencrypted)
|
||||||
|
if looksLikeEmail(data) {
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, fmt.Errorf("could not decode piler file")
|
||||||
|
}
|
||||||
|
|
||||||
|
func aes256CBCDecrypt(data, key []byte) ([]byte, error) {
|
||||||
|
if len(data) < aes.BlockSize {
|
||||||
|
return nil, fmt.Errorf("data too short")
|
||||||
|
}
|
||||||
|
block, err := aes.NewCipher(key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
iv := data[:aes.BlockSize]
|
||||||
|
ciphertext := data[aes.BlockSize:]
|
||||||
|
if len(ciphertext)%aes.BlockSize != 0 {
|
||||||
|
return nil, fmt.Errorf("ciphertext not aligned to block size")
|
||||||
|
}
|
||||||
|
plaintext := make([]byte, len(ciphertext))
|
||||||
|
cipher.NewCBCDecrypter(block, iv).CryptBlocks(plaintext, ciphertext)
|
||||||
|
// Remove PKCS7 padding
|
||||||
|
if len(plaintext) == 0 {
|
||||||
|
return nil, fmt.Errorf("empty plaintext")
|
||||||
|
}
|
||||||
|
pad := int(plaintext[len(plaintext)-1])
|
||||||
|
if pad == 0 || pad > aes.BlockSize {
|
||||||
|
return nil, fmt.Errorf("invalid PKCS7 padding")
|
||||||
|
}
|
||||||
|
return plaintext[:len(plaintext)-pad], nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func zlibDecompress(data []byte) ([]byte, error) {
|
||||||
|
r, err := zlib.NewReader(bytes.NewReader(data))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer r.Close()
|
||||||
|
out, err := io.ReadAll(r)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// looksLikeEmail returns true if the data resembles an RFC 2822 email.
|
||||||
|
func looksLikeEmail(data []byte) bool {
|
||||||
|
if len(data) < 10 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
header := strings.ToLower(string(data[:min(512, len(data))]))
|
||||||
|
return strings.Contains(header, "from:") ||
|
||||||
|
strings.Contains(header, "date:") ||
|
||||||
|
strings.Contains(header, "message-id:") ||
|
||||||
|
strings.Contains(header, "subject:")
|
||||||
|
}
|
||||||
|
|
||||||
|
func min(a, b int) int {
|
||||||
|
if a < b {
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolvePilerexport returns the path to the pilerexport binary, or "" if not found.
|
||||||
|
func resolvePilerexport(hint string) string {
|
||||||
|
candidates := []string{hint, "pilerexport", "/usr/sbin/pilerexport", "/usr/local/sbin/pilerexport", "/opt/piler/bin/pilerexport"}
|
||||||
|
for _, c := range candidates {
|
||||||
|
if c == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if path, err := exec.LookPath(c); err == nil {
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(c); err == nil {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
@@ -32,6 +32,9 @@ func main() {
|
|||||||
case "import":
|
case "import":
|
||||||
runImport(os.Args[2:])
|
runImport(os.Args[2:])
|
||||||
return
|
return
|
||||||
|
case "import-piler":
|
||||||
|
runImportPiler(os.Args[2:])
|
||||||
|
return
|
||||||
case "export":
|
case "export":
|
||||||
runExport(os.Args[2:])
|
runExport(os.Args[2:])
|
||||||
return
|
return
|
||||||
|
|||||||
+8
-7
@@ -14,15 +14,15 @@
|
|||||||
|----|---------|--------|------|---------|
|
|----|---------|--------|------|---------|
|
||||||
| PROJ-1 | Nutzer-Authentifizierung & Rollen (User/Admin) | Deployed | [PROJ-1](PROJ-1-authentifizierung-und-rollen.md) | 2026-03-12 |
|
| PROJ-1 | Nutzer-Authentifizierung & Rollen (User/Admin) | Deployed | [PROJ-1](PROJ-1-authentifizierung-und-rollen.md) | 2026-03-12 |
|
||||||
| PROJ-2 | E-Mail-Import: EML/MBOX Upload | In Progress | [PROJ-2](PROJ-2-import-eml-mbox.md) | 2026-03-12 |
|
| PROJ-2 | E-Mail-Import: EML/MBOX Upload | In Progress | [PROJ-2](PROJ-2-import-eml-mbox.md) | 2026-03-12 |
|
||||||
| PROJ-3 | E-Mail-Import: IMAP-Verbindung | In Progress | [PROJ-3](PROJ-3-import-imap.md) | 2026-03-12 |
|
| PROJ-3 | E-Mail-Import: IMAP-Verbindung | Deployed | [PROJ-3](PROJ-3-import-imap.md) | 2026-03-12 |
|
||||||
| PROJ-4 | E-Mail-Import: SMTP-Eingang via BCC (primär) | In Progress | [PROJ-4](PROJ-4-import-smtp.md) | 2026-03-12 |
|
| PROJ-4 | E-Mail-Import: SMTP-Eingang via BCC (primär) | Deployed | [PROJ-4](PROJ-4-import-smtp.md) | 2026-03-12 |
|
||||||
| PROJ-5 | E-Mail-Speicherung & Volltext-Indexierung | Deployed | [PROJ-5](PROJ-5-speicherung-und-indexierung.md) | 2026-03-12 |
|
| PROJ-5 | E-Mail-Speicherung & Volltext-Indexierung | Deployed | [PROJ-5](PROJ-5-speicherung-und-indexierung.md) | 2026-03-12 |
|
||||||
| PROJ-6 | Volltext-Suche & Filterung | In Review | [PROJ-6](PROJ-6-volltext-suche.md) | 2026-03-12 |
|
| PROJ-6 | Volltext-Suche & Filterung | Deployed | [PROJ-6](PROJ-6-volltext-suche.md) | 2026-03-12 |
|
||||||
| PROJ-7 | E-Mail-Ansicht (Lesen & Anhänge) | In Progress | [PROJ-7](PROJ-7-email-ansicht.md) | 2026-03-12 |
|
| PROJ-7 | E-Mail-Ansicht (Lesen & Anhänge) | Deployed | [PROJ-7](PROJ-7-email-ansicht.md) | 2026-03-12 |
|
||||||
| PROJ-8 | Automatischer IMAP-Sync (Cron-Job) | Deployed | [PROJ-8](PROJ-8-imap-auto-sync.md) | 2026-03-12 |
|
| PROJ-8 | Automatischer IMAP-Sync (Cron-Job) | Deployed | [PROJ-8](PROJ-8-imap-auto-sync.md) | 2026-03-12 |
|
||||||
| PROJ-9 | Ordner- & Label-Verwaltung | In Progress | [PROJ-9](PROJ-9-ordner-und-labels.md) | 2026-03-12 |
|
| PROJ-9 | Ordner- & Label-Verwaltung | In Progress | [PROJ-9](PROJ-9-ordner-und-labels.md) | 2026-03-12 |
|
||||||
| PROJ-10 | Admin-Bereich: Nutzer- & Postfachverwaltung | In Progress | [PROJ-10](PROJ-10-admin-bereich.md) | 2026-03-12 |
|
| PROJ-10 | Admin-Bereich: Nutzer- & Postfachverwaltung | Deployed | [PROJ-10](PROJ-10-admin-bereich.md) | 2026-03-12 |
|
||||||
| PROJ-11 | Audit-Log & Compliance-Berichte | In Progress | [PROJ-11](PROJ-11-audit-log.md) | 2026-03-12 |
|
| PROJ-11 | Audit-Log & Compliance-Berichte | Deployed | [PROJ-11](PROJ-11-audit-log.md) | 2026-03-12 |
|
||||||
| PROJ-12 | E-Mail-Export (EML/PDF) | Deployed | [PROJ-12](PROJ-12-export.md) | 2026-03-12 |
|
| PROJ-12 | E-Mail-Export (EML/PDF) | Deployed | [PROJ-12](PROJ-12-export.md) | 2026-03-12 |
|
||||||
| PROJ-13 | REST API für externe CRM-Anbindung | In Progress | [PROJ-13](PROJ-13-rest-api-crm.md) | 2026-03-13 |
|
| PROJ-13 | REST API für externe CRM-Anbindung | In Progress | [PROJ-13](PROJ-13-rest-api-crm.md) | 2026-03-13 |
|
||||||
| PROJ-14 | E-Mail-Import: POP3-Verbindung | In Progress | [PROJ-14](PROJ-14-import-pop3.md) | 2026-03-13 |
|
| PROJ-14 | E-Mail-Import: POP3-Verbindung | In Progress | [PROJ-14](PROJ-14-import-pop3.md) | 2026-03-13 |
|
||||||
@@ -31,7 +31,8 @@
|
|||||||
|
|
||||||
| PROJ-17 | Admin Dashboard – Systemauslastung & Archiv-Übersicht | Deployed | [PROJ-17](PROJ-17-system-dashboard.md) | 2026-03-14 |
|
| PROJ-17 | Admin Dashboard – Systemauslastung & Archiv-Übersicht | Deployed | [PROJ-17](PROJ-17-system-dashboard.md) | 2026-03-14 |
|
||||||
| PROJ-18 | E-Mail Integritätsprüfung | Deployed | [PROJ-18](PROJ-18-integritaetspruefung.md) | 2026-03-14 |
|
| PROJ-18 | E-Mail Integritätsprüfung | Deployed | [PROJ-18](PROJ-18-integritaetspruefung.md) | 2026-03-14 |
|
||||||
|
| PROJ-19 | Mailpiler → archivmail Migrationstool | Deployed | [PROJ-19](PROJ-19-import-piler.md) | 2026-03-17 |
|
||||||
|
|
||||||
<!-- Add features above this line -->
|
<!-- Add features above this line -->
|
||||||
|
|
||||||
## Next Available ID: PROJ-19
|
## Next Available ID: PROJ-20
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# PROJ-10: Admin-Bereich: Nutzer- & Postfachverwaltung
|
# PROJ-10: Admin-Bereich: Nutzer- & Postfachverwaltung
|
||||||
|
|
||||||
## Status: In Progress
|
## Status: Deployed
|
||||||
**Created:** 2026-03-12
|
**Created:** 2026-03-12
|
||||||
**Last Updated:** 2026-03-12
|
**Last Updated:** 2026-03-12
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# PROJ-11: Audit-Log & Compliance-Berichte
|
# PROJ-11: Audit-Log & Compliance-Berichte
|
||||||
|
|
||||||
## Status: In Progress
|
## Status: Deployed
|
||||||
**Created:** 2026-03-12
|
**Created:** 2026-03-12
|
||||||
**Last Updated:** 2026-03-13
|
**Last Updated:** 2026-03-13
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# PROJ-19: Mailpiler → archivmail Migrationstool
|
||||||
|
|
||||||
|
## Status: Deployed
|
||||||
|
**Created:** 2026-03-17
|
||||||
|
**Last Updated:** 2026-03-17
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- Requires: PROJ-5 (Speicherung & Indexierung)
|
||||||
|
- Requires: PROJ-15 (CLI Import)
|
||||||
|
|
||||||
|
## User Stories
|
||||||
|
- Als Admin möchte ich alle E-Mails aus einem bestehenden mailpiler-Archiv nach archivmail migrieren, damit ich die Plattform wechseln kann ohne E-Mails zu verlieren.
|
||||||
|
- Als Admin möchte ich den Fortschritt der Migration in Echtzeit sehen.
|
||||||
|
- Als Admin möchte ich Duplikate automatisch überspringen, damit bei Teil-Migrationen oder Wiederholungsläufen keine Daten doppelt archiviert werden.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
- [x] Methode 1: `pilerexport`-Wrapper – ruft das mailpiler-eigene Export-Tool auf, importiert die EML-Dateien
|
||||||
|
- [x] Methode 2: Direkt – liest `.m`-Dateien aus dem mailpiler Store-Verzeichnis, entschlüsselt (AES-256-CBC) und dekomprimiert (zlib)
|
||||||
|
- [x] Automatische Methodenwahl (`--method auto`): pilerexport → direct
|
||||||
|
- [x] Fortschrittsanzeige (importiert / übersprungen / Fehler)
|
||||||
|
- [x] `--dry-run` Modus
|
||||||
|
- [x] JSON-Ausgabe für Skripting
|
||||||
|
- [x] Datums-Filter (`--date-from`, `--date-to`) für pilerexport-Methode
|
||||||
|
|
||||||
|
## Aufruf
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Auf dem mailpiler-Server (pilerexport-Methode, empfohlen):
|
||||||
|
archivmail import-piler \
|
||||||
|
--config /etc/archivmail/config.yml \
|
||||||
|
--method pilerexport
|
||||||
|
|
||||||
|
# Mit Datumsfilter:
|
||||||
|
archivmail import-piler \
|
||||||
|
--config /etc/archivmail/config.yml \
|
||||||
|
--date-from 2020-01-01 \
|
||||||
|
--date-to 2024-12-31
|
||||||
|
|
||||||
|
# Direkte Methode (kein mailpiler nötig, kein MySQL nötig):
|
||||||
|
archivmail import-piler \
|
||||||
|
--config /etc/archivmail/config.yml \
|
||||||
|
--method direct \
|
||||||
|
--store-dir /var/piler/store \
|
||||||
|
--key-file /var/piler/store/piler.key
|
||||||
|
|
||||||
|
# Nur Simulation (kein Speichern):
|
||||||
|
archivmail import-piler --dry-run
|
||||||
|
|
||||||
|
# JSON-Ausgabe für Skripte:
|
||||||
|
archivmail import-piler --json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Technische Details
|
||||||
|
|
||||||
|
| Aspekt | Detail |
|
||||||
|
|--------|--------|
|
||||||
|
| pilerexport-Ausgabe | EML-Dateien im temp-Verzeichnis, jede `.eml` = eine E-Mail |
|
||||||
|
| mailpiler-Dateiformat | `{storedir}/**/{piler_id}.m` – AES-256-CBC verschlüsselt, zlib komprimiert |
|
||||||
|
| Entschlüsselung | Erste 16 Bytes = IV, Rest = CBC-Ciphertext, Key aus `piler.key` (32 Bytes) |
|
||||||
|
| Dekomprimierung | zlib (ohne AES falls kein Key vorhanden) |
|
||||||
|
| Duplikat-Erkennung | Storage.Save() + IsIndexed() – identische Inhalts-Hashes werden übersprungen |
|
||||||
|
| Keine ext. Abhängigkeiten | Nur Go stdlib (compress/zlib, crypto/aes) + vorhandene archivmail-Pakete |
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
# PROJ-3: E-Mail-Import: IMAP-Verbindung
|
# PROJ-3: E-Mail-Import: IMAP-Verbindung
|
||||||
|
|
||||||
## Status: In Review
|
## Status: Deployed
|
||||||
**Created:** 2026-03-12
|
**Created:** 2026-03-12
|
||||||
**Last Updated:** 2026-03-14
|
**Last Updated:** 2026-03-17
|
||||||
|
|
||||||
## Dependencies
|
## Dependencies
|
||||||
- Requires: PROJ-1 (Authentifizierung) – nur Admins verwalten IMAP-Verbindungen
|
- Requires: PROJ-1 (Authentifizierung) – nur Admins verwalten IMAP-Verbindungen
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# PROJ-4: E-Mail-Import: SMTP-Eingang (primär via BCC)
|
# PROJ-4: E-Mail-Import: SMTP-Eingang (primär via BCC)
|
||||||
|
|
||||||
## Status: In Progress
|
## Status: Deployed
|
||||||
**Created:** 2026-03-12
|
**Created:** 2026-03-12
|
||||||
**Last Updated:** 2026-03-12
|
**Last Updated:** 2026-03-17
|
||||||
|
|
||||||
## Hinweis
|
## Hinweis
|
||||||
**Dies ist der primäre Eingangsweg.** archivmail enthält einen eingebetteten SMTP-Daemon, der **ausschließlich E-Mails empfängt** – kein Versand, keine Weiterleitung, kein MTA. Postfix (oder ein anderer Mailserver) wird per BCC-Mapping oder Always-BCC-Regel so konfiguriert, dass er eine Kopie jeder E-Mail an archivmails SMTP-Daemon zustellt.
|
**Dies ist der primäre Eingangsweg.** archivmail enthält einen eingebetteten SMTP-Daemon, der **ausschließlich E-Mails empfängt** – kein Versand, keine Weiterleitung, kein MTA. Postfix (oder ein anderer Mailserver) wird per BCC-Mapping oder Always-BCC-Regel so konfiguriert, dass er eine Kopie jeder E-Mail an archivmails SMTP-Daemon zustellt.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# PROJ-6: Volltext-Suche & Filterung
|
# PROJ-6: Volltext-Suche & Filterung
|
||||||
|
|
||||||
## Status: In Review
|
## Status: Deployed
|
||||||
**Created:** 2026-03-12
|
**Created:** 2026-03-12
|
||||||
**Last Updated:** 2026-03-17
|
**Last Updated:** 2026-03-17
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# PROJ-7: E-Mail-Ansicht (Lesen & Anhänge)
|
# PROJ-7: E-Mail-Ansicht (Lesen & Anhänge)
|
||||||
|
|
||||||
## Status: In Progress
|
## Status: Deployed
|
||||||
**Created:** 2026-03-12
|
**Created:** 2026-03-12
|
||||||
**Last Updated:** 2026-03-12
|
**Last Updated:** 2026-03-17
|
||||||
|
|
||||||
## Dependencies
|
## Dependencies
|
||||||
- Requires: PROJ-1 (Authentifizierung) – nur eingeloggte Nutzer mit Zugriffsrecht
|
- Requires: PROJ-1 (Authentifizierung) – nur eingeloggte Nutzer mit Zugriffsrecht
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ import (
|
|||||||
"runtime"
|
"runtime"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -45,6 +46,7 @@ type Server struct {
|
|||||||
imapStore *imapstore.Store
|
imapStore *imapstore.Store
|
||||||
imapImporter *imapstore.Importer
|
imapImporter *imapstore.Importer
|
||||||
imapScheduler *imapstore.Scheduler
|
imapScheduler *imapstore.Scheduler
|
||||||
|
uploadJobs sync.Map // jobID → *UploadJob
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetSMTPDaemon wires the SMTP daemon into the API server after construction.
|
// SetSMTPDaemon wires the SMTP daemon into the API server after construction.
|
||||||
@@ -108,6 +110,10 @@ func (s *Server) routes() {
|
|||||||
s.mux.HandleFunc("GET /api/export/pdf/{id}", s.authMiddleware(s.requireMailAccess(s.handleExportPDF)))
|
s.mux.HandleFunc("GET /api/export/pdf/{id}", s.authMiddleware(s.requireMailAccess(s.handleExportPDF)))
|
||||||
s.mux.HandleFunc("POST /api/export/zip", s.authMiddleware(s.requireMailAccess(s.handleExportZIP)))
|
s.mux.HandleFunc("POST /api/export/zip", s.authMiddleware(s.requireMailAccess(s.handleExportZIP)))
|
||||||
|
|
||||||
|
// Upload routes (admin only)
|
||||||
|
s.mux.HandleFunc("POST /api/admin/upload", s.authMiddleware(s.requireRole(userstore.RoleAdmin, s.handleUpload)))
|
||||||
|
s.mux.HandleFunc("GET /api/admin/upload/{jobID}/progress", s.authMiddleware(s.requireRole(userstore.RoleAdmin, s.handleUploadProgress)))
|
||||||
|
|
||||||
// IMAP routes (accessible to all authenticated users)
|
// IMAP routes (accessible to all authenticated users)
|
||||||
s.mux.HandleFunc("GET /api/imap", s.authMiddleware(s.handleListImap))
|
s.mux.HandleFunc("GET /api/imap", s.authMiddleware(s.handleListImap))
|
||||||
s.mux.HandleFunc("POST /api/imap", s.authMiddleware(s.handleCreateImap))
|
s.mux.HandleFunc("POST /api/imap", s.authMiddleware(s.handleCreateImap))
|
||||||
|
|||||||
@@ -0,0 +1,211 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"github.com/archivmail/internal/index"
|
||||||
|
"github.com/archivmail/pkg/mailparser"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UploadJob tracks the progress of an EML/MBOX import job.
|
||||||
|
type UploadJob struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
ID string `json:"id"`
|
||||||
|
Status string `json:"status"` // "running" | "done" | "error"
|
||||||
|
Total int `json:"total"`
|
||||||
|
Imported int `json:"imported"`
|
||||||
|
Skipped int `json:"skipped"`
|
||||||
|
Errors int `json:"errors"`
|
||||||
|
ErrMsg string `json:"error_msg,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (j *UploadJob) snapshot() uploadJobSnapshot {
|
||||||
|
j.mu.Lock()
|
||||||
|
defer j.mu.Unlock()
|
||||||
|
return uploadJobSnapshot{
|
||||||
|
ID: j.ID,
|
||||||
|
Status: j.Status,
|
||||||
|
Total: j.Total,
|
||||||
|
Imported: j.Imported,
|
||||||
|
Skipped: j.Skipped,
|
||||||
|
Errors: j.Errors,
|
||||||
|
ErrMsg: j.ErrMsg,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type uploadJobSnapshot struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Status string `json:"status"`
|
||||||
|
Total int `json:"total"`
|
||||||
|
Imported int `json:"imported"`
|
||||||
|
Skipped int `json:"skipped"`
|
||||||
|
Errors int `json:"errors"`
|
||||||
|
ErrMsg string `json:"error_msg,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleUpload accepts a multipart upload of one or more .eml or .mbox files,
|
||||||
|
// starts a background import job and returns its ID immediately.
|
||||||
|
func (s *Server) handleUpload(w http.ResponseWriter, r *http.Request) {
|
||||||
|
// 512 MB max total upload
|
||||||
|
if err := r.ParseMultipartForm(512 << 20); err != nil {
|
||||||
|
writeError(w, http.StatusBadRequest, "multipart parse failed: "+err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
files := r.MultipartForm.File["files"]
|
||||||
|
if len(files) == 0 {
|
||||||
|
writeError(w, http.StatusBadRequest, "no files uploaded")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Collect all raw messages from uploaded files
|
||||||
|
type rawEntry struct {
|
||||||
|
data []byte
|
||||||
|
isMbox bool
|
||||||
|
}
|
||||||
|
var entries []rawEntry
|
||||||
|
|
||||||
|
for _, fh := range files {
|
||||||
|
f, err := fh.Open()
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
buf := make([]byte, fh.Size)
|
||||||
|
f.Read(buf) //nolint
|
||||||
|
f.Close()
|
||||||
|
|
||||||
|
name := strings.ToLower(fh.Filename)
|
||||||
|
isMbox := strings.HasSuffix(name, ".mbox")
|
||||||
|
entries = append(entries, rawEntry{buf, isMbox})
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(entries) == 0 {
|
||||||
|
writeError(w, http.StatusBadRequest, "no readable files")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Count total messages upfront
|
||||||
|
var allMessages [][]byte
|
||||||
|
for _, e := range entries {
|
||||||
|
if e.isMbox {
|
||||||
|
msgs := mailparser.SplitMbox(e.data)
|
||||||
|
allMessages = append(allMessages, msgs...)
|
||||||
|
} else {
|
||||||
|
allMessages = append(allMessages, e.data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
jobID := newJobID()
|
||||||
|
job := &UploadJob{
|
||||||
|
ID: jobID,
|
||||||
|
Status: "running",
|
||||||
|
Total: len(allMessages),
|
||||||
|
}
|
||||||
|
s.uploadJobs.Store(jobID, job)
|
||||||
|
|
||||||
|
// Run import in background
|
||||||
|
go s.runUploadJob(job, allMessages)
|
||||||
|
|
||||||
|
writeJSON(w, http.StatusAccepted, map[string]string{"job_id": jobID})
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleUploadProgress returns the current status of an upload job.
|
||||||
|
func (s *Server) handleUploadProgress(w http.ResponseWriter, r *http.Request) {
|
||||||
|
jobID := r.PathValue("jobID")
|
||||||
|
val, ok := s.uploadJobs.Load(jobID)
|
||||||
|
if !ok {
|
||||||
|
writeError(w, http.StatusNotFound, "job not found")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
job := val.(*UploadJob)
|
||||||
|
writeJSON(w, http.StatusOK, job.snapshot())
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) runUploadJob(job *UploadJob, messages [][]byte) {
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
for _, raw := range messages {
|
||||||
|
result := s.importRawMessage(ctx, raw)
|
||||||
|
job.mu.Lock()
|
||||||
|
switch result {
|
||||||
|
case "imported":
|
||||||
|
job.Imported++
|
||||||
|
case "skipped":
|
||||||
|
job.Skipped++
|
||||||
|
default:
|
||||||
|
job.Errors++
|
||||||
|
}
|
||||||
|
job.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
job.mu.Lock()
|
||||||
|
job.Status = "done"
|
||||||
|
job.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// importRawMessage stores and indexes a single raw message.
|
||||||
|
// Returns "imported", "skipped", or "error".
|
||||||
|
func (s *Server) importRawMessage(ctx context.Context, raw []byte) string {
|
||||||
|
pm, err := mailparser.Parse(raw)
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Warn("upload: parse failed", "err", err)
|
||||||
|
return "error"
|
||||||
|
}
|
||||||
|
|
||||||
|
id, err := s.store.Save(raw, pm.Date)
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Warn("upload: save failed", "err", err)
|
||||||
|
return "error"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check dedup: storage.Save returns same id for duplicate content.
|
||||||
|
// If already indexed, skip indexing.
|
||||||
|
if already, _ := s.store.IsIndexed(ctx, id); already {
|
||||||
|
return "skipped"
|
||||||
|
}
|
||||||
|
|
||||||
|
var attachNames []string
|
||||||
|
for _, a := range pm.Attachments {
|
||||||
|
if a.Filename != "" {
|
||||||
|
attachNames = append(attachNames, a.Filename)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
doc := index.MailDocument{
|
||||||
|
ID: id,
|
||||||
|
From: pm.From,
|
||||||
|
To: strings.Join(pm.To, " "),
|
||||||
|
Subject: pm.Subject,
|
||||||
|
Body: pm.TextBody + " " + pm.HTMLBody,
|
||||||
|
AttachNames: strings.Join(attachNames, " "),
|
||||||
|
HasAttachment: len(pm.Attachments) > 0,
|
||||||
|
Date: pm.Date,
|
||||||
|
Size: int64(len(raw)),
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.idx.IndexSync(doc); err != nil {
|
||||||
|
s.logger.Warn("upload: index failed", "id", id, "err", err)
|
||||||
|
return "error"
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.store.SetIndexedAt(ctx, id); err != nil {
|
||||||
|
s.logger.Warn("upload: set indexed_at failed", "id", id, "err", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.store.SaveMeta(ctx, id, pm, len(raw)); err != nil {
|
||||||
|
s.logger.Warn("upload: save meta failed", "id", id, "err", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return "imported"
|
||||||
|
}
|
||||||
|
|
||||||
|
func newJobID() string {
|
||||||
|
b := make([]byte, 8)
|
||||||
|
rand.Read(b) //nolint
|
||||||
|
return hex.EncodeToString(b)
|
||||||
|
}
|
||||||
@@ -15,12 +15,15 @@ import {
|
|||||||
getServices,
|
getServices,
|
||||||
serviceAction,
|
serviceAction,
|
||||||
getSystemStats,
|
getSystemStats,
|
||||||
|
uploadMailFiles,
|
||||||
|
getUploadProgress,
|
||||||
type User,
|
type User,
|
||||||
type AuditEntry,
|
type AuditEntry,
|
||||||
type SMTPStatus,
|
type SMTPStatus,
|
||||||
type StorageStats,
|
type StorageStats,
|
||||||
type ServiceStatus,
|
type ServiceStatus,
|
||||||
type SystemStats,
|
type SystemStats,
|
||||||
|
type UploadJob,
|
||||||
} from "@/lib/api";
|
} from "@/lib/api";
|
||||||
import { Navbar } from "@/components/navbar";
|
import { Navbar } from "@/components/navbar";
|
||||||
import { Button } from "@/components/ui/button";
|
import { Button } from "@/components/ui/button";
|
||||||
@@ -111,6 +114,13 @@ export default function AdminPage() {
|
|||||||
const [auditPage, setAuditPage] = useState(1);
|
const [auditPage, setAuditPage] = useState(1);
|
||||||
const [auditLoading, setAuditLoading] = useState(false);
|
const [auditLoading, setAuditLoading] = useState(false);
|
||||||
|
|
||||||
|
// Upload state
|
||||||
|
const [uploadDragging, setUploadDragging] = useState(false);
|
||||||
|
const [uploadJob, setUploadJob] = useState<UploadJob | null>(null);
|
||||||
|
const [uploadError, setUploadError] = useState("");
|
||||||
|
const [uploadLoading, setUploadLoading] = useState(false);
|
||||||
|
const uploadPollRef = useRef<ReturnType<typeof setInterval> | null>(null);
|
||||||
|
|
||||||
const loadDashboard = useCallback(async () => {
|
const loadDashboard = useCallback(async () => {
|
||||||
setDashLoading(true);
|
setDashLoading(true);
|
||||||
try {
|
try {
|
||||||
@@ -170,6 +180,40 @@ export default function AdminPage() {
|
|||||||
}
|
}
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
|
async function handleUploadFiles(files: File[]) {
|
||||||
|
const valid = files.filter(f => f.name.toLowerCase().endsWith(".eml") || f.name.toLowerCase().endsWith(".mbox"));
|
||||||
|
if (valid.length === 0) {
|
||||||
|
setUploadError("Nur .eml und .mbox Dateien erlaubt.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setUploadError("");
|
||||||
|
setUploadJob(null);
|
||||||
|
setUploadLoading(true);
|
||||||
|
try {
|
||||||
|
const { job_id } = await uploadMailFiles(valid);
|
||||||
|
// Start polling
|
||||||
|
const poll = setInterval(async () => {
|
||||||
|
try {
|
||||||
|
const job = await getUploadProgress(job_id);
|
||||||
|
setUploadJob(job);
|
||||||
|
if (job.status !== "running") {
|
||||||
|
clearInterval(poll);
|
||||||
|
uploadPollRef.current = null;
|
||||||
|
setUploadLoading(false);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
clearInterval(poll);
|
||||||
|
uploadPollRef.current = null;
|
||||||
|
setUploadLoading(false);
|
||||||
|
}
|
||||||
|
}, 1500);
|
||||||
|
uploadPollRef.current = poll;
|
||||||
|
} catch (e: unknown) {
|
||||||
|
setUploadError(e instanceof Error ? e.message : "Upload fehlgeschlagen.");
|
||||||
|
setUploadLoading(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function handleServiceAction(name: string, action: "start" | "stop" | "restart" | "enable" | "disable" | "block_external" | "allow_external") {
|
async function handleServiceAction(name: string, action: "start" | "stop" | "restart" | "enable" | "disable" | "block_external" | "allow_external") {
|
||||||
setServiceActionLoading(`${name}:${action}`);
|
setServiceActionLoading(`${name}:${action}`);
|
||||||
setServiceError("");
|
setServiceError("");
|
||||||
@@ -296,6 +340,7 @@ export default function AdminPage() {
|
|||||||
<TabsTrigger value="services">Dienste</TabsTrigger>
|
<TabsTrigger value="services">Dienste</TabsTrigger>
|
||||||
<TabsTrigger value="users">Benutzer</TabsTrigger>
|
<TabsTrigger value="users">Benutzer</TabsTrigger>
|
||||||
<TabsTrigger value="audit">Audit-Log</TabsTrigger>
|
<TabsTrigger value="audit">Audit-Log</TabsTrigger>
|
||||||
|
<TabsTrigger value="import">Import</TabsTrigger>
|
||||||
<TabsTrigger value="modules">Module</TabsTrigger>
|
<TabsTrigger value="modules">Module</TabsTrigger>
|
||||||
</TabsList>
|
</TabsList>
|
||||||
|
|
||||||
@@ -1019,6 +1064,102 @@ export default function AdminPage() {
|
|||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
</TabsContent>
|
</TabsContent>
|
||||||
|
{/* ── Import ── */}
|
||||||
|
<TabsContent value="import" className="mt-4 space-y-4">
|
||||||
|
<h2 className="text-lg font-semibold">EML / MBOX importieren</h2>
|
||||||
|
<p className="text-sm text-muted-foreground">
|
||||||
|
Lade .eml oder .mbox Dateien hoch um sie ins Archiv zu importieren. Duplikate werden automatisch übersprungen.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{/* Drop zone */}
|
||||||
|
<div
|
||||||
|
onDragOver={(e) => { e.preventDefault(); setUploadDragging(true); }}
|
||||||
|
onDragLeave={() => setUploadDragging(false)}
|
||||||
|
onDrop={(e) => {
|
||||||
|
e.preventDefault();
|
||||||
|
setUploadDragging(false);
|
||||||
|
const files = Array.from(e.dataTransfer.files);
|
||||||
|
handleUploadFiles(files);
|
||||||
|
}}
|
||||||
|
className={`border-2 border-dashed rounded-lg p-10 text-center transition-colors cursor-pointer ${
|
||||||
|
uploadDragging ? "border-primary bg-primary/5" : "border-muted-foreground/25 hover:border-primary/50"
|
||||||
|
}`}
|
||||||
|
onClick={() => document.getElementById("upload-file-input")?.click()}
|
||||||
|
>
|
||||||
|
<input
|
||||||
|
id="upload-file-input"
|
||||||
|
type="file"
|
||||||
|
multiple
|
||||||
|
accept=".eml,.mbox"
|
||||||
|
className="hidden"
|
||||||
|
onChange={(e) => {
|
||||||
|
if (e.target.files) handleUploadFiles(Array.from(e.target.files));
|
||||||
|
e.target.value = "";
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<p className="text-sm font-medium">Dateien hierher ziehen oder klicken zum Auswählen</p>
|
||||||
|
<p className="text-xs text-muted-foreground mt-1">Akzeptiert: .eml, .mbox</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{uploadError && (
|
||||||
|
<Alert variant="destructive">
|
||||||
|
<AlertDescription>{uploadError}</AlertDescription>
|
||||||
|
</Alert>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Progress */}
|
||||||
|
{(uploadLoading || uploadJob) && uploadJob && (
|
||||||
|
<Card>
|
||||||
|
<CardContent className="p-4 space-y-3">
|
||||||
|
<div className="flex items-center justify-between">
|
||||||
|
<span className="text-sm font-medium">
|
||||||
|
{uploadJob.status === "running" ? "Import läuft..." : "Import abgeschlossen"}
|
||||||
|
</span>
|
||||||
|
<Badge variant={uploadJob.status === "done" ? "default" : "secondary"}>
|
||||||
|
{uploadJob.status === "done" ? "Fertig" : "Läuft"}
|
||||||
|
</Badge>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Progress bar */}
|
||||||
|
{uploadJob.total > 0 && (
|
||||||
|
<div className="space-y-1">
|
||||||
|
<div className="w-full bg-muted rounded-full h-2">
|
||||||
|
<div
|
||||||
|
className="bg-primary h-2 rounded-full transition-all"
|
||||||
|
style={{ width: `${Math.min(100, ((uploadJob.imported + uploadJob.skipped + uploadJob.errors) / uploadJob.total) * 100)}%` }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<p className="text-xs text-muted-foreground">
|
||||||
|
{uploadJob.imported + uploadJob.skipped + uploadJob.errors} / {uploadJob.total} verarbeitet
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{uploadJob.status === "done" && (
|
||||||
|
<div className="grid grid-cols-3 gap-3 text-center text-sm">
|
||||||
|
<div className="rounded bg-green-50 dark:bg-green-950 p-2">
|
||||||
|
<p className="font-bold text-green-700 dark:text-green-400">{uploadJob.imported}</p>
|
||||||
|
<p className="text-xs text-muted-foreground">Importiert</p>
|
||||||
|
</div>
|
||||||
|
<div className="rounded bg-yellow-50 dark:bg-yellow-950 p-2">
|
||||||
|
<p className="font-bold text-yellow-700 dark:text-yellow-400">{uploadJob.skipped}</p>
|
||||||
|
<p className="text-xs text-muted-foreground">Übersprungen</p>
|
||||||
|
</div>
|
||||||
|
<div className="rounded bg-red-50 dark:bg-red-950 p-2">
|
||||||
|
<p className="font-bold text-red-700 dark:text-red-400">{uploadJob.errors}</p>
|
||||||
|
<p className="text-xs text-muted-foreground">Fehler</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</CardContent>
|
||||||
|
</Card>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{uploadLoading && !uploadJob && (
|
||||||
|
<p className="text-sm text-muted-foreground animate-pulse">Upload läuft...</p>
|
||||||
|
)}
|
||||||
|
</TabsContent>
|
||||||
|
|
||||||
<TabsContent value="modules" className="mt-4">
|
<TabsContent value="modules" className="mt-4">
|
||||||
<ModulesTab />
|
<ModulesTab />
|
||||||
</TabsContent>
|
</TabsContent>
|
||||||
|
|||||||
@@ -453,3 +453,34 @@ export async function exportMailsZIP(ids: string[], attachments: boolean): Promi
|
|||||||
if (!res.ok) throw new Error("ZIP export failed");
|
if (!res.ok) throw new Error("ZIP export failed");
|
||||||
return { blob: await res.blob() };
|
return { blob: await res.blob() };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Upload ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
export interface UploadJob {
|
||||||
|
id: string;
|
||||||
|
status: "running" | "done" | "error";
|
||||||
|
total: number;
|
||||||
|
imported: number;
|
||||||
|
skipped: number;
|
||||||
|
errors: number;
|
||||||
|
error_msg?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function uploadMailFiles(files: File[]): Promise<{ job_id: string }> {
|
||||||
|
const form = new FormData();
|
||||||
|
for (const f of files) form.append("files", f);
|
||||||
|
const res = await fetch(`${API_BASE}/api/admin/upload`, {
|
||||||
|
method: "POST",
|
||||||
|
credentials: "include",
|
||||||
|
body: form,
|
||||||
|
});
|
||||||
|
if (!res.ok) {
|
||||||
|
const body = await res.text();
|
||||||
|
throw new Error(body || `Upload failed: ${res.status}`);
|
||||||
|
}
|
||||||
|
return res.json();
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function getUploadProgress(jobID: string): Promise<UploadJob> {
|
||||||
|
return request<UploadJob>(`/api/admin/upload/${jobID}/progress`);
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user