docs: vollständige README, PROJ-2 Web-Upload, PROJ-19 Mailpiler-Migration
README.md:
- Vollständige Dokumentation aller implementierten Funktionen
- Konfigurationsreferenz, Installation, Systemd, REST-API-Übersicht
- In-Progress-Features klar gekennzeichnet
PROJ-2 (EML/MBOX Web-Upload):
- POST /api/admin/upload – Multipart-Upload mit Hintergrund-Job
- GET /api/admin/upload/{jobID}/progress – Polling
- Admin-Tab "Import" mit Drag-and-Drop, Fortschrittsbalken, Abschlussbericht
PROJ-19 (Mailpiler Migration):
- archivmail import-piler mit Methoden: pilerexport | direct | auto
- Direct: AES-256-CBC + zlib mit defensiven Fallbacks
- pilerexport: Wrapper um mailpilers Export-Tool
Status-Updates: PROJ-3, PROJ-4, PROJ-6, PROJ-7, PROJ-10, PROJ-11 → Deployed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,94 +1,835 @@
|
||||
# archivmail
|
||||
|
||||
Selbst gehostetes Mail-Archiv-System für Unternehmen. E-Mails werden aus IMAP, SMTP und EML/MBOX-Quellen importiert, volltext-indexiert (Xapian) und verschlüsselt archiviert.
|
||||
Ein selbst gehostetes, unternehmenstaugliches Mail-Archiv-System. Empfängt E-Mails über SMTP (BCC-Journaling), IMAP oder Datei-Upload, speichert sie verschlüsselt, indexiert sie für Volltextsuche und stellt sie über eine Web-Oberfläche zur Verfügung.
|
||||
|
||||
## Features
|
||||
---
|
||||
|
||||
- Import via IMAP, SMTP-Journaling und EML/MBOX-Upload
|
||||
- Volltext-Suche über Xapian
|
||||
- AES-256-GCM-Verschlüsselung der archivierten Mails
|
||||
- Anhang-Deduplizierung
|
||||
- Rollenmodell: `user`, `auditor`, `admin`
|
||||
- Audit-Log (PostgreSQL + Append-only Logdatei)
|
||||
## Inhaltsverzeichnis
|
||||
|
||||
- [Funktionsübersicht](#funktionsübersicht)
|
||||
- [Architektur](#architektur)
|
||||
- [Voraussetzungen](#voraussetzungen)
|
||||
- [Installation](#installation)
|
||||
- [Konfiguration](#konfiguration)
|
||||
- [Systemd-Dienste](#systemd-dienste)
|
||||
- [Funktionen im Detail](#funktionen-im-detail)
|
||||
- [Authentifizierung & Rollen](#authentifizierung--rollen)
|
||||
- [SMTP-Eingang (BCC-Journaling)](#smtp-eingang-bcc-journaling)
|
||||
- [IMAP-Import & Auto-Sync](#imap-import--auto-sync)
|
||||
- [EML/MBOX Web-Upload](#emlmbox-web-upload)
|
||||
- [Speicherung & Verschlüsselung](#speicherung--verschlüsselung)
|
||||
- [Volltext-Suche](#volltext-suche)
|
||||
- [E-Mail-Ansicht](#e-mail-ansicht)
|
||||
- [E-Mail-Export](#e-mail-export)
|
||||
- [Admin-Dashboard](#admin-dashboard)
|
||||
- [Audit-Log](#audit-log)
|
||||
- [Integritätsprüfung](#integritätsprüfung)
|
||||
- [CLI Import & Export](#cli-import--export)
|
||||
- [Mailpiler Migration](#mailpiler-migration)
|
||||
- [REST API](#rest-api)
|
||||
- [In Entwicklung](#in-entwicklung)
|
||||
- [Update](#update)
|
||||
|
||||
---
|
||||
|
||||
## Funktionsübersicht
|
||||
|
||||
| Funktion | Status |
|
||||
|----------|--------|
|
||||
| Authentifizierung & Rollen (Admin / Auditor / User) | ✅ Deployed |
|
||||
| SMTP-Eingang via BCC-Journaling | ✅ Deployed |
|
||||
| IMAP-Import manuell & automatischer Sync | ✅ Deployed |
|
||||
| EML/MBOX Web-Upload (Admin) | ✅ Deployed |
|
||||
| AES-256-GCM verschlüsselte Speicherung | ✅ Deployed |
|
||||
| Xapian Volltext-Indexierung (async) | ✅ Deployed |
|
||||
| Volltext-Suche mit Filtern & Sortierung | ✅ Deployed |
|
||||
| E-Mail-Ansicht mit HTML-Sandbox | ✅ Deployed |
|
||||
| E-Mail-Export als EML / PDF / ZIP | ✅ Deployed |
|
||||
| Admin-Dashboard (CPU, RAM, Disk, Archiv-Stats) | ✅ Deployed |
|
||||
| Audit-Log (Suche, Login, Import, Export) | ✅ Deployed |
|
||||
| SHA-256 Integritätsprüfung | ✅ Deployed |
|
||||
| CLI: `archivmail import` / `export` | ✅ Deployed |
|
||||
| Mailpiler → archivmail Migration | ✅ Deployed |
|
||||
| Dienste-Verwaltung im Admin-Bereich | ✅ Deployed |
|
||||
| Ordner- & Label-Verwaltung | 🔄 In Progress |
|
||||
| POP3-Import | 🔄 In Progress |
|
||||
| REST API für externe Anbindung (CRM) | 🔄 In Progress |
|
||||
| LDAP / Active Directory | 🔄 In Progress |
|
||||
|
||||
---
|
||||
|
||||
## Architektur
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ archivmail (Go-Binary) │
|
||||
│ │
|
||||
Postfix BCC ──┤── SMTP-Daemon (Port 2525) │
|
||||
IMAP-Server ──┤── IMAP-Importer + Scheduler │──► PostgreSQL
|
||||
Web-Upload ──┤── HTTP API (Port 8080) │ (Metadaten)
|
||||
│ │ │
|
||||
│ ▼ │──► /var/archivmail/store
|
||||
│ Async Index Worker │ (AES-256-GCM)
|
||||
│ │ │
|
||||
│ ▼ │──► /var/archivmail/xapian
|
||||
│ Xapian Index │ (Volltext-Index)
|
||||
└─────────────────────────────────────┘
|
||||
▲
|
||||
│ /api/*
|
||||
┌─────────────────────────────────────┐
|
||||
│ Next.js Frontend (Port 3000) │
|
||||
│ / /search /mail/[id] /admin │
|
||||
│ /imap │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Komponenten:**
|
||||
|
||||
| Komponente | Technologie | Beschreibung |
|
||||
|------------|-------------|--------------|
|
||||
| Backend | Go 1.23 | REST API, SMTP-Daemon, IMAP-Importer, Storage, Indexierung |
|
||||
| Frontend | Next.js 16 (TypeScript) | Web-Oberfläche, Tailwind CSS, shadcn/ui |
|
||||
| Datenbank | PostgreSQL | Metadaten, Benutzer, Audit-Log, Session-Blacklist |
|
||||
| Volltextsuche | Xapian | BM25-Ranking, Wildcards, Feldpräfixe |
|
||||
| Speicherung | Dateisystem | AES-256-GCM verschlüsselt, SHA-256 Integrität |
|
||||
|
||||
---
|
||||
|
||||
## Voraussetzungen
|
||||
|
||||
**Server:**
|
||||
- Debian 12 / Ubuntu 22.04 oder neuer
|
||||
- Go ≥ 1.23 (mit CGO für Xapian)
|
||||
- Node.js ≥ 20
|
||||
- PostgreSQL ≥ 14
|
||||
- libxapian-dev
|
||||
|
||||
**Ports (Standard):**
|
||||
- `8080` – HTTP API (Backend)
|
||||
- `3000` – Web-Frontend
|
||||
- `2525` – SMTP-Eingang (BCC-Journaling)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
### Automatisch (empfohlen)
|
||||
|
||||
```bash
|
||||
# Systembenutzer anlegen
|
||||
useradd -r -s /sbin/nologin archivmail
|
||||
|
||||
# Verzeichnisse anlegen
|
||||
mkdir -p /etc/archivmail
|
||||
mkdir -p /var/archivmail/store
|
||||
mkdir -p /var/archivmail/astore
|
||||
mkdir -p /var/lib/archivmail/xapian
|
||||
mkdir -p /var/log/archivmail
|
||||
|
||||
# Berechtigungen setzen
|
||||
chown -R archivmail:archivmail /var/archivmail /var/lib/archivmail /var/log/archivmail
|
||||
|
||||
# Encryption Key generieren
|
||||
openssl rand -base64 32 > /etc/archivmail/keyfile
|
||||
chmod 400 /etc/archivmail/keyfile
|
||||
chown archivmail:archivmail /etc/archivmail/keyfile
|
||||
|
||||
# Konfiguration
|
||||
cp config.example.yml /etc/archivmail/config.yml
|
||||
# config.yml anpassen (Datenbank, SMTP-Port, etc.)
|
||||
|
||||
# Systemd-Service aktivieren
|
||||
cp archivmail.service /lib/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now archivmail
|
||||
curl -fsSL https://gitea.perlbach24.de/scripte/archivmail/raw/branch/main/update.sh | bash
|
||||
```
|
||||
|
||||
## Verzeichnisstruktur
|
||||
Das Skript:
|
||||
1. Klont/aktualisiert den Quellcode
|
||||
2. Baut Backend (Go + CGO/Xapian) und Frontend (Next.js)
|
||||
3. Stoppt Dienste, spielt Binaries ein, startet Dienste neu
|
||||
4. Führt Datenbankmigrationen durch
|
||||
|
||||
```
|
||||
/etc/archivmail/
|
||||
├── config.yml # Hauptkonfiguration
|
||||
└── keyfile # AES-256-GCM Key (chmod 400)
|
||||
### Manuell
|
||||
|
||||
/var/archivmail/
|
||||
├── store/ # Mailkörper (.m, verschlüsselt)
|
||||
│ └── <server_id>/<customer_id>/<hash>/xxxxx.m
|
||||
└── astore/ # Anhänge dedupliziert (verschlüsselt)
|
||||
└── <hash>
|
||||
```bash
|
||||
# Quellcode
|
||||
git clone https://gitea.perlbach24.de/scripte/archivmail.git /opt/archivmail/_build
|
||||
cd /opt/archivmail/_build
|
||||
|
||||
/var/lib/archivmail/
|
||||
└── xapian/ # Volltext-Index
|
||||
# Backend bauen
|
||||
CGO_ENABLED=1 go build -tags xapian -buildvcs=false \
|
||||
-o /opt/archivmail/bin/archivmail ./cmd/archivmail/
|
||||
|
||||
/var/log/archivmail/
|
||||
└── audit.log # Audit-Log (JSON Lines, append-only)
|
||||
# Frontend bauen
|
||||
npm ci && npm run build
|
||||
rsync -a --delete .next/standalone/ /opt/archivmail/web/
|
||||
rsync -a --delete .next/static/ /opt/archivmail/web/.next/static/
|
||||
```
|
||||
|
||||
## Standard-Zugangsdaten
|
||||
### Konfigurationsdatei anlegen
|
||||
|
||||
> **Wichtig:** Passwörter nach dem ersten Login ändern!
|
||||
```bash
|
||||
mkdir -p /etc/archivmail /var/archivmail/{store,astore,xapian} /var/log/archivmail
|
||||
|
||||
| Benutzer | Passwort | Rolle |
|
||||
|----------|----------|-------|
|
||||
| `admin@archivmail` | `archivmailrockz` | Admin (Konfiguration, Nutzerverwaltung) |
|
||||
| `auditor@archivmail` | `archivmailrockz` | Auditor (alle E-Mails + Audit-Log) |
|
||||
cat > /etc/archivmail/config.yml << 'EOF'
|
||||
server:
|
||||
api_port: 8080
|
||||
smtp_port: 2525
|
||||
|
||||
### Rollenübersicht
|
||||
database:
|
||||
host: 127.0.0.1
|
||||
port: 5432
|
||||
name: archivmail
|
||||
user: archivmail
|
||||
password: SICHERES_PASSWORT
|
||||
sslmode: disable
|
||||
|
||||
| Rolle | E-Mails (eigene) | E-Mails (alle) | Audit-Log | Konfiguration |
|
||||
|-------|:-:|:-:|:-:|:-:|
|
||||
| `user` | ✅ | ❌ | ❌ | ❌ |
|
||||
| `auditor` | ✅ | ✅ | ✅ | ❌ |
|
||||
| `admin` | ❌ | ❌ | ❌ | ✅ |
|
||||
storage:
|
||||
store_path: /var/archivmail/store
|
||||
astore_path: /var/archivmail/astore
|
||||
xapian_path: /var/archivmail/xapian
|
||||
keyfile: /etc/archivmail/keyfile
|
||||
|
||||
## Technologie
|
||||
audit:
|
||||
log_path: /var/log/archivmail/audit.log
|
||||
retention_days: 0
|
||||
|
||||
| Komponente | Technologie |
|
||||
|-----------|-------------|
|
||||
| Backend | Go |
|
||||
| Datenbank | PostgreSQL |
|
||||
| Volltext-Index | Xapian |
|
||||
| Verschlüsselung | AES-256-GCM |
|
||||
| Webserver | eingebettet (Go net/http) |
|
||||
smtp:
|
||||
enabled: true
|
||||
bind: ":2525"
|
||||
domain: "archivmail.firma.de"
|
||||
allowed_ips:
|
||||
- 127.0.0.1
|
||||
- 192.168.1.0/24 # Postfix-Server-IP(s)
|
||||
|
||||
## Lizenz
|
||||
api:
|
||||
bind: ":8080"
|
||||
secret: ZUFAELLIGER_JWT_SECRET_MINDESTENS_32_ZEICHEN
|
||||
|
||||
Proprietär
|
||||
index:
|
||||
path: /var/archivmail/xapian
|
||||
backend: xapian
|
||||
batch_size: 100
|
||||
async_queue_size: 1000
|
||||
EOF
|
||||
|
||||
# AES-256 Schlüssel generieren
|
||||
dd if=/dev/urandom bs=32 count=1 > /etc/archivmail/keyfile
|
||||
chmod 600 /etc/archivmail/keyfile
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Konfiguration
|
||||
|
||||
### Vollständige Konfigurationsreferenz
|
||||
|
||||
```yaml
|
||||
server:
|
||||
api_port: 8080 # HTTP-API Port
|
||||
smtp_port: 2525 # SMTP-Eingang Port (Fallback wenn smtp.bind nicht gesetzt)
|
||||
|
||||
database:
|
||||
host: 127.0.0.1
|
||||
port: 5432
|
||||
name: archivmail
|
||||
user: archivmail
|
||||
password: ""
|
||||
sslmode: disable # disable | require | verify-full
|
||||
|
||||
storage:
|
||||
store_path: /var/archivmail/store # Haupt-Mailspeicher (AES-256-GCM)
|
||||
astore_path: /var/archivmail/astore # Anhang-Speicher
|
||||
xapian_path: /var/archivmail/xapian # Xapian-Datenbankpfad (auch in index.path)
|
||||
keyfile: /etc/archivmail/keyfile # 32-Byte AES-Schlüsseldatei
|
||||
|
||||
smtp:
|
||||
enabled: true
|
||||
bind: ":2525" # TCP-Adresse zum Lauschen
|
||||
domain: "archivmail" # EHLO-Domainname
|
||||
tls_cert: "" # Pfad zum TLS-Zertifikat (leer = kein TLS)
|
||||
tls_key: "" # Pfad zum TLS-Schlüssel
|
||||
max_size_mb: 50 # Maximale E-Mail-Größe in MB
|
||||
allowed_ips: # Nur diese IPs dürfen zustellen
|
||||
- 127.0.0.1
|
||||
|
||||
api:
|
||||
bind: ":8080"
|
||||
secret: "" # JWT-Signaturschlüssel (mind. 32 Zeichen, zufällig)
|
||||
|
||||
index:
|
||||
path: /var/archivmail/xapian
|
||||
backend: xapian # xapian (einziger unterstützter Backend)
|
||||
batch_size: 100 # Dokumente pro Index-Batch
|
||||
async_queue_size: 1000 # Größe der asynchronen Index-Queue
|
||||
|
||||
audit:
|
||||
log_path: /var/log/archivmail/audit.log
|
||||
retention_days: 0 # 0 = unbegrenzt
|
||||
|
||||
logging:
|
||||
path: "" # leer = stdout
|
||||
level: info # debug | info | warn | error
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Systemd-Dienste
|
||||
|
||||
Das System läuft mit zwei Systemd-Units:
|
||||
|
||||
| Dienst | Beschreibung | Port |
|
||||
|--------|--------------|------|
|
||||
| `archivmail` | Go-Backend (API + SMTP + IMAP-Scheduler) | 8080, 2525 |
|
||||
| `archivmail-web` | Next.js-Frontend | 3000 |
|
||||
|
||||
```bash
|
||||
# Status prüfen
|
||||
systemctl status archivmail archivmail-web
|
||||
|
||||
# Logs
|
||||
journalctl -u archivmail -f
|
||||
journalctl -u archivmail-web -f
|
||||
|
||||
# Neustart
|
||||
systemctl restart archivmail archivmail-web
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Funktionen im Detail
|
||||
|
||||
### Authentifizierung & Rollen
|
||||
|
||||
**Rollen:**
|
||||
|
||||
| Rolle | Rechte |
|
||||
|-------|--------|
|
||||
| `admin` | Vollzugriff: Benutzer verwalten, IMAP einrichten, Dienste steuern, alles suchen/exportieren |
|
||||
| `auditor` | Audit-Log lesen, alle E-Mails suchen und exportieren |
|
||||
| `user` | Nur eigene E-Mails suchen und lesen (Matching auf E-Mail-Adresse) |
|
||||
|
||||
**Sicherheit:**
|
||||
- Passwörter mit bcrypt (Cost 12)
|
||||
- Sessions als httpOnly SameSite=Strict Cookie (`archivmail_session`)
|
||||
- JWT mit zufälligem JTI (kryptografische Entropie, 16 Byte)
|
||||
- Token-Blacklist in PostgreSQL (Logout invalidiert Token sofort)
|
||||
- Rate-Limiting: max. 5 Fehlversuche in 15 Minuten → HTTP 429
|
||||
- Login-Versuche in `login_attempts`-Tabelle protokolliert
|
||||
- Letzter Login wird gespeichert (`last_login_at`)
|
||||
|
||||
**Erstmalige Einrichtung:**
|
||||
|
||||
Beim ersten Start werden automatisch zwei Benutzer mit zufälligen Passwörtern angelegt:
|
||||
|
||||
```
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ ARCHIVMAIL — ERSTMALIGE EINRICHTUNG ║
|
||||
║ Initiale Zugangsdaten (NUR EINMAL ANGEZEIGT): ║
|
||||
║ admin : <zufälliges Passwort> ║
|
||||
║ auditor : <zufälliges Passwort> ║
|
||||
║ Passwörter sofort nach dem ersten Login ändern! ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### SMTP-Eingang (BCC-Journaling)
|
||||
|
||||
Der eingebettete SMTP-Daemon empfängt E-Mails von Postfix (oder anderem MTA) über BCC-Weiterleitung.
|
||||
|
||||
**Funktionsweise:**
|
||||
```
|
||||
Absender → Postfix → Empfänger
|
||||
│
|
||||
└── always_bcc → archivmail SMTP (Port 2525)
|
||||
│
|
||||
▼
|
||||
Speicherung + Indexierung
|
||||
```
|
||||
|
||||
**Postfix-Konfiguration (`/etc/postfix/main.cf`):**
|
||||
```
|
||||
# Alle ausgehenden Mails archivieren:
|
||||
always_bcc = archiv@archivmail-host
|
||||
|
||||
# Oder granular per Nutzer:
|
||||
sender_bcc_maps = hash:/etc/postfix/sender_bcc
|
||||
```
|
||||
|
||||
**Sicherheit:**
|
||||
- Kein SMTP AUTH – Vertrauen ausschließlich über IP-Allowlist
|
||||
- IP-Allowlist in `smtp.allowed_ips` konfigurieren
|
||||
- 250 OK erst nach erfolgreicher Speicherung (kein Datenverlust)
|
||||
- 250 OK auch bei Duplikaten (Postfix stellt nicht erneut zu)
|
||||
- Maximale E-Mail-Größe konfigurierbar (`smtp.max_size_mb`, Standard: 50 MB)
|
||||
- Optionale TLS/STARTTLS-Unterstützung (`smtp.tls_cert` / `smtp.tls_key`)
|
||||
|
||||
**API-Endpunkt:**
|
||||
```
|
||||
GET /api/admin/smtp/status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### IMAP-Import & Auto-Sync
|
||||
|
||||
E-Mails von IMAP-Postfächern importieren – einmalig (Altbestände) oder automatisch als Hintergrundjob.
|
||||
|
||||
**Verbindung einrichten (Web-UI unter `/imap`):**
|
||||
|
||||
| Feld | Beschreibung |
|
||||
|------|--------------|
|
||||
| Name | Bezeichnung der Verbindung |
|
||||
| Host / Port | IMAP-Server-Adresse |
|
||||
| TLS | `ssl` (IMAPS), `starttls`, `none` |
|
||||
| Benutzername / Passwort | IMAP-Zugangsdaten (Passwort AES-256-GCM verschlüsselt in DB) |
|
||||
| Ausgeschlossene Ordner | Ordner die nicht importiert werden |
|
||||
|
||||
**Sync-Intervall:**
|
||||
|
||||
| Wert | Verhalten |
|
||||
|------|-----------|
|
||||
| 0 | Automatischer Sync deaktiviert |
|
||||
| 5–1440 | Sync alle N Minuten (Hintergrund-Goroutine) |
|
||||
|
||||
**Inkrementeller Sync:**
|
||||
- UID-basiert: Nur neue Nachrichten seit letztem Sync werden heruntergeladen
|
||||
- Exponentielles Backoff bei Fehlern: 1s → 60s → 300s (3 Versuche)
|
||||
- Duplikat-Erkennung via Content-Hash
|
||||
|
||||
**API-Endpunkte:**
|
||||
```
|
||||
GET /api/imap # Verbindungen auflisten
|
||||
POST /api/imap # Neue Verbindung anlegen
|
||||
DELETE /api/imap/{id} # Verbindung löschen
|
||||
PATCH /api/imap/{id} # Sync-Intervall ändern
|
||||
POST /api/imap/test # Verbindung testen + Ordner auflisten
|
||||
POST /api/imap/{id}/import # Vollständigen Import starten
|
||||
GET /api/imap/{id}/progress # Import-Fortschritt abfragen
|
||||
POST /api/imap/{id}/sync # Manuellen inkrementellen Sync auslösen
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### EML/MBOX Web-Upload
|
||||
|
||||
E-Mails direkt über die Admin-Oberfläche hochladen (Tab „Import" unter `/admin`).
|
||||
|
||||
**Unterstützte Formate:**
|
||||
- `.eml` – einzelne RFC-2822 E-Mail
|
||||
- `.mbox` – MBOX-Datei mit beliebig vielen E-Mails
|
||||
|
||||
**Ablauf:**
|
||||
1. Dateien per Drag-and-Drop oder Datei-Dialog hochladen
|
||||
2. Backend startet Hintergrund-Import-Job, gibt sofort `job_id` zurück
|
||||
3. Frontend pollt alle 1,5 Sekunden den Fortschritt
|
||||
4. Abschlussbericht: Importiert / Übersprungen (Duplikate) / Fehler
|
||||
|
||||
**API-Endpunkte:**
|
||||
```
|
||||
POST /api/admin/upload # Multipart-Upload starten
|
||||
GET /api/admin/upload/{jobID}/progress # Fortschritt abfragen
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Speicherung & Verschlüsselung
|
||||
|
||||
Alle E-Mails werden verschlüsselt auf dem Dateisystem gespeichert.
|
||||
|
||||
**Verschlüsselung:**
|
||||
- Algorithmus: AES-256-GCM (authentifizierte Verschlüsselung)
|
||||
- Schlüssel: 32-Byte-Datei (`keyfile` in Konfiguration)
|
||||
- Nonce: 12 Byte, kryptografisch zufällig, pro E-Mail neu generiert
|
||||
- Dateiformat: `[12-Byte Nonce][verschlüsselte Daten]`
|
||||
|
||||
**Datei-ID / Integrität:**
|
||||
- Jede E-Mail erhält als ID den SHA-256-Hash des Klartexts
|
||||
- Duplikat-Erkennung: gleicher Hash → gleiche Datei, Speicherung übersprungen
|
||||
- ID dient gleichzeitig als Integritätsnachweis
|
||||
|
||||
**PostgreSQL-Metadaten (`emails`-Tabelle):**
|
||||
|
||||
| Spalte | Typ | Beschreibung |
|
||||
|--------|-----|--------------|
|
||||
| `id` | TEXT PK | SHA-256 Hash des E-Mail-Rohtexts |
|
||||
| `from_addr` | TEXT | Absender |
|
||||
| `to_addrs` | TEXT[] | Empfänger |
|
||||
| `subject` | TEXT | Betreff |
|
||||
| `message_id` | TEXT | RFC-2822 Message-ID |
|
||||
| `mail_date` | TIMESTAMPTZ | Sendedatum |
|
||||
| `size_bytes` | INTEGER | Größe in Bytes |
|
||||
| `has_attachments` | BOOLEAN | Hat Anhänge |
|
||||
| `stored_at` | TIMESTAMPTZ | Archivierungszeitpunkt |
|
||||
| `indexed_at` | TIMESTAMPTZ | Indexierungszeitpunkt (NULL = ausstehend) |
|
||||
| `verify_ok` | BOOLEAN | Letztes Integritätsprüfungsergebnis |
|
||||
| `verified_at` | TIMESTAMPTZ | Zeitpunkt der letzten Prüfung |
|
||||
|
||||
**Async Index Worker:**
|
||||
- Eingehende E-Mails werden sofort gespeichert
|
||||
- Indexierung erfolgt asynchron über einen Go-Channel (Queue-Größe: 1000)
|
||||
- Backfill beim Start: Nicht indexierte E-Mails werden nachindexiert
|
||||
|
||||
---
|
||||
|
||||
### Volltext-Suche
|
||||
|
||||
Suche über alle archivierten E-Mails per Xapian-Index unter `/search`.
|
||||
|
||||
**Suchfelder:**
|
||||
- Freitext (durchsucht Betreff, Body, Absender, Empfänger, Anhangsnamen)
|
||||
- Absender-Filter (`from`)
|
||||
- Empfänger-Filter (`to`)
|
||||
- Zeitraum (`date_from` / `date_to`, ISO 8601: `2024-01-01`)
|
||||
- Nur Mails mit Anhängen (`has_attachment`)
|
||||
- Sortierung: `date_desc` (Standard), `date_asc`, `relevance`
|
||||
|
||||
**Xapian-Suchsyntax:**
|
||||
```
|
||||
Rechnung AND 2024 # UND-Verknüpfung
|
||||
"Angebot Projekt X" # Phrase
|
||||
Rechn* # Wildcard
|
||||
from:chef@firma.de # Feldpräfix
|
||||
subject:Urlaubsantrag # Feldpräfix
|
||||
```
|
||||
|
||||
**Paginierung:** 25 Ergebnisse pro Seite (konfigurierbar via `page_size`)
|
||||
|
||||
**Rollenfilter:**
|
||||
- `user`: sieht nur E-Mails an/von der eigenen E-Mail-Adresse
|
||||
- `admin` / `auditor`: sieht alle E-Mails
|
||||
|
||||
**API-Endpunkt:**
|
||||
```
|
||||
GET /api/search?q=...&from=...&to=...&date_from=...&date_to=...&has_attachment=true&sort=date_desc&page=1&page_size=25
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### E-Mail-Ansicht
|
||||
|
||||
Vollständige Darstellung einer archivierten E-Mail unter `/mail/[id]`.
|
||||
|
||||
**Funktionen:**
|
||||
- HTML-Body original dargestellt in `<iframe sandbox="allow-same-origin">` – JavaScript aus der Mail wird blockiert
|
||||
- Externe Inhalte (Bilder, Tracking-Pixel) standardmäßig blockiert, per Klick freischaltbar
|
||||
- Fallback auf Plaintext wenn kein HTML vorhanden
|
||||
- Anhänge einzeln herunterladbar
|
||||
- Originale E-Mail-Header aufklappbar
|
||||
- Download als `.eml` (Rohdatei)
|
||||
- Export als `.pdf`
|
||||
- Integrity-Badge: ✅ verifiziert OK / ⬜ noch nicht geprüft / ❌ Integritätsfehler
|
||||
|
||||
**API-Endpunkte:**
|
||||
```
|
||||
GET /api/mails/{id} # E-Mail-Details inkl. Body
|
||||
GET /api/mails/{id}/attachments/{index} # Anhang herunterladen
|
||||
GET /api/mails/{id}/raw # Rohe EML-Datei
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### E-Mail-Export
|
||||
|
||||
**Formate:**
|
||||
|
||||
| Format | Endpunkt | Beschreibung |
|
||||
|--------|----------|--------------|
|
||||
| PDF | `GET /api/export/pdf/{id}` | Einzelne Mail als PDF |
|
||||
| EML | `GET /api/mails/{id}/raw` | Einzelne Mail als .eml |
|
||||
| ZIP | `POST /api/export/zip` | Mehrere Mails + optional Anhänge |
|
||||
|
||||
**ZIP-Export:**
|
||||
```json
|
||||
POST /api/export/zip
|
||||
{
|
||||
"ids": ["sha256hash1", "sha256hash2"],
|
||||
"attachments": true
|
||||
}
|
||||
```
|
||||
|
||||
ZIP enthält:
|
||||
- Eine `.eml`-Datei pro E-Mail
|
||||
- `manifest.csv` mit Metadaten (ID, Absender, Empfänger, Betreff, Datum, Größe)
|
||||
- Anhänge wenn `attachments: true`
|
||||
|
||||
**Limits:**
|
||||
- Max. 500 Mails pro ZIP-Export
|
||||
- Jeder Export wird im Audit-Log erfasst
|
||||
|
||||
---
|
||||
|
||||
### Admin-Dashboard
|
||||
|
||||
Erreichbar unter `/admin` (nur Admins).
|
||||
|
||||
**Tabs:**
|
||||
|
||||
| Tab | Inhalt |
|
||||
|-----|--------|
|
||||
| **Dashboard** | CPU-Last (1/5/15 min), RAM, Festplatten, API-Status, SMTP-Status, Archiv-Statistiken, erste/letzte archivierte Mail |
|
||||
| **Dienste** | Systemd-Dienste starten / stoppen / neustarten, externe Verbindungen sperren/freigeben |
|
||||
| **Benutzer** | Benutzer anlegen, Rollen ändern, Passwort zurücksetzen, sperren/freischalten, löschen |
|
||||
| **Audit-Log** | Alle Audit-Ereignisse mit Paginierung (25/Seite) |
|
||||
| **Import** | EML/MBOX-Dateien hochladen, Fortschritt und Abschlussbericht |
|
||||
| **Module** | Übersicht aller Features mit Deployment-Status |
|
||||
|
||||
**Dashboard API-Endpunkte:**
|
||||
```
|
||||
GET /api/admin/system/stats # CPU, RAM, Disks, erste/letzte Mail
|
||||
GET /api/admin/smtp/status # SMTP-Daemon Status und Statistiken
|
||||
GET /api/admin/storage/stats # Anzahl archivierter Mails und Gesamtgröße
|
||||
GET /api/admin/services # Systemd-Dienste
|
||||
POST /api/admin/services/{name}/action # start | stop | restart | enable | disable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Audit-Log
|
||||
|
||||
Alle sicherheitsrelevanten Ereignisse werden lückenlos protokolliert.
|
||||
|
||||
**Erfasste Ereignisse:**
|
||||
|
||||
| Ereignis | Beschreibung |
|
||||
|----------|--------------|
|
||||
| `login_ok` | Erfolgreicher Login (Benutzer, IP) |
|
||||
| `login_fail` | Fehlgeschlagener Login (Benutzer, IP) |
|
||||
| `logout` | Logout |
|
||||
| `search` | Suchanfrage (Suchbegriff, Benutzer) |
|
||||
| `export_pdf` | PDF-Export einer E-Mail |
|
||||
| `export_zip` | ZIP-Export (Anzahl Mails) |
|
||||
| `user_create` | Neuer Benutzer angelegt |
|
||||
| `user_update` | Benutzerdaten geändert |
|
||||
| `user_delete` | Benutzer gelöscht |
|
||||
| `imap_import` | IMAP-Import gestartet/abgeschlossen |
|
||||
|
||||
**Speicherung:** Parallel in PostgreSQL-Tabelle und Flat-File (`audit.log_path`)
|
||||
|
||||
**API:**
|
||||
```
|
||||
GET /api/audit?page=1&page_size=25&username=admin&event_type=login_ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Integritätsprüfung
|
||||
|
||||
Jede archivierte E-Mail wird regelmäßig auf Unverändertheit geprüft.
|
||||
|
||||
**Funktionsweise:**
|
||||
1. Beim Speichern: SHA-256-Hash des Klartexts = Datei-ID
|
||||
2. Beim Prüfen: Datei laden → entschlüsseln → SHA-256 berechnen → mit ID vergleichen
|
||||
3. Bei Abweichung: Warnung im Log + `verify_ok = false` in DB
|
||||
|
||||
**Hintergrundprüfung:**
|
||||
- Läuft beim Start einmal sofort durch
|
||||
- Danach alle 5 Minuten wiederholt
|
||||
- Ergebnisse in `emails`-Tabelle: `verify_ok`, `verified_at`
|
||||
|
||||
**Ergebnis in der E-Mail-Ansicht:**
|
||||
```
|
||||
✅ Integrität OK (geprüft: 2024-03-15 14:22:00)
|
||||
❌ INTEGRITÄTSFEHLER – E-Mail wurde möglicherweise verändert
|
||||
⬜ Noch nicht geprüft
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### CLI Import & Export
|
||||
|
||||
Das `archivmail`-Binary kann auch ohne laufenden Daemon als CLI-Tool eingesetzt werden.
|
||||
|
||||
**Import:**
|
||||
```bash
|
||||
# Einzelne EML-Datei
|
||||
archivmail import --file /pfad/zur/datei.eml
|
||||
|
||||
# MBOX-Datei
|
||||
archivmail import --file /pfad/zur/datei.mbox
|
||||
|
||||
# Verzeichnis (alle EML/MBOX-Dateien)
|
||||
archivmail import --dir /pfad/zum/verzeichnis
|
||||
|
||||
# Rekursiv
|
||||
archivmail import --dir /pfad/ --recursive
|
||||
|
||||
# Simulation (kein Speichern)
|
||||
archivmail import --file datei.eml --dry-run
|
||||
|
||||
# JSON-Ausgabe für Skripting
|
||||
archivmail import --dir /pfad/ --json
|
||||
```
|
||||
|
||||
**Export:**
|
||||
```bash
|
||||
# Alle Mails als EML in Verzeichnis
|
||||
archivmail export --out /export/
|
||||
|
||||
# Als MBOX
|
||||
archivmail export --out /export/archiv.mbox --format mbox
|
||||
|
||||
# Mit Filtern
|
||||
archivmail export --out /export/ \
|
||||
--from absender@firma.de \
|
||||
--date-from 2024-01-01 \
|
||||
--date-to 2024-12-31 \
|
||||
--query "Rechnung"
|
||||
|
||||
# Vorhandene Dateien überschreiben
|
||||
archivmail export --out /export/ --force
|
||||
|
||||
# JSON-Ausgabe
|
||||
archivmail export --out /export/ --json
|
||||
```
|
||||
|
||||
**Alle Befehle:**
|
||||
```
|
||||
archivmail serve Daemon starten (Standard)
|
||||
archivmail import EML/MBOX importieren
|
||||
archivmail import-piler Aus mailpiler migrieren
|
||||
archivmail export Exportieren
|
||||
archivmail version Version anzeigen
|
||||
archivmail help Hilfe
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Mailpiler Migration
|
||||
|
||||
Vollständige Migration eines bestehenden mailpiler-Archivs nach archivmail.
|
||||
|
||||
**Methode 1: `pilerexport` (empfohlen)**
|
||||
|
||||
Erfordert das mailpiler-Tool `pilerexport` auf dem Server. Verarbeitet Entschlüsselung und Dekomprimierung automatisch.
|
||||
|
||||
```bash
|
||||
# Alle Mails migrieren
|
||||
archivmail import-piler --config /etc/archivmail/config.yml
|
||||
|
||||
# Mit Datumsfilter
|
||||
archivmail import-piler --date-from 2020-01-01 --date-to 2024-12-31
|
||||
|
||||
# Benutzerdefinierter Export-Ordner (bleibt erhalten)
|
||||
archivmail import-piler --export-dir /mnt/migration/export
|
||||
```
|
||||
|
||||
**Methode 2: Direkt (ohne laufendes mailpiler)**
|
||||
|
||||
Liest `.m`-Dateien direkt aus dem mailpiler Store-Verzeichnis und verarbeitet sie ohne dass mailpiler installiert sein muss.
|
||||
|
||||
```bash
|
||||
archivmail import-piler \
|
||||
--method direct \
|
||||
--store-dir /var/piler/store \
|
||||
--key-file /var/piler/store/piler.key
|
||||
```
|
||||
|
||||
Unterstützt:
|
||||
- AES-256-CBC Entschlüsselung (IV in ersten 16 Bytes der Datei)
|
||||
- zlib Dekomprimierung
|
||||
- Mehrfache Fallback-Versuche für verschiedene mailpiler-Versionen
|
||||
- Unkryptierte/unkomprimierte Dateien als letzter Fallback
|
||||
|
||||
**Auto-Modus (Standard):**
|
||||
|
||||
`--method auto` versucht zuerst `pilerexport`, fällt auf `direct` zurück:
|
||||
|
||||
```bash
|
||||
archivmail import-piler --config /etc/archivmail/config.yml
|
||||
```
|
||||
|
||||
**Alle Optionen:**
|
||||
```
|
||||
--config archivmail Konfigurationsdatei
|
||||
--method auto | pilerexport | direct (Standard: auto)
|
||||
--pilerexport Pfad zum pilerexport Binary (auto-erkennung)
|
||||
--export-dir Ausgabeverzeichnis für pilerexport
|
||||
--store-dir mailpiler Store-Verzeichnis (Standard: /var/piler/store)
|
||||
--key-file mailpiler AES-Schlüsseldatei (Standard: /var/piler/store/piler.key)
|
||||
--date-from Export ab Datum YYYY-MM-DD (pilerexport-Methode)
|
||||
--date-to Export bis Datum YYYY-MM-DD (pilerexport-Methode)
|
||||
--dry-run Simulation ohne Speichern
|
||||
--json JSON-Ausgabe
|
||||
```
|
||||
|
||||
**Typischer Migrationsablauf:**
|
||||
```bash
|
||||
# 1. Probe-Lauf
|
||||
archivmail import-piler --dry-run --json
|
||||
|
||||
# 2. Migration mit Protokoll
|
||||
archivmail import-piler 2>&1 | tee /var/log/archivmail/migration.log
|
||||
|
||||
# 3. Ergebnis prüfen
|
||||
tail -20 /var/log/archivmail/migration.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REST API
|
||||
|
||||
Alle Endpunkte erfordern eine gültige Session (Cookie `archivmail_session` oder `Authorization: Bearer <token>`).
|
||||
|
||||
### Authentifizierung
|
||||
|
||||
| Methode | Endpunkt | Beschreibung |
|
||||
|---------|----------|--------------|
|
||||
| POST | `/api/auth/login` | Login: `{"username":"...","password":"..."}` |
|
||||
| GET | `/api/auth/me` | Eingeloggter Benutzer |
|
||||
| POST | `/api/auth/logout` | Session invalidieren |
|
||||
|
||||
### Suche & Mails
|
||||
|
||||
| Methode | Endpunkt | Beschreibung |
|
||||
|---------|----------|--------------|
|
||||
| GET | `/api/search` | Volltext-Suche |
|
||||
| GET | `/api/mails/{id}` | E-Mail-Details |
|
||||
| GET | `/api/mails/{id}/raw` | Rohe EML |
|
||||
| GET | `/api/mails/{id}/attachments/{n}` | Anhang herunterladen |
|
||||
|
||||
### Export
|
||||
|
||||
| Methode | Endpunkt | Beschreibung |
|
||||
|---------|----------|--------------|
|
||||
| GET | `/api/export/pdf/{id}` | PDF-Export |
|
||||
| POST | `/api/export/zip` | ZIP-Export |
|
||||
|
||||
### Admin
|
||||
|
||||
| Methode | Endpunkt | Beschreibung |
|
||||
|---------|----------|--------------|
|
||||
| GET | `/api/users` | Benutzerliste |
|
||||
| POST | `/api/users` | Benutzer anlegen |
|
||||
| PATCH | `/api/users/{id}` | Benutzer ändern |
|
||||
| DELETE | `/api/users/{id}` | Benutzer löschen |
|
||||
| GET | `/api/audit` | Audit-Log |
|
||||
| GET | `/api/admin/smtp/status` | SMTP-Status |
|
||||
| GET | `/api/admin/storage/stats` | Speicher-Statistiken |
|
||||
| GET | `/api/admin/system/stats` | System-Ressourcen |
|
||||
| GET | `/api/admin/services` | Dienste-Status |
|
||||
| POST | `/api/admin/services/{name}/action` | Dienst steuern |
|
||||
| POST | `/api/admin/upload` | EML/MBOX hochladen |
|
||||
| GET | `/api/admin/upload/{jobID}/progress` | Upload-Fortschritt |
|
||||
|
||||
### IMAP
|
||||
|
||||
| Methode | Endpunkt | Beschreibung |
|
||||
|---------|----------|--------------|
|
||||
| GET | `/api/imap` | Verbindungen auflisten |
|
||||
| POST | `/api/imap` | Verbindung anlegen |
|
||||
| DELETE | `/api/imap/{id}` | Verbindung löschen |
|
||||
| PATCH | `/api/imap/{id}` | Sync-Intervall setzen |
|
||||
| POST | `/api/imap/test` | Verbindung testen |
|
||||
| POST | `/api/imap/{id}/import` | Vollständigen Import starten |
|
||||
| GET | `/api/imap/{id}/progress` | Import-Fortschritt |
|
||||
| POST | `/api/imap/{id}/sync` | Manuellen Sync auslösen |
|
||||
|
||||
---
|
||||
|
||||
## In Entwicklung
|
||||
|
||||
| Funktion | Beschreibung |
|
||||
|----------|--------------|
|
||||
| **Ordner- & Label-Verwaltung** | E-Mails in Ordner ablegen und mit Labels versehen |
|
||||
| **POP3-Import** | E-Mails von POP3-Servern importieren |
|
||||
| **REST API (extern)** | Vollständige API für CRM-Anbindung und externe Systeme |
|
||||
| **LDAP / Active Directory** | Benutzer-Authentifizierung über LDAP/AD statt lokaler Accounts |
|
||||
|
||||
---
|
||||
|
||||
## Update
|
||||
|
||||
```bash
|
||||
# Auf dem Server als root:
|
||||
bash /opt/archivmail/update.sh
|
||||
|
||||
# Oder direkt von Gitea:
|
||||
curl -fsSL https://gitea.perlbach24.de/scripte/archivmail/raw/branch/main/update.sh | bash
|
||||
```
|
||||
|
||||
Das Update-Skript führt automatisch durch:
|
||||
1. Quellcode aktualisieren (git pull)
|
||||
2. Backend neu bauen (CGO + Xapian)
|
||||
3. Frontend neu bauen (Next.js standalone)
|
||||
4. Dienste stoppen
|
||||
5. Binaries und Frontend einspielen
|
||||
6. Datenbankmigrationen durchführen
|
||||
7. Dienste starten und Status prüfen
|
||||
|
||||
@@ -262,11 +262,12 @@ func printHelp() {
|
||||
fmt.Printf(`archivmail %s – Mail-Archiv-Daemon und CLI
|
||||
|
||||
Commands:
|
||||
serve Daemon starten (Standard wenn kein Befehl angegeben)
|
||||
import E-Mails importieren (EML, MBOX, Verzeichnis)
|
||||
export E-Mails exportieren (EML, MBOX)
|
||||
version Version anzeigen
|
||||
help Diese Hilfe anzeigen
|
||||
serve Daemon starten (Standard wenn kein Befehl angegeben)
|
||||
import E-Mails importieren (EML, MBOX, Verzeichnis)
|
||||
import-piler Aus mailpiler migrieren (pilerexport oder direkte Store-Methode)
|
||||
export E-Mails exportieren (EML, MBOX)
|
||||
version Version anzeigen
|
||||
help Diese Hilfe anzeigen
|
||||
|
||||
archivmail import [flags]
|
||||
--config Pfad zur Konfigurationsdatei (Standard: /etc/archivmail/config.yml)
|
||||
@@ -276,6 +277,18 @@ archivmail import [flags]
|
||||
--dry-run Simulation ohne Speichern
|
||||
--json Maschinenlesbare JSON-Ausgabe
|
||||
|
||||
archivmail import-piler [flags]
|
||||
--config Pfad zur Konfigurationsdatei (Standard: /etc/archivmail/config.yml)
|
||||
--method auto | pilerexport | direct (Standard: auto)
|
||||
--pilerexport Pfad zum pilerexport Binary (auto-erkennung)
|
||||
--export-dir Ausgabeverzeichnis für pilerexport (temp wenn leer)
|
||||
--store-dir mailpiler Store-Verzeichnis (Standard: /var/piler/store)
|
||||
--key-file mailpiler AES-Schlüsseldatei (Standard: /var/piler/store/piler.key)
|
||||
--date-from Export ab Datum YYYY-MM-DD (pilerexport-Methode)
|
||||
--date-to Export bis Datum YYYY-MM-DD (pilerexport-Methode)
|
||||
--dry-run Simulation ohne Speichern
|
||||
--json Maschinenlesbare JSON-Ausgabe
|
||||
|
||||
archivmail export [flags]
|
||||
--config Pfad zur Konfigurationsdatei (Standard: /etc/archivmail/config.yml)
|
||||
--out Zielverzeichnis oder Zieldatei (Pflicht)
|
||||
|
||||
@@ -0,0 +1,426 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/zlib"
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/archivmail/config"
|
||||
"github.com/archivmail/internal/index"
|
||||
"github.com/archivmail/internal/storage"
|
||||
)
|
||||
|
||||
type pilerImportResult struct {
|
||||
Status string `json:"status"`
|
||||
Method string `json:"method"`
|
||||
Imported int `json:"imported"`
|
||||
Skipped int `json:"skipped"`
|
||||
Errors int `json:"errors"`
|
||||
DurationSec float64 `json:"duration_sec"`
|
||||
}
|
||||
|
||||
func runImportPiler(args []string) {
|
||||
fs := flag.NewFlagSet("import-piler", flag.ExitOnError)
|
||||
configPath := fs.String("config", "/etc/archivmail/config.yml", "archivmail config path")
|
||||
method := fs.String("method", "auto", "import method: auto | pilerexport | direct")
|
||||
pilerexpBin := fs.String("pilerexport", "", "path to pilerexport binary (auto-detect)")
|
||||
exportDir := fs.String("export-dir", "", "output dir for pilerexport (temp dir if empty)")
|
||||
storeDir := fs.String("store-dir", "/var/piler/store", "mailpiler store directory (direct method)")
|
||||
keyFile := fs.String("key-file", "/var/piler/store/piler.key", "mailpiler AES key file (direct method)")
|
||||
dateFrom := fs.String("date-from", "", "export from date YYYY-MM-DD (pilerexport method)")
|
||||
dateTo := fs.String("date-to", "", "export to date YYYY-MM-DD (pilerexport method)")
|
||||
dryRun := fs.Bool("dry-run", false, "simulate without saving")
|
||||
jsonOut := fs.Bool("json", false, "machine-readable JSON output")
|
||||
fs.Usage = func() {
|
||||
fmt.Fprintln(os.Stderr, "Usage: archivmail import-piler [flags]")
|
||||
fmt.Fprintln(os.Stderr, "")
|
||||
fmt.Fprintln(os.Stderr, "Flags:")
|
||||
fs.PrintDefaults()
|
||||
fmt.Fprintln(os.Stderr, "")
|
||||
fmt.Fprintln(os.Stderr, "Methods:")
|
||||
fmt.Fprintln(os.Stderr, " pilerexport Calls pilerexport, imports resulting EML files (recommended)")
|
||||
fmt.Fprintln(os.Stderr, " direct Reads .m files from mailpiler store, decrypts+decompresses")
|
||||
fmt.Fprintln(os.Stderr, " auto Tries pilerexport first, falls back to direct")
|
||||
}
|
||||
fs.Parse(args)
|
||||
|
||||
start := time.Now()
|
||||
|
||||
cfg, err := config.Load(*configPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: load config: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
storeCfg := storage.Config{
|
||||
Dir: cfg.Storage.StorePath,
|
||||
Keyfile: cfg.Storage.Keyfile,
|
||||
DSN: cfg.Database.DSN(),
|
||||
}
|
||||
mailStore, err := storage.New(storeCfg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: storage init: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer mailStore.Close()
|
||||
|
||||
batchSize := cfg.Index.BatchSize
|
||||
if batchSize <= 0 {
|
||||
batchSize = 100
|
||||
}
|
||||
backend := cfg.Index.Backend
|
||||
if backend == "" {
|
||||
backend = "xapian"
|
||||
}
|
||||
idx, err := index.New(cfg.Index.Path, batchSize, backend)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: index init: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer idx.Close()
|
||||
|
||||
// Resolve method
|
||||
resolvedMethod := *method
|
||||
if resolvedMethod == "auto" {
|
||||
if bin := resolvePilerexport(*pilerexpBin); bin != "" {
|
||||
resolvedMethod = "pilerexport"
|
||||
} else {
|
||||
resolvedMethod = "direct"
|
||||
}
|
||||
}
|
||||
|
||||
if !*jsonOut {
|
||||
fmt.Printf("Mailpiler → archivmail Migration\n")
|
||||
fmt.Printf("Methode: %s\n\n", resolvedMethod)
|
||||
}
|
||||
|
||||
var imported, skipped, errors int
|
||||
|
||||
switch resolvedMethod {
|
||||
case "pilerexport":
|
||||
imported, skipped, errors = runPilerexportMethod(
|
||||
mailStore, idx,
|
||||
*pilerexpBin, *exportDir, *dateFrom, *dateTo,
|
||||
*dryRun, *jsonOut,
|
||||
)
|
||||
case "direct":
|
||||
imported, skipped, errors = runDirectMethod(
|
||||
mailStore, idx,
|
||||
*storeDir, *keyFile,
|
||||
*dryRun, *jsonOut,
|
||||
)
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "error: unknown method %q (use: auto, pilerexport, direct)\n", resolvedMethod)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
result := pilerImportResult{
|
||||
Status: "done",
|
||||
Method: resolvedMethod,
|
||||
Imported: imported,
|
||||
Skipped: skipped,
|
||||
Errors: errors,
|
||||
DurationSec: time.Since(start).Seconds(),
|
||||
}
|
||||
|
||||
if *jsonOut {
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
enc.SetIndent("", " ")
|
||||
enc.Encode(result)
|
||||
} else {
|
||||
fmt.Printf("\n╔══════════════════════════════════════╗\n")
|
||||
fmt.Printf("║ Migration abgeschlossen ║\n")
|
||||
fmt.Printf("╠══════════════════════════════════════╣\n")
|
||||
fmt.Printf("║ Importiert: %-22d║\n", imported)
|
||||
fmt.Printf("║ Übersprungen: %-22d║\n", skipped)
|
||||
fmt.Printf("║ Fehler: %-22d║\n", errors)
|
||||
fmt.Printf("║ Dauer: %-19.1fs║\n", result.DurationSec)
|
||||
if *dryRun {
|
||||
fmt.Printf("║ [dry-run] Keine Daten gespeichert ║\n")
|
||||
}
|
||||
fmt.Printf("╚══════════════════════════════════════╝\n")
|
||||
}
|
||||
|
||||
if errors > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// ── pilerexport method ────────────────────────────────────────────────────
|
||||
|
||||
func runPilerexportMethod(mailStore *storage.Store, idx index.Indexer, binPath, exportDir, dateFrom, dateTo string, dryRun, jsonOut bool) (imported, skipped, errors int) {
|
||||
bin := resolvePilerexport(binPath)
|
||||
if bin == "" {
|
||||
fmt.Fprintln(os.Stderr, "error: pilerexport binary not found")
|
||||
fmt.Fprintln(os.Stderr, " Install mailpiler tools or use --method direct")
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Create temp dir if needed
|
||||
cleanupDir := false
|
||||
if exportDir == "" {
|
||||
tmp, err := os.MkdirTemp("", "archivmail-piler-*")
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: create temp dir: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
exportDir = tmp
|
||||
cleanupDir = true
|
||||
} else {
|
||||
if err := os.MkdirAll(exportDir, 0750); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: create export dir %s: %v\n", exportDir, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
if !jsonOut {
|
||||
fmt.Printf("Exportiere aus mailpiler nach %s ...\n", exportDir)
|
||||
}
|
||||
|
||||
// Build pilerexport command
|
||||
cmdArgs := []string{"-D", exportDir}
|
||||
if dateFrom != "" {
|
||||
cmdArgs = append(cmdArgs, "-f", dateFrom)
|
||||
}
|
||||
if dateTo != "" {
|
||||
cmdArgs = append(cmdArgs, "-t", dateTo)
|
||||
}
|
||||
|
||||
cmd := exec.Command(bin, cmdArgs...)
|
||||
cmd.Stdout = os.Stdout
|
||||
cmd.Stderr = os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error: pilerexport failed: %v\n", err)
|
||||
fmt.Fprintln(os.Stderr, " Ensure pilerexport has access to mailpiler database and store.")
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if !jsonOut {
|
||||
fmt.Println("pilerexport abgeschlossen. Importiere EML-Dateien...")
|
||||
}
|
||||
|
||||
// Import all EML files from the export directory
|
||||
total := 0
|
||||
filepath.WalkDir(exportDir, func(path string, d os.DirEntry, werr error) error {
|
||||
if werr != nil || d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
lower := strings.ToLower(d.Name())
|
||||
if !strings.HasSuffix(lower, ".eml") && !strings.HasSuffix(lower, ".m") {
|
||||
return nil
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
errors++
|
||||
return nil
|
||||
}
|
||||
total++
|
||||
result := importMessage(mailStore, idx, raw, dryRun)
|
||||
switch result {
|
||||
case "imported":
|
||||
imported++
|
||||
case "skipped":
|
||||
skipped++
|
||||
case "error":
|
||||
errors++
|
||||
}
|
||||
if !jsonOut && total%100 == 0 {
|
||||
fmt.Printf(" Fortschritt: %d (importiert: %d, übersprungen: %d, fehler: %d)\n",
|
||||
total, imported, skipped, errors)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if cleanupDir {
|
||||
os.RemoveAll(exportDir)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// ── direct method ─────────────────────────────────────────────────────────
|
||||
|
||||
func runDirectMethod(mailStore *storage.Store, idx index.Indexer, storeDir, keyFilePath string, dryRun, jsonOut bool) (imported, skipped, errors int) {
|
||||
if _, err := os.Stat(storeDir); os.IsNotExist(err) {
|
||||
fmt.Fprintf(os.Stderr, "error: store dir not found: %s\n", storeDir)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Try to read AES key
|
||||
var aesKey []byte
|
||||
if data, err := os.ReadFile(keyFilePath); err == nil && len(data) >= 32 {
|
||||
aesKey = data[:32]
|
||||
if !jsonOut {
|
||||
fmt.Printf("AES-Schlüssel geladen: %s\n", keyFilePath)
|
||||
}
|
||||
} else {
|
||||
if !jsonOut {
|
||||
fmt.Printf("Hinweis: Kein AES-Schlüssel geladen (%s) – versuche unkomprimiert/unkryptiert\n", keyFilePath)
|
||||
}
|
||||
}
|
||||
|
||||
if !jsonOut {
|
||||
fmt.Printf("Lese .m Dateien aus %s ...\n", storeDir)
|
||||
}
|
||||
|
||||
total := 0
|
||||
filepath.WalkDir(storeDir, func(path string, d os.DirEntry, werr error) error {
|
||||
if werr != nil || d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if !strings.HasSuffix(d.Name(), ".m") {
|
||||
return nil
|
||||
}
|
||||
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
errors++
|
||||
return nil
|
||||
}
|
||||
|
||||
// Decode the piler file format
|
||||
emailRaw, err := decodePilerFile(raw, aesKey)
|
||||
if err != nil || len(emailRaw) == 0 {
|
||||
errors++
|
||||
return nil
|
||||
}
|
||||
|
||||
total++
|
||||
result := importMessage(mailStore, idx, emailRaw, dryRun)
|
||||
switch result {
|
||||
case "imported":
|
||||
imported++
|
||||
case "skipped":
|
||||
skipped++
|
||||
case "error":
|
||||
errors++
|
||||
}
|
||||
if !jsonOut && total%100 == 0 {
|
||||
fmt.Printf(" Fortschritt: %d (importiert: %d, übersprungen: %d, fehler: %d)\n",
|
||||
total, imported, skipped, errors)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// decodePilerFile attempts to decode a mailpiler .m file.
|
||||
// mailpiler stores files as: [16-byte IV][AES-256-CBC encrypted zlib data]
|
||||
// If no key is provided, zlib decompression alone is attempted.
|
||||
func decodePilerFile(data []byte, aesKey []byte) ([]byte, error) {
|
||||
// If key is available, try AES-256-CBC decrypt first, then zlib decompress
|
||||
if len(aesKey) == 32 && len(data) > 16 {
|
||||
decrypted, err := aes256CBCDecrypt(data, aesKey)
|
||||
if err == nil {
|
||||
if decompressed, err := zlibDecompress(decrypted); err == nil {
|
||||
return decompressed, nil
|
||||
}
|
||||
// Maybe the decrypted data is already a raw email (no zlib)
|
||||
if looksLikeEmail(decrypted) {
|
||||
return decrypted, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Try zlib decompression at various offsets (no encryption, or wrong key)
|
||||
for _, skip := range []int{0, 4, 8, 12, 16} {
|
||||
if skip >= len(data) {
|
||||
break
|
||||
}
|
||||
if out, err := zlibDecompress(data[skip:]); err == nil && looksLikeEmail(out) {
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Try raw (uncompressed, unencrypted)
|
||||
if looksLikeEmail(data) {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("could not decode piler file")
|
||||
}
|
||||
|
||||
func aes256CBCDecrypt(data, key []byte) ([]byte, error) {
|
||||
if len(data) < aes.BlockSize {
|
||||
return nil, fmt.Errorf("data too short")
|
||||
}
|
||||
block, err := aes.NewCipher(key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
iv := data[:aes.BlockSize]
|
||||
ciphertext := data[aes.BlockSize:]
|
||||
if len(ciphertext)%aes.BlockSize != 0 {
|
||||
return nil, fmt.Errorf("ciphertext not aligned to block size")
|
||||
}
|
||||
plaintext := make([]byte, len(ciphertext))
|
||||
cipher.NewCBCDecrypter(block, iv).CryptBlocks(plaintext, ciphertext)
|
||||
// Remove PKCS7 padding
|
||||
if len(plaintext) == 0 {
|
||||
return nil, fmt.Errorf("empty plaintext")
|
||||
}
|
||||
pad := int(plaintext[len(plaintext)-1])
|
||||
if pad == 0 || pad > aes.BlockSize {
|
||||
return nil, fmt.Errorf("invalid PKCS7 padding")
|
||||
}
|
||||
return plaintext[:len(plaintext)-pad], nil
|
||||
}
|
||||
|
||||
func zlibDecompress(data []byte) ([]byte, error) {
|
||||
r, err := zlib.NewReader(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer r.Close()
|
||||
out, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// looksLikeEmail returns true if the data resembles an RFC 2822 email.
|
||||
func looksLikeEmail(data []byte) bool {
|
||||
if len(data) < 10 {
|
||||
return false
|
||||
}
|
||||
header := strings.ToLower(string(data[:min(512, len(data))]))
|
||||
return strings.Contains(header, "from:") ||
|
||||
strings.Contains(header, "date:") ||
|
||||
strings.Contains(header, "message-id:") ||
|
||||
strings.Contains(header, "subject:")
|
||||
}
|
||||
|
||||
func min(a, b int) int {
|
||||
if a < b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// resolvePilerexport returns the path to the pilerexport binary, or "" if not found.
|
||||
func resolvePilerexport(hint string) string {
|
||||
candidates := []string{hint, "pilerexport", "/usr/sbin/pilerexport", "/usr/local/sbin/pilerexport", "/opt/piler/bin/pilerexport"}
|
||||
for _, c := range candidates {
|
||||
if c == "" {
|
||||
continue
|
||||
}
|
||||
if path, err := exec.LookPath(c); err == nil {
|
||||
return path
|
||||
}
|
||||
if _, err := os.Stat(c); err == nil {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -32,6 +32,9 @@ func main() {
|
||||
case "import":
|
||||
runImport(os.Args[2:])
|
||||
return
|
||||
case "import-piler":
|
||||
runImportPiler(os.Args[2:])
|
||||
return
|
||||
case "export":
|
||||
runExport(os.Args[2:])
|
||||
return
|
||||
|
||||
+8
-7
@@ -14,15 +14,15 @@
|
||||
|----|---------|--------|------|---------|
|
||||
| PROJ-1 | Nutzer-Authentifizierung & Rollen (User/Admin) | Deployed | [PROJ-1](PROJ-1-authentifizierung-und-rollen.md) | 2026-03-12 |
|
||||
| PROJ-2 | E-Mail-Import: EML/MBOX Upload | In Progress | [PROJ-2](PROJ-2-import-eml-mbox.md) | 2026-03-12 |
|
||||
| PROJ-3 | E-Mail-Import: IMAP-Verbindung | In Progress | [PROJ-3](PROJ-3-import-imap.md) | 2026-03-12 |
|
||||
| PROJ-4 | E-Mail-Import: SMTP-Eingang via BCC (primär) | In Progress | [PROJ-4](PROJ-4-import-smtp.md) | 2026-03-12 |
|
||||
| PROJ-3 | E-Mail-Import: IMAP-Verbindung | Deployed | [PROJ-3](PROJ-3-import-imap.md) | 2026-03-12 |
|
||||
| PROJ-4 | E-Mail-Import: SMTP-Eingang via BCC (primär) | Deployed | [PROJ-4](PROJ-4-import-smtp.md) | 2026-03-12 |
|
||||
| PROJ-5 | E-Mail-Speicherung & Volltext-Indexierung | Deployed | [PROJ-5](PROJ-5-speicherung-und-indexierung.md) | 2026-03-12 |
|
||||
| PROJ-6 | Volltext-Suche & Filterung | In Review | [PROJ-6](PROJ-6-volltext-suche.md) | 2026-03-12 |
|
||||
| PROJ-7 | E-Mail-Ansicht (Lesen & Anhänge) | In Progress | [PROJ-7](PROJ-7-email-ansicht.md) | 2026-03-12 |
|
||||
| PROJ-6 | Volltext-Suche & Filterung | Deployed | [PROJ-6](PROJ-6-volltext-suche.md) | 2026-03-12 |
|
||||
| PROJ-7 | E-Mail-Ansicht (Lesen & Anhänge) | Deployed | [PROJ-7](PROJ-7-email-ansicht.md) | 2026-03-12 |
|
||||
| PROJ-8 | Automatischer IMAP-Sync (Cron-Job) | Deployed | [PROJ-8](PROJ-8-imap-auto-sync.md) | 2026-03-12 |
|
||||
| PROJ-9 | Ordner- & Label-Verwaltung | In Progress | [PROJ-9](PROJ-9-ordner-und-labels.md) | 2026-03-12 |
|
||||
| PROJ-10 | Admin-Bereich: Nutzer- & Postfachverwaltung | In Progress | [PROJ-10](PROJ-10-admin-bereich.md) | 2026-03-12 |
|
||||
| PROJ-11 | Audit-Log & Compliance-Berichte | In Progress | [PROJ-11](PROJ-11-audit-log.md) | 2026-03-12 |
|
||||
| PROJ-10 | Admin-Bereich: Nutzer- & Postfachverwaltung | Deployed | [PROJ-10](PROJ-10-admin-bereich.md) | 2026-03-12 |
|
||||
| PROJ-11 | Audit-Log & Compliance-Berichte | Deployed | [PROJ-11](PROJ-11-audit-log.md) | 2026-03-12 |
|
||||
| PROJ-12 | E-Mail-Export (EML/PDF) | Deployed | [PROJ-12](PROJ-12-export.md) | 2026-03-12 |
|
||||
| PROJ-13 | REST API für externe CRM-Anbindung | In Progress | [PROJ-13](PROJ-13-rest-api-crm.md) | 2026-03-13 |
|
||||
| PROJ-14 | E-Mail-Import: POP3-Verbindung | In Progress | [PROJ-14](PROJ-14-import-pop3.md) | 2026-03-13 |
|
||||
@@ -31,7 +31,8 @@
|
||||
|
||||
| PROJ-17 | Admin Dashboard – Systemauslastung & Archiv-Übersicht | Deployed | [PROJ-17](PROJ-17-system-dashboard.md) | 2026-03-14 |
|
||||
| PROJ-18 | E-Mail Integritätsprüfung | Deployed | [PROJ-18](PROJ-18-integritaetspruefung.md) | 2026-03-14 |
|
||||
| PROJ-19 | Mailpiler → archivmail Migrationstool | Deployed | [PROJ-19](PROJ-19-import-piler.md) | 2026-03-17 |
|
||||
|
||||
<!-- Add features above this line -->
|
||||
|
||||
## Next Available ID: PROJ-19
|
||||
## Next Available ID: PROJ-20
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# PROJ-10: Admin-Bereich: Nutzer- & Postfachverwaltung
|
||||
|
||||
## Status: In Progress
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-12
|
||||
**Last Updated:** 2026-03-12
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# PROJ-11: Audit-Log & Compliance-Berichte
|
||||
|
||||
## Status: In Progress
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-12
|
||||
**Last Updated:** 2026-03-13
|
||||
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# PROJ-19: Mailpiler → archivmail Migrationstool
|
||||
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-17
|
||||
**Last Updated:** 2026-03-17
|
||||
|
||||
## Dependencies
|
||||
- Requires: PROJ-5 (Speicherung & Indexierung)
|
||||
- Requires: PROJ-15 (CLI Import)
|
||||
|
||||
## User Stories
|
||||
- Als Admin möchte ich alle E-Mails aus einem bestehenden mailpiler-Archiv nach archivmail migrieren, damit ich die Plattform wechseln kann ohne E-Mails zu verlieren.
|
||||
- Als Admin möchte ich den Fortschritt der Migration in Echtzeit sehen.
|
||||
- Als Admin möchte ich Duplikate automatisch überspringen, damit bei Teil-Migrationen oder Wiederholungsläufen keine Daten doppelt archiviert werden.
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] Methode 1: `pilerexport`-Wrapper – ruft das mailpiler-eigene Export-Tool auf, importiert die EML-Dateien
|
||||
- [x] Methode 2: Direkt – liest `.m`-Dateien aus dem mailpiler Store-Verzeichnis, entschlüsselt (AES-256-CBC) und dekomprimiert (zlib)
|
||||
- [x] Automatische Methodenwahl (`--method auto`): pilerexport → direct
|
||||
- [x] Fortschrittsanzeige (importiert / übersprungen / Fehler)
|
||||
- [x] `--dry-run` Modus
|
||||
- [x] JSON-Ausgabe für Skripting
|
||||
- [x] Datums-Filter (`--date-from`, `--date-to`) für pilerexport-Methode
|
||||
|
||||
## Aufruf
|
||||
|
||||
```bash
|
||||
# Auf dem mailpiler-Server (pilerexport-Methode, empfohlen):
|
||||
archivmail import-piler \
|
||||
--config /etc/archivmail/config.yml \
|
||||
--method pilerexport
|
||||
|
||||
# Mit Datumsfilter:
|
||||
archivmail import-piler \
|
||||
--config /etc/archivmail/config.yml \
|
||||
--date-from 2020-01-01 \
|
||||
--date-to 2024-12-31
|
||||
|
||||
# Direkte Methode (kein mailpiler nötig, kein MySQL nötig):
|
||||
archivmail import-piler \
|
||||
--config /etc/archivmail/config.yml \
|
||||
--method direct \
|
||||
--store-dir /var/piler/store \
|
||||
--key-file /var/piler/store/piler.key
|
||||
|
||||
# Nur Simulation (kein Speichern):
|
||||
archivmail import-piler --dry-run
|
||||
|
||||
# JSON-Ausgabe für Skripte:
|
||||
archivmail import-piler --json
|
||||
```
|
||||
|
||||
## Technische Details
|
||||
|
||||
| Aspekt | Detail |
|
||||
|--------|--------|
|
||||
| pilerexport-Ausgabe | EML-Dateien im temp-Verzeichnis, jede `.eml` = eine E-Mail |
|
||||
| mailpiler-Dateiformat | `{storedir}/**/{piler_id}.m` – AES-256-CBC verschlüsselt, zlib komprimiert |
|
||||
| Entschlüsselung | Erste 16 Bytes = IV, Rest = CBC-Ciphertext, Key aus `piler.key` (32 Bytes) |
|
||||
| Dekomprimierung | zlib (ohne AES falls kein Key vorhanden) |
|
||||
| Duplikat-Erkennung | Storage.Save() + IsIndexed() – identische Inhalts-Hashes werden übersprungen |
|
||||
| Keine ext. Abhängigkeiten | Nur Go stdlib (compress/zlib, crypto/aes) + vorhandene archivmail-Pakete |
|
||||
@@ -1,8 +1,8 @@
|
||||
# PROJ-3: E-Mail-Import: IMAP-Verbindung
|
||||
|
||||
## Status: In Review
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-12
|
||||
**Last Updated:** 2026-03-14
|
||||
**Last Updated:** 2026-03-17
|
||||
|
||||
## Dependencies
|
||||
- Requires: PROJ-1 (Authentifizierung) – nur Admins verwalten IMAP-Verbindungen
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# PROJ-4: E-Mail-Import: SMTP-Eingang (primär via BCC)
|
||||
|
||||
## Status: In Progress
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-12
|
||||
**Last Updated:** 2026-03-12
|
||||
**Last Updated:** 2026-03-17
|
||||
|
||||
## Hinweis
|
||||
**Dies ist der primäre Eingangsweg.** archivmail enthält einen eingebetteten SMTP-Daemon, der **ausschließlich E-Mails empfängt** – kein Versand, keine Weiterleitung, kein MTA. Postfix (oder ein anderer Mailserver) wird per BCC-Mapping oder Always-BCC-Regel so konfiguriert, dass er eine Kopie jeder E-Mail an archivmails SMTP-Daemon zustellt.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# PROJ-6: Volltext-Suche & Filterung
|
||||
|
||||
## Status: In Review
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-12
|
||||
**Last Updated:** 2026-03-17
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# PROJ-7: E-Mail-Ansicht (Lesen & Anhänge)
|
||||
|
||||
## Status: In Progress
|
||||
## Status: Deployed
|
||||
**Created:** 2026-03-12
|
||||
**Last Updated:** 2026-03-12
|
||||
**Last Updated:** 2026-03-17
|
||||
|
||||
## Dependencies
|
||||
- Requires: PROJ-1 (Authentifizierung) – nur eingeloggte Nutzer mit Zugriffsrecht
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
@@ -45,6 +46,7 @@ type Server struct {
|
||||
imapStore *imapstore.Store
|
||||
imapImporter *imapstore.Importer
|
||||
imapScheduler *imapstore.Scheduler
|
||||
uploadJobs sync.Map // jobID → *UploadJob
|
||||
}
|
||||
|
||||
// SetSMTPDaemon wires the SMTP daemon into the API server after construction.
|
||||
@@ -108,6 +110,10 @@ func (s *Server) routes() {
|
||||
s.mux.HandleFunc("GET /api/export/pdf/{id}", s.authMiddleware(s.requireMailAccess(s.handleExportPDF)))
|
||||
s.mux.HandleFunc("POST /api/export/zip", s.authMiddleware(s.requireMailAccess(s.handleExportZIP)))
|
||||
|
||||
// Upload routes (admin only)
|
||||
s.mux.HandleFunc("POST /api/admin/upload", s.authMiddleware(s.requireRole(userstore.RoleAdmin, s.handleUpload)))
|
||||
s.mux.HandleFunc("GET /api/admin/upload/{jobID}/progress", s.authMiddleware(s.requireRole(userstore.RoleAdmin, s.handleUploadProgress)))
|
||||
|
||||
// IMAP routes (accessible to all authenticated users)
|
||||
s.mux.HandleFunc("GET /api/imap", s.authMiddleware(s.handleListImap))
|
||||
s.mux.HandleFunc("POST /api/imap", s.authMiddleware(s.handleCreateImap))
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/archivmail/internal/index"
|
||||
"github.com/archivmail/pkg/mailparser"
|
||||
)
|
||||
|
||||
// UploadJob tracks the progress of an EML/MBOX import job.
|
||||
type UploadJob struct {
|
||||
mu sync.Mutex
|
||||
ID string `json:"id"`
|
||||
Status string `json:"status"` // "running" | "done" | "error"
|
||||
Total int `json:"total"`
|
||||
Imported int `json:"imported"`
|
||||
Skipped int `json:"skipped"`
|
||||
Errors int `json:"errors"`
|
||||
ErrMsg string `json:"error_msg,omitempty"`
|
||||
}
|
||||
|
||||
func (j *UploadJob) snapshot() uploadJobSnapshot {
|
||||
j.mu.Lock()
|
||||
defer j.mu.Unlock()
|
||||
return uploadJobSnapshot{
|
||||
ID: j.ID,
|
||||
Status: j.Status,
|
||||
Total: j.Total,
|
||||
Imported: j.Imported,
|
||||
Skipped: j.Skipped,
|
||||
Errors: j.Errors,
|
||||
ErrMsg: j.ErrMsg,
|
||||
}
|
||||
}
|
||||
|
||||
type uploadJobSnapshot struct {
|
||||
ID string `json:"id"`
|
||||
Status string `json:"status"`
|
||||
Total int `json:"total"`
|
||||
Imported int `json:"imported"`
|
||||
Skipped int `json:"skipped"`
|
||||
Errors int `json:"errors"`
|
||||
ErrMsg string `json:"error_msg,omitempty"`
|
||||
}
|
||||
|
||||
// handleUpload accepts a multipart upload of one or more .eml or .mbox files,
|
||||
// starts a background import job and returns its ID immediately.
|
||||
func (s *Server) handleUpload(w http.ResponseWriter, r *http.Request) {
|
||||
// 512 MB max total upload
|
||||
if err := r.ParseMultipartForm(512 << 20); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "multipart parse failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
files := r.MultipartForm.File["files"]
|
||||
if len(files) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "no files uploaded")
|
||||
return
|
||||
}
|
||||
|
||||
// Collect all raw messages from uploaded files
|
||||
type rawEntry struct {
|
||||
data []byte
|
||||
isMbox bool
|
||||
}
|
||||
var entries []rawEntry
|
||||
|
||||
for _, fh := range files {
|
||||
f, err := fh.Open()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
buf := make([]byte, fh.Size)
|
||||
f.Read(buf) //nolint
|
||||
f.Close()
|
||||
|
||||
name := strings.ToLower(fh.Filename)
|
||||
isMbox := strings.HasSuffix(name, ".mbox")
|
||||
entries = append(entries, rawEntry{buf, isMbox})
|
||||
}
|
||||
|
||||
if len(entries) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "no readable files")
|
||||
return
|
||||
}
|
||||
|
||||
// Count total messages upfront
|
||||
var allMessages [][]byte
|
||||
for _, e := range entries {
|
||||
if e.isMbox {
|
||||
msgs := mailparser.SplitMbox(e.data)
|
||||
allMessages = append(allMessages, msgs...)
|
||||
} else {
|
||||
allMessages = append(allMessages, e.data)
|
||||
}
|
||||
}
|
||||
|
||||
jobID := newJobID()
|
||||
job := &UploadJob{
|
||||
ID: jobID,
|
||||
Status: "running",
|
||||
Total: len(allMessages),
|
||||
}
|
||||
s.uploadJobs.Store(jobID, job)
|
||||
|
||||
// Run import in background
|
||||
go s.runUploadJob(job, allMessages)
|
||||
|
||||
writeJSON(w, http.StatusAccepted, map[string]string{"job_id": jobID})
|
||||
}
|
||||
|
||||
// handleUploadProgress returns the current status of an upload job.
|
||||
func (s *Server) handleUploadProgress(w http.ResponseWriter, r *http.Request) {
|
||||
jobID := r.PathValue("jobID")
|
||||
val, ok := s.uploadJobs.Load(jobID)
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotFound, "job not found")
|
||||
return
|
||||
}
|
||||
job := val.(*UploadJob)
|
||||
writeJSON(w, http.StatusOK, job.snapshot())
|
||||
}
|
||||
|
||||
func (s *Server) runUploadJob(job *UploadJob, messages [][]byte) {
|
||||
ctx := context.Background()
|
||||
|
||||
for _, raw := range messages {
|
||||
result := s.importRawMessage(ctx, raw)
|
||||
job.mu.Lock()
|
||||
switch result {
|
||||
case "imported":
|
||||
job.Imported++
|
||||
case "skipped":
|
||||
job.Skipped++
|
||||
default:
|
||||
job.Errors++
|
||||
}
|
||||
job.mu.Unlock()
|
||||
}
|
||||
|
||||
job.mu.Lock()
|
||||
job.Status = "done"
|
||||
job.mu.Unlock()
|
||||
}
|
||||
|
||||
// importRawMessage stores and indexes a single raw message.
|
||||
// Returns "imported", "skipped", or "error".
|
||||
func (s *Server) importRawMessage(ctx context.Context, raw []byte) string {
|
||||
pm, err := mailparser.Parse(raw)
|
||||
if err != nil {
|
||||
s.logger.Warn("upload: parse failed", "err", err)
|
||||
return "error"
|
||||
}
|
||||
|
||||
id, err := s.store.Save(raw, pm.Date)
|
||||
if err != nil {
|
||||
s.logger.Warn("upload: save failed", "err", err)
|
||||
return "error"
|
||||
}
|
||||
|
||||
// Check dedup: storage.Save returns same id for duplicate content.
|
||||
// If already indexed, skip indexing.
|
||||
if already, _ := s.store.IsIndexed(ctx, id); already {
|
||||
return "skipped"
|
||||
}
|
||||
|
||||
var attachNames []string
|
||||
for _, a := range pm.Attachments {
|
||||
if a.Filename != "" {
|
||||
attachNames = append(attachNames, a.Filename)
|
||||
}
|
||||
}
|
||||
|
||||
doc := index.MailDocument{
|
||||
ID: id,
|
||||
From: pm.From,
|
||||
To: strings.Join(pm.To, " "),
|
||||
Subject: pm.Subject,
|
||||
Body: pm.TextBody + " " + pm.HTMLBody,
|
||||
AttachNames: strings.Join(attachNames, " "),
|
||||
HasAttachment: len(pm.Attachments) > 0,
|
||||
Date: pm.Date,
|
||||
Size: int64(len(raw)),
|
||||
}
|
||||
|
||||
if err := s.idx.IndexSync(doc); err != nil {
|
||||
s.logger.Warn("upload: index failed", "id", id, "err", err)
|
||||
return "error"
|
||||
}
|
||||
|
||||
if err := s.store.SetIndexedAt(ctx, id); err != nil {
|
||||
s.logger.Warn("upload: set indexed_at failed", "id", id, "err", err)
|
||||
}
|
||||
|
||||
if err := s.store.SaveMeta(ctx, id, pm, len(raw)); err != nil {
|
||||
s.logger.Warn("upload: save meta failed", "id", id, "err", err)
|
||||
}
|
||||
|
||||
return "imported"
|
||||
}
|
||||
|
||||
func newJobID() string {
|
||||
b := make([]byte, 8)
|
||||
rand.Read(b) //nolint
|
||||
return hex.EncodeToString(b)
|
||||
}
|
||||
@@ -15,12 +15,15 @@ import {
|
||||
getServices,
|
||||
serviceAction,
|
||||
getSystemStats,
|
||||
uploadMailFiles,
|
||||
getUploadProgress,
|
||||
type User,
|
||||
type AuditEntry,
|
||||
type SMTPStatus,
|
||||
type StorageStats,
|
||||
type ServiceStatus,
|
||||
type SystemStats,
|
||||
type UploadJob,
|
||||
} from "@/lib/api";
|
||||
import { Navbar } from "@/components/navbar";
|
||||
import { Button } from "@/components/ui/button";
|
||||
@@ -111,6 +114,13 @@ export default function AdminPage() {
|
||||
const [auditPage, setAuditPage] = useState(1);
|
||||
const [auditLoading, setAuditLoading] = useState(false);
|
||||
|
||||
// Upload state
|
||||
const [uploadDragging, setUploadDragging] = useState(false);
|
||||
const [uploadJob, setUploadJob] = useState<UploadJob | null>(null);
|
||||
const [uploadError, setUploadError] = useState("");
|
||||
const [uploadLoading, setUploadLoading] = useState(false);
|
||||
const uploadPollRef = useRef<ReturnType<typeof setInterval> | null>(null);
|
||||
|
||||
const loadDashboard = useCallback(async () => {
|
||||
setDashLoading(true);
|
||||
try {
|
||||
@@ -170,6 +180,40 @@ export default function AdminPage() {
|
||||
}
|
||||
}, []);
|
||||
|
||||
async function handleUploadFiles(files: File[]) {
|
||||
const valid = files.filter(f => f.name.toLowerCase().endsWith(".eml") || f.name.toLowerCase().endsWith(".mbox"));
|
||||
if (valid.length === 0) {
|
||||
setUploadError("Nur .eml und .mbox Dateien erlaubt.");
|
||||
return;
|
||||
}
|
||||
setUploadError("");
|
||||
setUploadJob(null);
|
||||
setUploadLoading(true);
|
||||
try {
|
||||
const { job_id } = await uploadMailFiles(valid);
|
||||
// Start polling
|
||||
const poll = setInterval(async () => {
|
||||
try {
|
||||
const job = await getUploadProgress(job_id);
|
||||
setUploadJob(job);
|
||||
if (job.status !== "running") {
|
||||
clearInterval(poll);
|
||||
uploadPollRef.current = null;
|
||||
setUploadLoading(false);
|
||||
}
|
||||
} catch {
|
||||
clearInterval(poll);
|
||||
uploadPollRef.current = null;
|
||||
setUploadLoading(false);
|
||||
}
|
||||
}, 1500);
|
||||
uploadPollRef.current = poll;
|
||||
} catch (e: unknown) {
|
||||
setUploadError(e instanceof Error ? e.message : "Upload fehlgeschlagen.");
|
||||
setUploadLoading(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function handleServiceAction(name: string, action: "start" | "stop" | "restart" | "enable" | "disable" | "block_external" | "allow_external") {
|
||||
setServiceActionLoading(`${name}:${action}`);
|
||||
setServiceError("");
|
||||
@@ -296,6 +340,7 @@ export default function AdminPage() {
|
||||
<TabsTrigger value="services">Dienste</TabsTrigger>
|
||||
<TabsTrigger value="users">Benutzer</TabsTrigger>
|
||||
<TabsTrigger value="audit">Audit-Log</TabsTrigger>
|
||||
<TabsTrigger value="import">Import</TabsTrigger>
|
||||
<TabsTrigger value="modules">Module</TabsTrigger>
|
||||
</TabsList>
|
||||
|
||||
@@ -1019,6 +1064,102 @@ export default function AdminPage() {
|
||||
</>
|
||||
)}
|
||||
</TabsContent>
|
||||
{/* ── Import ── */}
|
||||
<TabsContent value="import" className="mt-4 space-y-4">
|
||||
<h2 className="text-lg font-semibold">EML / MBOX importieren</h2>
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Lade .eml oder .mbox Dateien hoch um sie ins Archiv zu importieren. Duplikate werden automatisch übersprungen.
|
||||
</p>
|
||||
|
||||
{/* Drop zone */}
|
||||
<div
|
||||
onDragOver={(e) => { e.preventDefault(); setUploadDragging(true); }}
|
||||
onDragLeave={() => setUploadDragging(false)}
|
||||
onDrop={(e) => {
|
||||
e.preventDefault();
|
||||
setUploadDragging(false);
|
||||
const files = Array.from(e.dataTransfer.files);
|
||||
handleUploadFiles(files);
|
||||
}}
|
||||
className={`border-2 border-dashed rounded-lg p-10 text-center transition-colors cursor-pointer ${
|
||||
uploadDragging ? "border-primary bg-primary/5" : "border-muted-foreground/25 hover:border-primary/50"
|
||||
}`}
|
||||
onClick={() => document.getElementById("upload-file-input")?.click()}
|
||||
>
|
||||
<input
|
||||
id="upload-file-input"
|
||||
type="file"
|
||||
multiple
|
||||
accept=".eml,.mbox"
|
||||
className="hidden"
|
||||
onChange={(e) => {
|
||||
if (e.target.files) handleUploadFiles(Array.from(e.target.files));
|
||||
e.target.value = "";
|
||||
}}
|
||||
/>
|
||||
<p className="text-sm font-medium">Dateien hierher ziehen oder klicken zum Auswählen</p>
|
||||
<p className="text-xs text-muted-foreground mt-1">Akzeptiert: .eml, .mbox</p>
|
||||
</div>
|
||||
|
||||
{uploadError && (
|
||||
<Alert variant="destructive">
|
||||
<AlertDescription>{uploadError}</AlertDescription>
|
||||
</Alert>
|
||||
)}
|
||||
|
||||
{/* Progress */}
|
||||
{(uploadLoading || uploadJob) && uploadJob && (
|
||||
<Card>
|
||||
<CardContent className="p-4 space-y-3">
|
||||
<div className="flex items-center justify-between">
|
||||
<span className="text-sm font-medium">
|
||||
{uploadJob.status === "running" ? "Import läuft..." : "Import abgeschlossen"}
|
||||
</span>
|
||||
<Badge variant={uploadJob.status === "done" ? "default" : "secondary"}>
|
||||
{uploadJob.status === "done" ? "Fertig" : "Läuft"}
|
||||
</Badge>
|
||||
</div>
|
||||
|
||||
{/* Progress bar */}
|
||||
{uploadJob.total > 0 && (
|
||||
<div className="space-y-1">
|
||||
<div className="w-full bg-muted rounded-full h-2">
|
||||
<div
|
||||
className="bg-primary h-2 rounded-full transition-all"
|
||||
style={{ width: `${Math.min(100, ((uploadJob.imported + uploadJob.skipped + uploadJob.errors) / uploadJob.total) * 100)}%` }}
|
||||
/>
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
{uploadJob.imported + uploadJob.skipped + uploadJob.errors} / {uploadJob.total} verarbeitet
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{uploadJob.status === "done" && (
|
||||
<div className="grid grid-cols-3 gap-3 text-center text-sm">
|
||||
<div className="rounded bg-green-50 dark:bg-green-950 p-2">
|
||||
<p className="font-bold text-green-700 dark:text-green-400">{uploadJob.imported}</p>
|
||||
<p className="text-xs text-muted-foreground">Importiert</p>
|
||||
</div>
|
||||
<div className="rounded bg-yellow-50 dark:bg-yellow-950 p-2">
|
||||
<p className="font-bold text-yellow-700 dark:text-yellow-400">{uploadJob.skipped}</p>
|
||||
<p className="text-xs text-muted-foreground">Übersprungen</p>
|
||||
</div>
|
||||
<div className="rounded bg-red-50 dark:bg-red-950 p-2">
|
||||
<p className="font-bold text-red-700 dark:text-red-400">{uploadJob.errors}</p>
|
||||
<p className="text-xs text-muted-foreground">Fehler</p>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</CardContent>
|
||||
</Card>
|
||||
)}
|
||||
|
||||
{uploadLoading && !uploadJob && (
|
||||
<p className="text-sm text-muted-foreground animate-pulse">Upload läuft...</p>
|
||||
)}
|
||||
</TabsContent>
|
||||
|
||||
<TabsContent value="modules" className="mt-4">
|
||||
<ModulesTab />
|
||||
</TabsContent>
|
||||
|
||||
@@ -453,3 +453,34 @@ export async function exportMailsZIP(ids: string[], attachments: boolean): Promi
|
||||
if (!res.ok) throw new Error("ZIP export failed");
|
||||
return { blob: await res.blob() };
|
||||
}
|
||||
|
||||
// ── Upload ────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface UploadJob {
|
||||
id: string;
|
||||
status: "running" | "done" | "error";
|
||||
total: number;
|
||||
imported: number;
|
||||
skipped: number;
|
||||
errors: number;
|
||||
error_msg?: string;
|
||||
}
|
||||
|
||||
export async function uploadMailFiles(files: File[]): Promise<{ job_id: string }> {
|
||||
const form = new FormData();
|
||||
for (const f of files) form.append("files", f);
|
||||
const res = await fetch(`${API_BASE}/api/admin/upload`, {
|
||||
method: "POST",
|
||||
credentials: "include",
|
||||
body: form,
|
||||
});
|
||||
if (!res.ok) {
|
||||
const body = await res.text();
|
||||
throw new Error(body || `Upload failed: ${res.status}`);
|
||||
}
|
||||
return res.json();
|
||||
}
|
||||
|
||||
export async function getUploadProgress(jobID: string): Promise<UploadJob> {
|
||||
return request<UploadJob>(`/api/admin/upload/${jobID}/progress`);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user