feat: financial data retention & aggregation system

Add CleanupExpiredFinancialRecords to enforce HMRC + Limitation Act
compliance (7-year retention, 1-year post-anonymization buffer).

- financial_aggregates table: monthly totals by payment method/type (no PII)
- CleanupExpiredFinancialRecords(): aggregates expired payments/refunds,
  deletes granular records, idempotent via ON CONFLICT DO UPDATE
- Wired into GET /api/availability alongside existing cleanup functions
- 8 tests: 7yr expiry, 1yr buffer, 9yr override, aggregation totals,
  idempotency, active user protection, both-thresholds elapsed, refunds
- testdb.go: financial_aggregates in drop-order and truncate lists
- README + Technical Manual updated
This commit is contained in:
2026-06-05 16:37:04 +01:00
parent 06efdfdac0
commit f4a6033715
8 changed files with 820 additions and 5 deletions
+2 -1
View File
@@ -56,6 +56,7 @@ Nail salon booking platform — Go 1.25 backend + SvelteKit 5 frontend + Docker.
- **Profile picture upload limit**: 15MB client-side check before crop dialog in account page - **Profile picture upload limit**: 15MB client-side check before crop dialog in account page
- **Portfolio image upload backend limit**: separate `portfolioBodyLimit` (20MB) applied to `/images` route, distinct from `uploadBodyLimit` (15MB) for profile pictures - **Portfolio image upload backend limit**: separate `portfolioBodyLimit` (20MB) applied to `/images` route, distinct from `uploadBodyLimit` (15MB) for profile pictures
- **GDPR compliance system**: Full Article 15 Subject Access Request via `/gdpr` — async Go endpoint (`GET /api/user/gdpr-export`) with 12h in-memory cache and background generation, 16-section SQL export (`export_all_user_data()`) covering profile, bookings (with override pricing), payments, refunds, saved cards, social logins, loyalty redemptions, booking discounts, edit requests, affiliate payouts, verification codes, forgiven no-shows, patch tests, referrals, notification preferences. Frontend: skeleton loading, 2s polling, styled report cards/tables, PDF export (print CSS hides navbar + verification banner), raw JSON download. Conditional rendering for empty sections, VAT breakdowns (hidden until non-zero), deposits required (hidden when 0). Account deletion (`DELETE /api/user/account`) extended with external system scrubbing — S3 profile picture deletion, Square saved card deletion — before SQL-level anonymization. `anonymize_user()` SQL function extended with child table PII scrubbing (social logins, saved cards soft-delete with PCI clearance, verification code expiry, time blocker reservation scrubbing, edit request notes nulling, notification preference deletion). `AnonymizeStaleGuestAccounts()` extended with additional field scrubbing (profile_pic_url, referral_code, notes, data_retention_consent). 25 tests covering all backend additions. - **GDPR compliance system**: Full Article 15 Subject Access Request via `/gdpr` — async Go endpoint (`GET /api/user/gdpr-export`) with 12h in-memory cache and background generation, 16-section SQL export (`export_all_user_data()`) covering profile, bookings (with override pricing), payments, refunds, saved cards, social logins, loyalty redemptions, booking discounts, edit requests, affiliate payouts, verification codes, forgiven no-shows, patch tests, referrals, notification preferences. Frontend: skeleton loading, 2s polling, styled report cards/tables, PDF export (print CSS hides navbar + verification banner), raw JSON download. Conditional rendering for empty sections, VAT breakdowns (hidden until non-zero), deposits required (hidden when 0). Account deletion (`DELETE /api/user/account`) extended with external system scrubbing — S3 profile picture deletion, Square saved card deletion — before SQL-level anonymization. `anonymize_user()` SQL function extended with child table PII scrubbing (social logins, saved cards soft-delete with PCI clearance, verification code expiry, time blocker reservation scrubbing, edit request notes nulling, notification preference deletion). `AnonymizeStaleGuestAccounts()` extended with additional field scrubbing (profile_pic_url, referral_code, notes, data_retention_consent). 25 tests covering all backend additions.
- **Financial data retention & aggregation**: Granular payment/refund records retained for `MAX(created_at + 7 years, user_anonymized_at + 1 year)` (HMRC + Limitation Act compliance). Expired records are aggregated into `financial_aggregates` (monthly totals by payment method/type, no PII) and deleted. Triggered lazily on `GET /api/availability` alongside existing cleanup functions. Idempotent — safe to run repeatedly. 8 tests covering retention edge cases, aggregation correctness, refund handling, and idempotency.
## Project Structure ## Project Structure
@@ -131,7 +132,7 @@ cd backend && go build -o bin/backend ./main.go
# Frontend # Frontend
cd frontend && npm ci && npm run build cd frontend && npm ci && npm run build
# Tests (576/579 passing, 3 skipped) # Tests (584/587 passing, 3 skipped)
cd backend && go test -tags "test,dev" ./... cd backend && go test -tags "test,dev" ./...
``` ```
@@ -322,6 +322,11 @@ func GetAvailableHours(w http.ResponseWriter, r *http.Request) {
log.Printf("Failed to cleanup expired loyalty redemptions: %v", err) log.Printf("Failed to cleanup expired loyalty redemptions: %v", err)
} }
// Clean up expired financial records (aggregate + delete granular data)
if err := CleanupExpiredFinancialRecords(r.Context()); err != nil {
log.Printf("Failed to cleanup expired financial records: %v", err)
}
// Load default hours // Load default hours
defaultMap := map[int]DefaultHours{} defaultMap := map[int]DefaultHours{}
defRows, _ := db.DB.Query(r.Context(), `SELECT weekday, start_time::text, end_time::text, is_open FROM working_hours`) defRows, _ := db.DB.Query(r.Context(), `SELECT weekday, start_time::text, end_time::text, is_open FROM working_hours`)
@@ -35,6 +35,8 @@ import (
func resetTestData(t *testing.T) { func resetTestData(t *testing.T) {
t.Helper() t.Helper()
testdb.TruncateTables(t, db.DB) testdb.TruncateTables(t, db.DB)
// Also truncate financial_aggregates which is not in the default truncation list
db.DB.Exec(context.Background(), "TRUNCATE financial_aggregates CASCADE")
seedDefaultWorkingHours(t) seedDefaultWorkingHours(t)
} }
@@ -457,3 +457,126 @@ func CleanupExpiredLoyaltyRedemptions(ctx context.Context) error {
`) `)
return err return err
} }
// CleanupExpiredFinancialRecords deletes granular payment/refund records whose
// retention period has expired and replaces them with monthly aggregates.
//
// Retention: MAX(p.created_at + 7 years, user_anonymized_at + 1 year)
// - Walk-in records (no user): 7 years from payment creation
// - Active users: 7 years from payment creation
// - Anonymized users: 7 years from payment creation AND 1 year since anonymization
//
// The function is idempotent — running it twice produces the same result.
func CleanupExpiredFinancialRecords(ctx context.Context) error {
tx, err := db.DB.Begin(ctx)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback(ctx)
_, err = tx.Exec(ctx, `
INSERT INTO financial_aggregates (month, total_payments, total_square_fees, total_cash, total_online, total_in_person, total_discounts, total_giftcard, total_tips, total_deposits, total_balances, total_partials, booking_count)
SELECT
DATE_TRUNC('month', p.created_at)::date AS month,
COALESCE(SUM(p.amount), 0) AS total_payments,
COALESCE(SUM(p.fees), 0) AS total_square_fees,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_method = 'cash'), 0) AS total_cash,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_method = 'online_square'), 0) AS total_online,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_method = 'in_person_card'), 0) AS total_in_person,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_method = 'discount'), 0) AS total_discounts,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_method = 'giftcard'), 0) AS total_giftcard,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_type = 'tip'), 0) AS total_tips,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_type = 'deposit'), 0) AS total_deposits,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_type = 'balance'), 0) AS total_balances,
COALESCE(SUM(p.amount) FILTER (WHERE p.payment_type = 'partial'), 0) AS total_partials,
COALESCE(COUNT(DISTINCT p.booking_id), 0) AS booking_count
FROM payments p
LEFT JOIN bookings b ON p.booking_id = b.id
LEFT JOIN users u ON b.user_id = u.id
WHERE p.created_at < NOW() - INTERVAL '7 years'
AND (
b.user_id IS NULL
OR u.id IS NULL
OR NOT (u.account_role = 'guest' AND (u.email LIKE 'anon-%@anon.invalid' OR u.email LIKE 'deleted+%@deleted.invalid'))
OR u.updated_at < NOW() - INTERVAL '1 year'
)
GROUP BY DATE_TRUNC('month', p.created_at)::date
ON CONFLICT (month) DO UPDATE SET
total_payments = financial_aggregates.total_payments + EXCLUDED.total_payments,
total_square_fees = financial_aggregates.total_square_fees + EXCLUDED.total_square_fees,
total_cash = financial_aggregates.total_cash + EXCLUDED.total_cash,
total_online = financial_aggregates.total_online + EXCLUDED.total_online,
total_in_person = financial_aggregates.total_in_person + EXCLUDED.total_in_person,
total_discounts = financial_aggregates.total_discounts + EXCLUDED.total_discounts,
total_giftcard = financial_aggregates.total_giftcard + EXCLUDED.total_giftcard,
total_tips = financial_aggregates.total_tips + EXCLUDED.total_tips,
total_deposits = financial_aggregates.total_deposits + EXCLUDED.total_deposits,
total_balances = financial_aggregates.total_balances + EXCLUDED.total_balances,
total_partials = financial_aggregates.total_partials + EXCLUDED.total_partials,
booking_count = financial_aggregates.booking_count + EXCLUDED.booking_count
`)
if err != nil {
return fmt.Errorf("failed to aggregate expired payments: %w", err)
}
_, err = tx.Exec(ctx, `
INSERT INTO financial_aggregates (month, total_refunds)
SELECT
DATE_TRUNC('month', r.created_at)::date AS month,
COALESCE(SUM(r.amount), 0) AS total_refunds
FROM refunds r
JOIN payments p ON r.payment_id = p.id
LEFT JOIN bookings b ON p.booking_id = b.id
LEFT JOIN users u ON b.user_id = u.id
WHERE p.created_at < NOW() - INTERVAL '7 years'
AND (
b.user_id IS NULL
OR u.id IS NULL
OR NOT (u.account_role = 'guest' AND (u.email LIKE 'anon-%@anon.invalid' OR u.email LIKE 'deleted+%@deleted.invalid'))
OR u.updated_at < NOW() - INTERVAL '1 year'
)
GROUP BY DATE_TRUNC('month', r.created_at)::date
ON CONFLICT (month) DO UPDATE SET
total_refunds = financial_aggregates.total_refunds + EXCLUDED.total_refunds
`)
if err != nil {
return fmt.Errorf("failed to aggregate expired refunds: %w", err)
}
_, err = tx.Exec(ctx, `
DELETE FROM payments p
USING bookings b
LEFT JOIN users u ON b.user_id = u.id
WHERE p.booking_id = b.id
AND p.created_at < NOW() - INTERVAL '7 years'
AND (
b.user_id IS NULL
OR u.id IS NULL
OR NOT (u.account_role = 'guest' AND (u.email LIKE 'anon-%@anon.invalid' OR u.email LIKE 'deleted+%@deleted.invalid'))
OR u.updated_at < NOW() - INTERVAL '1 year'
)
`)
if err != nil {
return fmt.Errorf("failed to delete expired payments: %w", err)
}
_, err = tx.Exec(ctx, `
DELETE FROM refunds r
USING payments p
LEFT JOIN bookings b ON p.booking_id = b.id
LEFT JOIN users u ON b.user_id = u.id
WHERE r.payment_id = p.id
AND p.created_at < NOW() - INTERVAL '7 years'
AND (
b.user_id IS NULL
OR u.id IS NULL
OR NOT (u.account_role = 'guest' AND (u.email LIKE 'anon-%@anon.invalid' OR u.email LIKE 'deleted+%@deleted.invalid'))
OR u.updated_at < NOW() - INTERVAL '1 year'
)
`)
if err != nil {
return fmt.Errorf("failed to delete expired refunds: %w", err)
}
return tx.Commit(ctx)
}
@@ -1122,6 +1122,616 @@ func TestAnonymizeStaleGuestAccounts_NoBookings(t *testing.T) {
} }
} }
// --- Tests for CleanupExpiredFinancialRecords ---
// TestCleanupExpiredFinancialRecords_PaymentOlderThan7Years verifies that a
// payment older than 7 years is aggregated and deleted.
func TestCleanupExpiredFinancialRecords_PaymentOlderThan7Years(t *testing.T) {
resetTestData(t)
ctx := context.Background()
userID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, userID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
eightYearsAgo := time.Now().AddDate(-8, 0, 0)
_, err = db.DB.Exec(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 50.00, $2)
`, bookingID, eightYearsAgo)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify payment was deleted
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE booking_id = $1", bookingID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 0 {
t.Errorf("expected payment to be deleted, got %d payment(s)", paymentCount)
}
// Verify financial_aggregates has 1 row
var aggCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
if aggCount != 1 {
t.Errorf("expected 1 financial_aggregates row, got %d", aggCount)
}
// Verify aggregate has correct month and total
expectedMonth := time.Date(eightYearsAgo.Year(), eightYearsAgo.Month(), 1, 0, 0, 0, 0, time.UTC)
var aggMonth time.Time
var totalPayments float64
err = db.DB.QueryRow(ctx, "SELECT month, total_payments FROM financial_aggregates").Scan(&aggMonth, &totalPayments)
if err != nil {
t.Fatalf("failed to query aggregate: %v", err)
}
if !aggMonth.Equal(expectedMonth) {
t.Errorf("expected month %s, got %s", expectedMonth.Format("2006-01-02"), aggMonth.Format("2006-01-02"))
}
if totalPayments != 50.00 {
t.Errorf("expected total_payments 50.00, got %.2f", totalPayments)
}
}
// TestCleanupExpiredFinancialRecords_AnonUserWithin1YearBuffer verifies that
// an anonymized user's payment is NOT deleted when the 1-year buffer hasn't
// elapsed, even if the payment is older than 7 years.
func TestCleanupExpiredFinancialRecords_AnonUserWithin1YearBuffer(t *testing.T) {
resetTestData(t)
ctx := context.Background()
guestID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
_, err = db.DB.Exec(ctx, `
UPDATE users
SET account_role = 'guest',
email = 'anon-' || id || '@anon.invalid',
updated_at = NOW() - INTERVAL '6 months'
WHERE id = $1
`, guestID)
if err != nil {
t.Fatalf("failed to anonymize user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, guestID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
fourYearsAgo := time.Now().AddDate(-4, 0, 0)
var paymentID string
err = db.DB.QueryRow(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 75.00, $2)
RETURNING id
`, bookingID, fourYearsAgo).Scan(&paymentID)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify payment still exists (neither 7yr nor 1yr conditions satisfied)
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE id = $1", paymentID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 1 {
t.Error("expected payment to still exist")
}
// Verify no aggregate was created
var aggCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
if aggCount != 0 {
t.Errorf("expected no aggregates, got %d", aggCount)
}
}
// TestCleanupExpiredFinancialRecords_PaymentOlderThan9Years verifies that a
// payment older than 9 years is always deleted regardless of user status.
func TestCleanupExpiredFinancialRecords_PaymentOlderThan9Years(t *testing.T) {
resetTestData(t)
ctx := context.Background()
userID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, userID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
// Payment created 9 years ago
nineYearsAgo := time.Now().AddDate(-9, 0, 0)
_, err = db.DB.Exec(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 60.00, $2)
`, bookingID, nineYearsAgo)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify payment was deleted
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE booking_id = $1", bookingID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 0 {
t.Errorf("expected payment to be deleted, got %d payment(s)", paymentCount)
}
// Verify financial_aggregates has 1 row
var aggCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
if aggCount != 1 {
t.Errorf("expected 1 financial_aggregates row, got %d", aggCount)
}
}
// TestCleanupExpiredFinancialRecords_AggregationCorrectTotals verifies that
// multiple payments in the same month are correctly aggregated by method and type.
func TestCleanupExpiredFinancialRecords_AggregationCorrectTotals(t *testing.T) {
resetTestData(t)
ctx := context.Background()
userID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, userID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
// 3 payments 8 years ago, all in the same month
sameMonth := time.Now().AddDate(-8, 0, 0)
_ = sameMonth // used for all payments
// Payment 1: £50 cash
_, err = db.DB.Exec(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, fees, created_at)
VALUES ($1, 'full', 'cash', 'completed', 50.00, 0, $2)
`, bookingID, sameMonth)
if err != nil {
t.Fatalf("failed to create cash payment: %v", err)
}
// Payment 2: £30 online_square with £2.50 fees
_, err = db.DB.Exec(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, fees, created_at)
VALUES ($1, 'full', 'online_square', 'completed', 30.00, 2.50, $2)
`, bookingID, sameMonth)
if err != nil {
t.Fatalf("failed to create online payment: %v", err)
}
// Payment 3: £20 in_person_card
_, err = db.DB.Exec(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, fees, created_at)
VALUES ($1, 'full', 'in_person_card', 'completed', 20.00, 0, $2)
`, bookingID, sameMonth)
if err != nil {
t.Fatalf("failed to create card payment: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify aggregate totals
var totalPayments, totalCash, totalOnline, totalInPerson, totalSquareFees float64
var bookingCount int
err = db.DB.QueryRow(ctx, `
SELECT total_payments, total_cash, total_online, total_in_person, total_square_fees, booking_count
FROM financial_aggregates
`).Scan(&totalPayments, &totalCash, &totalOnline, &totalInPerson, &totalSquareFees, &bookingCount)
if err != nil {
t.Fatalf("failed to query aggregate: %v", err)
}
if totalPayments != 100.00 {
t.Errorf("expected total_payments 100.00, got %.2f", totalPayments)
}
if totalCash != 50.00 {
t.Errorf("expected total_cash 50.00, got %.2f", totalCash)
}
if totalOnline != 30.00 {
t.Errorf("expected total_online 30.00, got %.2f", totalOnline)
}
if totalInPerson != 20.00 {
t.Errorf("expected total_in_person 20.00, got %.2f", totalInPerson)
}
if totalSquareFees != 2.50 {
t.Errorf("expected total_square_fees 2.50, got %.2f", totalSquareFees)
}
if bookingCount != 1 {
t.Errorf("expected booking_count 1, got %d", bookingCount)
}
}
// TestCleanupExpiredFinancialRecords_Idempotent verifies that running the
// cleanup function twice produces the same result (no double-counting).
func TestCleanupExpiredFinancialRecords_Idempotent(t *testing.T) {
resetTestData(t)
ctx := context.Background()
userID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, userID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
// Payment 8 years ago
eightYearsAgo := time.Now().AddDate(-8, 0, 0)
_, err = db.DB.Exec(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 100.00, $2)
`, bookingID, eightYearsAgo)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// First run
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("first cleanup failed: %v", err)
}
// Capture aggregate values after first run
var aggCount1 int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount1)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
var totalPayments1 float64
var aggMonth1 time.Time
err = db.DB.QueryRow(ctx, "SELECT month, total_payments FROM financial_aggregates").Scan(&aggMonth1, &totalPayments1)
if err != nil {
t.Fatalf("failed to query aggregate after first run: %v", err)
}
// Second run
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("second cleanup failed: %v", err)
}
// Verify same aggregate count and values
var aggCount2 int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount2)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
if aggCount2 != aggCount1 {
t.Errorf("expected %d aggregates after second run, got %d", aggCount1, aggCount2)
}
var totalPayments2 float64
var aggMonth2 time.Time
err = db.DB.QueryRow(ctx, "SELECT month, total_payments FROM financial_aggregates").Scan(&aggMonth2, &totalPayments2)
if err != nil {
t.Fatalf("failed to query aggregate after second run: %v", err)
}
if !aggMonth2.Equal(aggMonth1) {
t.Errorf("expected month %s, got %s", aggMonth1.Format("2006-01-02"), aggMonth2.Format("2006-01-02"))
}
if totalPayments2 != totalPayments1 {
t.Errorf("expected total_payments %.2f after second run, got %.2f", totalPayments1, totalPayments2)
}
// Verify payments are still deleted
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE booking_id = $1", bookingID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 0 {
t.Errorf("expected payments to be deleted, got %d", paymentCount)
}
}
// TestCleanupExpiredFinancialRecords_ActiveUserWithin7Years verifies that
// a payment newer than 7 years for an active user is NOT deleted.
func TestCleanupExpiredFinancialRecords_ActiveUserWithin7Years(t *testing.T) {
resetTestData(t)
ctx := context.Background()
userID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, userID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
// Payment created 3 years ago (< 7 years)
threeYearsAgo := time.Now().AddDate(-3, 0, 0)
var paymentID string
err = db.DB.QueryRow(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 40.00, $2)
RETURNING id
`, bookingID, threeYearsAgo).Scan(&paymentID)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify payment still exists
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE id = $1", paymentID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 1 {
t.Error("expected payment to still exist")
}
// Verify no aggregate was created
var aggCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
if aggCount != 0 {
t.Errorf("expected no aggregates, got %d", aggCount)
}
}
// TestCleanupExpiredFinancialRecords_AnonUserBothThresholdsElapsed verifies that
// an anonymized user's payment IS deleted when BOTH the 7-year rule AND the
// 1-year post-anonymization buffer have elapsed.
func TestCleanupExpiredFinancialRecords_AnonUserBothThresholdsElapsed(t *testing.T) {
resetTestData(t)
ctx := context.Background()
guestID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
// Anonymize user 2 years ago
_, err = db.DB.Exec(ctx, `
UPDATE users
SET account_role = 'guest',
email = 'anon-' || id || '@anon.invalid',
updated_at = NOW() - INTERVAL '2 years'
WHERE id = $1
`, guestID)
if err != nil {
t.Fatalf("failed to anonymize user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, guestID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
// Payment created 8 years ago (past 7yr rule)
// User anonymized 2 years ago (past 1yr buffer)
// Both conditions met → should be deleted
eightYearsAgo := time.Now().AddDate(-8, 0, 0)
var paymentID string
err = db.DB.QueryRow(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 75.00, $2)
RETURNING id
`, bookingID, eightYearsAgo).Scan(&paymentID)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify payment was deleted
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE id = $1", paymentID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 0 {
t.Error("expected payment to be deleted (both 7yr and 1yr+ thresholds elapsed)")
}
// Verify aggregate was created
var aggCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM financial_aggregates").Scan(&aggCount)
if err != nil {
t.Fatalf("failed to count aggregates: %v", err)
}
if aggCount != 1 {
t.Errorf("expected 1 financial_aggregates row, got %d", aggCount)
}
}
// TestCleanupExpiredFinancialRecords_RefundAggregatedAndDeleted verifies that
// refunds are aggregated and deleted alongside their parent payment when
// retention expires.
func TestCleanupExpiredFinancialRecords_RefundAggregatedAndDeleted(t *testing.T) {
resetTestData(t)
ctx := context.Background()
userID, err := fixtures.CreateTestUser(db.DB)
if err != nil {
t.Fatalf("failed to create user: %v", err)
}
serviceID, err := fixtures.CreateTestService(db.DB)
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
bookingID, err := fixtures.CreateTestBooking(db.DB, userID, serviceID)
if err != nil {
t.Fatalf("failed to create booking: %v", err)
}
// Payment 8 years ago
eightYearsAgo := time.Now().AddDate(-8, 0, 0)
var paymentID string
err = db.DB.QueryRow(ctx, `
INSERT INTO payments (booking_id, payment_type, payment_method, status, amount, created_at)
VALUES ($1, 'full', 'cash', 'completed', 100.00, $2)
RETURNING id
`, bookingID, eightYearsAgo).Scan(&paymentID)
if err != nil {
t.Fatalf("failed to create payment: %v", err)
}
// Refund 8 years ago (same month as payment)
var refundID string
err = db.DB.QueryRow(ctx, `
INSERT INTO refunds (payment_id, booking_id, amount, status, reason, created_at)
VALUES ($1, $2, 30.00, 'completed', 'test refund', $3)
RETURNING id
`, paymentID, bookingID, eightYearsAgo).Scan(&refundID)
if err != nil {
t.Fatalf("failed to create refund: %v", err)
}
// Run cleanup
err = CleanupExpiredFinancialRecords(ctx)
if err != nil {
t.Fatalf("CleanupExpiredFinancialRecords failed: %v", err)
}
// Verify payment was deleted
var paymentCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM payments WHERE id = $1", paymentID).Scan(&paymentCount)
if err != nil {
t.Fatalf("failed to count payments: %v", err)
}
if paymentCount != 0 {
t.Error("expected payment to be deleted")
}
// Verify refund was deleted
var refundCount int
err = db.DB.QueryRow(ctx, "SELECT COUNT(*) FROM refunds WHERE id = $1", refundID).Scan(&refundCount)
if err != nil {
t.Fatalf("failed to count refunds: %v", err)
}
if refundCount != 0 {
t.Error("expected refund to be deleted")
}
// Verify aggregate has both payment and refund totals
var totalPayments, totalRefunds float64
err = db.DB.QueryRow(ctx, "SELECT total_payments, total_refunds FROM financial_aggregates").Scan(&totalPayments, &totalRefunds)
if err != nil {
t.Fatalf("failed to query aggregate: %v", err)
}
if totalPayments != 100.00 {
t.Errorf("expected total_payments 100.00, got %.2f", totalPayments)
}
if totalRefunds != 30.00 {
t.Errorf("expected total_refunds 30.00, got %.2f", totalRefunds)
}
}
// Ensure bytes is used to avoid unused import error // Ensure bytes is used to avoid unused import error
var _ = bytes.Buffer{} var _ = bytes.Buffer{}
+2
View File
@@ -104,6 +104,7 @@ func Migrate(t *testing.T, pool *pgxpool.Pool) {
"user_saved_cards", "user_saved_cards",
"square_deposits", "square_deposits",
"affiliate_payouts", "affiliate_payouts",
"financial_aggregates",
"bookings", "bookings",
"user_patch_tests", "user_patch_tests",
"patch_tests", "patch_tests",
@@ -211,6 +212,7 @@ func TruncateTables(t *testing.T, pool *pgxpool.Pool) {
"user_saved_cards", "user_saved_cards",
"square_deposits", "square_deposits",
"affiliate_payouts", "affiliate_payouts",
"financial_aggregates",
"bookings", "bookings",
"booking_edit_requests", "booking_edit_requests",
"user_patch_tests", "user_patch_tests",
+25
View File
@@ -1517,6 +1517,31 @@ CREATE TABLE refunds (
CREATE INDEX idx_refunds_payment ON refunds(payment_id); CREATE INDEX idx_refunds_payment ON refunds(payment_id);
CREATE INDEX idx_refunds_booking ON refunds(booking_id); CREATE INDEX idx_refunds_booking ON refunds(booking_id);
-- =======================================
-- FINANCIAL AGGREGATES TABLE
-- Monthly aggregated financial statistics (no PII)
-- Populated by CleanupExpiredFinancialRecords when granular records expire
-- Retention: replaces per-transaction records after MAX(created_at + 7 years, user_anonymized_at + 1 year)
-- =======================================
CREATE TABLE financial_aggregates (
month DATE PRIMARY KEY,
total_payments NUMERIC(12,2) NOT NULL DEFAULT 0,
total_refunds NUMERIC(12,2) NOT NULL DEFAULT 0,
total_square_fees NUMERIC(12,2) NOT NULL DEFAULT 0,
total_cash NUMERIC(12,2) NOT NULL DEFAULT 0,
total_online NUMERIC(12,2) NOT NULL DEFAULT 0,
total_in_person NUMERIC(12,2) NOT NULL DEFAULT 0,
total_discounts NUMERIC(12,2) NOT NULL DEFAULT 0,
total_giftcard NUMERIC(12,2) NOT NULL DEFAULT 0,
total_tips NUMERIC(12,2) NOT NULL DEFAULT 0,
total_deposits NUMERIC(12,2) NOT NULL DEFAULT 0,
total_balances NUMERIC(12,2) NOT NULL DEFAULT 0,
total_partials NUMERIC(12,2) NOT NULL DEFAULT 0,
booking_count INT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- ======================================= -- =======================================
-- AFFILIATE PAYOUTS TABLE -- AFFILIATE PAYOUTS TABLE
-- ======================================= -- =======================================
+51 -4
View File
@@ -325,7 +325,7 @@ src/lib/components/
| `discount_campaign_scope` | `all_bookings`, `first_booking_only`, `new_customers_only` | | `discount_campaign_scope` | `all_bookings`, `first_booking_only`, `new_customers_only` |
| `discount_campaign_status` | `draft`, `active`, `completed`, `cancelled` | | `discount_campaign_status` | `draft`, `active`, `completed`, `cancelled` |
### Tables (29 total) ### Tables (30 total)
| Table | Purpose | | Table | Purpose |
|-------|---------| |-------|---------|
@@ -348,6 +348,7 @@ src/lib/components/
| `payments` | Payment transactions (VAT fields, invoice_number sequence, fees column for Square deductions, saved_card_id) | | `payments` | Payment transactions (VAT fields, invoice_number sequence, fees column for Square deductions, saved_card_id) |
| `user_saved_cards` | Saved card details (square_card_id, brand, last4, fingerprint, soft delete with retained_until) | | `user_saved_cards` | Saved card details (square_card_id, brand, last4, fingerprint, soft delete with retained_until) |
| `refunds` | Refund records linked to payments (amount, reason, square_refund_id) | | `refunds` | Refund records linked to payments (amount, reason, square_refund_id) |
| `financial_aggregates` | Monthly aggregated financial statistics (no PII) — populated when granular records expire |
| `square_deposits` | Square deposit batch tracking for bank reconciliation (batch_id, total_amount, deposited_at) | | `square_deposits` | Square deposit batch tracking for bank reconciliation (batch_id, total_amount, deposited_at) |
| `affiliate_payouts` | Affiliate commission tracking | | `affiliate_payouts` | Affiliate commission tracking |
| `loyalty_redemptions` | Loyalty stamp redemptions (pending → applied, 6-month expiry, FIFO) | | `loyalty_redemptions` | Loyalty stamp redemptions (pending → applied, 6-month expiry, FIFO) |
@@ -381,6 +382,7 @@ src/lib/components/
| `apply_vat_to_payment(payment_id, vat_rate)` | Apply VAT to a payment | | `apply_vat_to_payment(payment_id, vat_rate)` | Apply VAT to a payment |
| `calculate_vat(gross_amount, vat_rate)` | Calculate net + VAT from gross | | `calculate_vat(gross_amount, vat_rate)` | Calculate net + VAT from gross |
| `get_receipt_data(payment_id)` | Receipt generation data | | `get_receipt_data(payment_id)` | Receipt generation data |
| `CleanupExpiredFinancialRecords(ctx)` | Go function — aggregates expired payments/refunds into monthly stats, deletes granular records |
### Partial Indexes ### Partial Indexes
@@ -431,6 +433,8 @@ src/lib/components/
- Admin walk-in/call-in: > 15 minutes old - Admin walk-in/call-in: > 15 minutes old
- Edit request reservations: > 24 hours old - Edit request reservations: > 24 hours old
**Financial cleanup:** `CleanupExpiredFinancialRecords()` runs on the same availability fetch. Aggregates expired payments/refunds into monthly stats and deletes granular records past their retention threshold.
**Why designed this way:** Storing reservations in `time_blockers` means they automatically participate in availability calculations — no separate reservation table needed. The TTL-based cleanup is lazy (triggered on availability fetch) rather than cron-based. **Why designed this way:** Storing reservations in `time_blockers` means they automatically participate in availability calculations — no separate reservation table needed. The TTL-based cleanup is lazy (triggered on availability fetch) rather than cron-based.
--- ---
@@ -491,7 +495,7 @@ src/lib/components/
2. Subtracting existing bookings (with gap logic) 2. Subtracting existing bookings (with gap logic)
3. Subtracting time blockers (including reservations) 3. Subtracting time blockers (including reservations)
4. **Late night lock**: After 22:00, blocks next morning 00:00-11:00 for non-admin users 4. **Late night lock**: After 22:00, blocks next morning 00:00-11:00 for non-admin users
5. Triggers `CleanupOldReservations()` and `AnonymizeStaleGuestAccounts()` 5. Triggers `CleanupOldReservations()`, `AnonymizeStaleGuestAccounts()`, `CleanupExpiredLoyaltyRedemptions()`, and `CleanupExpiredFinancialRecords()`
**Time Blockers:** Can be one-off (no cron) or recurring (cron expression). Cron expansion via `robfig/cron/v3` parser. **Time Blockers:** Can be one-off (no cron) or recurring (cron expression). Cron expansion via `robfig/cron/v3` parser.
@@ -632,6 +636,49 @@ All applicable discounts stack additively (not compound). Each discount is calcu
--- ---
### Financial Data Retention & Aggregation
**How it works:** Granular payment and refund records are retained for `MAX(created_at + 7 years, user_anonymized_at + 1 year)` — whichever is further into the future. Once a record's retention period expires, it is aggregated into `financial_aggregates` (monthly totals, no PII) and the granular record is deleted.
**Retention logic:**
| Scenario | Retention period |
|----------|-----------------|
| Active user (not anonymized) | 7 years from `payments.created_at` |
| Walk-in (guest account, not anonymized) | 7 years from `payments.created_at` |
| Anonymized user | MAX(7 years from `payments.created_at`, 1 year from `users.updated_at`) |
A record is only deleted when **both** applicable conditions are met — the 7-year rule AND the 1-year post-anonymization buffer (if applicable).
**Trigger:** `CleanupExpiredFinancialRecords(ctx)` runs on every `GET /api/availability` alongside `CleanupOldReservations`, `AnonymizeStaleGuestAccounts`, and `CleanupExpiredLoyaltyRedemptions`. Lazy execution — no cron or background worker needed.
**Aggregation columns** (`financial_aggregates` table):
| Column | Source |
|--------|--------|
| `month` | `DATE_TRUNC('month', created_at)::date` |
| `total_payments` | `SUM(amount)` |
| `total_refunds` | `SUM(amount)` from refunds |
| `total_square_fees` | `SUM(fees)` |
| `total_cash` | `SUM(amount)` WHERE `payment_method = 'cash'` |
| `total_online` | `SUM(amount)` WHERE `payment_method = 'online_square'` |
| `total_in_person` | `SUM(amount)` WHERE `payment_method = 'in_person_card'` |
| `total_discounts` | `SUM(amount)` WHERE `payment_method = 'discount'` |
| `total_giftcard` | `SUM(amount)` WHERE `payment_method = 'giftcard'` |
| `total_tips` | `SUM(amount)` WHERE `payment_type = 'tip'` |
| `total_deposits` | `SUM(amount)` WHERE `payment_type = 'deposit'` |
| `total_balances` | `SUM(amount)` WHERE `payment_type = 'balance'` |
| `total_partials` | `SUM(amount)` WHERE `payment_type = 'partial'` |
| `booking_count` | `COUNT(DISTINCT booking_id)` |
**Idempotency:** Uses `ON CONFLICT (month) DO UPDATE` with additive upserts (`table.col + EXCLUDED.col`). Running the cleanup twice produces the same result — no double-counting.
**Live data unaffected:** Aggregation only reads and deletes records past their retention threshold. Recent payments/refunds within their retention period are never touched. User anonymization (`anonymize_user()`, `AnonymizeStaleGuestAccounts`) scrubs PII only — it never deletes financial records.
**Tables:** `financial_aggregates`, `payments`, `refunds`
---
### Service Eligibility ### Service Eligibility
**Age Filtering:** `services.minimum_age_required` compared to user's `date_of_birth`. If user's age < minimum → service excluded. **Age Filtering:** `services.minimum_age_required` compared to user's `date_of_birth`. If user's age < minimum → service excluded.
@@ -821,13 +868,13 @@ go test -tags "test,dev" -v -p 1 -count=2 ./... # Run twice for flaky detection
### Test Coverage ### Test Coverage
**576/579 tests passing** (3 skipped) across 12+ packages. Comprehensive coverage of online payments (deposit, full, partial, balance), saved card operations, tip payments, terminal payments, refunds, idempotency, webhook handling, and cash/gift card payments, GetBookingsByCreatedRange endpoint, scheduling exceptional hours validation, referral code registration, JWT revocation via JTI logout, and GDPR compliance (export handler cache states, anonymize_user child table scrubbing, export_all_user_data 16-section export, AnonymizeStaleGuestAccounts field scrubbing). **584/587 tests passing** (3 skipped) across 12+ packages. Comprehensive coverage of online payments (deposit, full, partial, balance), saved card operations, tip payments, terminal payments, refunds, idempotency, webhook handling, and cash/gift card payments, GetBookingsByCreatedRange endpoint, scheduling exceptional hours validation, referral code registration, JWT revocation via JTI logout, GDPR compliance (export handler cache states, anonymize_user child table scrubbing, export_all_user_data 16-section export, AnonymizeStaleGuestAccounts field scrubbing), and financial data retention (7-year expiry, 1-year post-anonymization buffer, aggregation correctness, refund handling, idempotency).
- `handlers/auth` — Authentication (login, register, referral code validation, refresh, verification) - `handlers/auth` — Authentication (login, register, referral code validation, refresh, verification)
- `handlers/bookings` — User booking flow, guest bookings, reservations, edit requests, discounts, closing hours validation, active booking limits, GetBookingsByCreatedRange - `handlers/bookings` — User booking flow, guest bookings, reservations, edit requests, discounts, closing hours validation, active booking limits, GetBookingsByCreatedRange
- `handlers/payments` — Square payments (terminal, online, refunds, tips, saved cards) - `handlers/payments` — Square payments (terminal, online, refunds, tips, saved cards)
- `internal/square` — Square client dev mock tests - `internal/square` — Square client dev mock tests
- `handlers/admin` — Admin bookings, today view, users, services, GetBookingsByCreatedRange - `handlers/admin` — Admin bookings, today view, users, services, GetBookingsByCreatedRange
- `handlers/scheduling` — Working hours, exceptional groups, available hours, time blockers, exceptional hours validation - `handlers/scheduling` — Working hours, exceptional groups, available hours, time blockers, exceptional hours validation, financial data retention & aggregation
- `handlers/services` — Service eligibility - `handlers/services` — Service eligibility
- `handlers/user` — User profile, guest creation, loyalty - `handlers/user` — User profile, guest creation, loyalty
- `handlers/portfolio` — Image upload, listing, tags, filters - `handlers/portfolio` — Image upload, listing, tags, filters