feat: financial data retention & aggregation system
Add CleanupExpiredFinancialRecords to enforce HMRC + Limitation Act compliance (7-year retention, 1-year post-anonymization buffer). - financial_aggregates table: monthly totals by payment method/type (no PII) - CleanupExpiredFinancialRecords(): aggregates expired payments/refunds, deletes granular records, idempotent via ON CONFLICT DO UPDATE - Wired into GET /api/availability alongside existing cleanup functions - 8 tests: 7yr expiry, 1yr buffer, 9yr override, aggregation totals, idempotency, active user protection, both-thresholds elapsed, refunds - testdb.go: financial_aggregates in drop-order and truncate lists - README + Technical Manual updated
This commit is contained in:
@@ -325,7 +325,7 @@ src/lib/components/
|
||||
| `discount_campaign_scope` | `all_bookings`, `first_booking_only`, `new_customers_only` |
|
||||
| `discount_campaign_status` | `draft`, `active`, `completed`, `cancelled` |
|
||||
|
||||
### Tables (29 total)
|
||||
### Tables (30 total)
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
@@ -348,6 +348,7 @@ src/lib/components/
|
||||
| `payments` | Payment transactions (VAT fields, invoice_number sequence, fees column for Square deductions, saved_card_id) |
|
||||
| `user_saved_cards` | Saved card details (square_card_id, brand, last4, fingerprint, soft delete with retained_until) |
|
||||
| `refunds` | Refund records linked to payments (amount, reason, square_refund_id) |
|
||||
| `financial_aggregates` | Monthly aggregated financial statistics (no PII) — populated when granular records expire |
|
||||
| `square_deposits` | Square deposit batch tracking for bank reconciliation (batch_id, total_amount, deposited_at) |
|
||||
| `affiliate_payouts` | Affiliate commission tracking |
|
||||
| `loyalty_redemptions` | Loyalty stamp redemptions (pending → applied, 6-month expiry, FIFO) |
|
||||
@@ -381,6 +382,7 @@ src/lib/components/
|
||||
| `apply_vat_to_payment(payment_id, vat_rate)` | Apply VAT to a payment |
|
||||
| `calculate_vat(gross_amount, vat_rate)` | Calculate net + VAT from gross |
|
||||
| `get_receipt_data(payment_id)` | Receipt generation data |
|
||||
| `CleanupExpiredFinancialRecords(ctx)` | Go function — aggregates expired payments/refunds into monthly stats, deletes granular records |
|
||||
|
||||
### Partial Indexes
|
||||
|
||||
@@ -431,6 +433,8 @@ src/lib/components/
|
||||
- Admin walk-in/call-in: > 15 minutes old
|
||||
- Edit request reservations: > 24 hours old
|
||||
|
||||
**Financial cleanup:** `CleanupExpiredFinancialRecords()` runs on the same availability fetch. Aggregates expired payments/refunds into monthly stats and deletes granular records past their retention threshold.
|
||||
|
||||
**Why designed this way:** Storing reservations in `time_blockers` means they automatically participate in availability calculations — no separate reservation table needed. The TTL-based cleanup is lazy (triggered on availability fetch) rather than cron-based.
|
||||
|
||||
---
|
||||
@@ -491,7 +495,7 @@ src/lib/components/
|
||||
2. Subtracting existing bookings (with gap logic)
|
||||
3. Subtracting time blockers (including reservations)
|
||||
4. **Late night lock**: After 22:00, blocks next morning 00:00-11:00 for non-admin users
|
||||
5. Triggers `CleanupOldReservations()` and `AnonymizeStaleGuestAccounts()`
|
||||
5. Triggers `CleanupOldReservations()`, `AnonymizeStaleGuestAccounts()`, `CleanupExpiredLoyaltyRedemptions()`, and `CleanupExpiredFinancialRecords()`
|
||||
|
||||
**Time Blockers:** Can be one-off (no cron) or recurring (cron expression). Cron expansion via `robfig/cron/v3` parser.
|
||||
|
||||
@@ -632,6 +636,49 @@ All applicable discounts stack additively (not compound). Each discount is calcu
|
||||
|
||||
---
|
||||
|
||||
### Financial Data Retention & Aggregation
|
||||
|
||||
**How it works:** Granular payment and refund records are retained for `MAX(created_at + 7 years, user_anonymized_at + 1 year)` — whichever is further into the future. Once a record's retention period expires, it is aggregated into `financial_aggregates` (monthly totals, no PII) and the granular record is deleted.
|
||||
|
||||
**Retention logic:**
|
||||
|
||||
| Scenario | Retention period |
|
||||
|----------|-----------------|
|
||||
| Active user (not anonymized) | 7 years from `payments.created_at` |
|
||||
| Walk-in (guest account, not anonymized) | 7 years from `payments.created_at` |
|
||||
| Anonymized user | MAX(7 years from `payments.created_at`, 1 year from `users.updated_at`) |
|
||||
|
||||
A record is only deleted when **both** applicable conditions are met — the 7-year rule AND the 1-year post-anonymization buffer (if applicable).
|
||||
|
||||
**Trigger:** `CleanupExpiredFinancialRecords(ctx)` runs on every `GET /api/availability` alongside `CleanupOldReservations`, `AnonymizeStaleGuestAccounts`, and `CleanupExpiredLoyaltyRedemptions`. Lazy execution — no cron or background worker needed.
|
||||
|
||||
**Aggregation columns** (`financial_aggregates` table):
|
||||
|
||||
| Column | Source |
|
||||
|--------|--------|
|
||||
| `month` | `DATE_TRUNC('month', created_at)::date` |
|
||||
| `total_payments` | `SUM(amount)` |
|
||||
| `total_refunds` | `SUM(amount)` from refunds |
|
||||
| `total_square_fees` | `SUM(fees)` |
|
||||
| `total_cash` | `SUM(amount)` WHERE `payment_method = 'cash'` |
|
||||
| `total_online` | `SUM(amount)` WHERE `payment_method = 'online_square'` |
|
||||
| `total_in_person` | `SUM(amount)` WHERE `payment_method = 'in_person_card'` |
|
||||
| `total_discounts` | `SUM(amount)` WHERE `payment_method = 'discount'` |
|
||||
| `total_giftcard` | `SUM(amount)` WHERE `payment_method = 'giftcard'` |
|
||||
| `total_tips` | `SUM(amount)` WHERE `payment_type = 'tip'` |
|
||||
| `total_deposits` | `SUM(amount)` WHERE `payment_type = 'deposit'` |
|
||||
| `total_balances` | `SUM(amount)` WHERE `payment_type = 'balance'` |
|
||||
| `total_partials` | `SUM(amount)` WHERE `payment_type = 'partial'` |
|
||||
| `booking_count` | `COUNT(DISTINCT booking_id)` |
|
||||
|
||||
**Idempotency:** Uses `ON CONFLICT (month) DO UPDATE` with additive upserts (`table.col + EXCLUDED.col`). Running the cleanup twice produces the same result — no double-counting.
|
||||
|
||||
**Live data unaffected:** Aggregation only reads and deletes records past their retention threshold. Recent payments/refunds within their retention period are never touched. User anonymization (`anonymize_user()`, `AnonymizeStaleGuestAccounts`) scrubs PII only — it never deletes financial records.
|
||||
|
||||
**Tables:** `financial_aggregates`, `payments`, `refunds`
|
||||
|
||||
---
|
||||
|
||||
### Service Eligibility
|
||||
|
||||
**Age Filtering:** `services.minimum_age_required` compared to user's `date_of_birth`. If user's age < minimum → service excluded.
|
||||
@@ -821,13 +868,13 @@ go test -tags "test,dev" -v -p 1 -count=2 ./... # Run twice for flaky detection
|
||||
|
||||
### Test Coverage
|
||||
|
||||
**576/579 tests passing** (3 skipped) across 12+ packages. Comprehensive coverage of online payments (deposit, full, partial, balance), saved card operations, tip payments, terminal payments, refunds, idempotency, webhook handling, and cash/gift card payments, GetBookingsByCreatedRange endpoint, scheduling exceptional hours validation, referral code registration, JWT revocation via JTI logout, and GDPR compliance (export handler cache states, anonymize_user child table scrubbing, export_all_user_data 16-section export, AnonymizeStaleGuestAccounts field scrubbing).
|
||||
**584/587 tests passing** (3 skipped) across 12+ packages. Comprehensive coverage of online payments (deposit, full, partial, balance), saved card operations, tip payments, terminal payments, refunds, idempotency, webhook handling, and cash/gift card payments, GetBookingsByCreatedRange endpoint, scheduling exceptional hours validation, referral code registration, JWT revocation via JTI logout, GDPR compliance (export handler cache states, anonymize_user child table scrubbing, export_all_user_data 16-section export, AnonymizeStaleGuestAccounts field scrubbing), and financial data retention (7-year expiry, 1-year post-anonymization buffer, aggregation correctness, refund handling, idempotency).
|
||||
- `handlers/auth` — Authentication (login, register, referral code validation, refresh, verification)
|
||||
- `handlers/bookings` — User booking flow, guest bookings, reservations, edit requests, discounts, closing hours validation, active booking limits, GetBookingsByCreatedRange
|
||||
- `handlers/payments` — Square payments (terminal, online, refunds, tips, saved cards)
|
||||
- `internal/square` — Square client dev mock tests
|
||||
- `handlers/admin` — Admin bookings, today view, users, services, GetBookingsByCreatedRange
|
||||
- `handlers/scheduling` — Working hours, exceptional groups, available hours, time blockers, exceptional hours validation
|
||||
- `handlers/scheduling` — Working hours, exceptional groups, available hours, time blockers, exceptional hours validation, financial data retention & aggregation
|
||||
- `handlers/services` — Service eligibility
|
||||
- `handlers/user` — User profile, guest creation, loyalty
|
||||
- `handlers/portfolio` — Image upload, listing, tags, filters
|
||||
|
||||
Reference in New Issue
Block a user