jey_erp/BANK_INTEGRATION_PLAN.md

Bank Integration in jey_erp — Implementation Plan

Created: 2026-05-08 Status: Approved, not yet started Owner: Ali (jeysoft.mmc@gmail.com)

---

Context

We're building a universal "Bank Integration" tool inside jey_erp for importing bank statements from Excel files. It complements the existing kapital_bank app (which integrates with the BIRBank B2B API). The new tool targets banks that don't expose an API — clients export Excel statements from their internet-banking and we import them via column mapping presets.

The motivation for putting it in jey_erp (rather than a new app per bank): the user expects to support many banks over time, and a single shared dropdown Import From... on the Bank Transaction list is cleaner than each bank's app injecting its own button group.

The kapital_bank app stays mostly untouched — only its buttons move into the new shared Import From... group, and its BRT-extension JS is removed (the functionality moves to jey_erp as a universal version that works with both Bank Integration mappings and Kapital Bank Settings mappings).

---

Architectural Decisions

Aspect	Decision	Reasoning
Settings shape	Multi-record DocType Bank Integration (not Single)	Future-proofs for multiple banks. Initial focus is one bank, but structure is ready.
Coexistence with kapital_bank	kapital_bank kept as-is; only buttons move to new group; BRT-patch removed and replaced by universal version in jey_erp	"Don't refactor kb" was an explicit constraint. Each bank can have its own dedicated app/doctypes if needed.
Outbound payments	NOT covered by Bank Integration	Excel banks have no API; outbound stays kb-only.
Excel format	Hardcoded standard fields + flexible column mapping per bank (presets)	User maps their bank's columns to our standard fields. Popular banks get fixture presets (ABB, Pasha, Custom).
Import scope	BT-only (no PE/JE creation at import time)	Mappings are applied later in BRT via "Create & Reconcile". This matches kb's BT-list import flow (import_bulk_bank_transactions).
Two-step registry flow	Yes, like kb — registries fill at import time (option A from discussion)	Lets user run "Match similar customers/suppliers" before reconciliation. Avoids preview-leakage of half-mapped data.
Tracking doctype	NOT in iteration 1	Deduplication via direct query on Bank Transaction.reference_number + bank_account. Cleanup hooks rely on standard ERPNext relationships. Add later if cases emerge.
Bank Account ↔ Bank Integration link	Hidden read-only Dynamic Link on Bank Account, set by user in BRT	One source of truth on the BA, points to either a Bank Integration record or Kapital Bank Settings (Single). Allows resolving mappings for either system uniformly.
BRT mapping selection UX	Explicit select after Bank Account, with default loaded from BA's hidden field, written back on confirm	Trades one extra click for zero ambiguity. Default makes repeat use frictionless.
Fixtures vs default_data	Fixtures (fixtures/bank_integration_excel_preset.json)	Presets are global, not per-company; standard Frappe fixture loader fits.

---

Data Flow

[Excel file] ─→ Excel Parser (driven by Bank Integration Excel Preset)
                    │
                    ▼
              [Preview dialog]
                    │
                    ▼ user selects txns + clicks Import
                    │
       ┌────────────┴────────────┐
       ▼                         ▼
[Bank Transaction]      [Bank Integration Customer / Supplier / Purpose]
                         (status=New, awaiting manual mapping)
                                  │
                                  ▼
                  Bank Integration Settings tabs
                                  │
                user clicks "Match similar" / "Create unmapped"
                                  │
                                  ▼
                Customer/Supplier/Transaction Mappings populated
                                  │
                                  ▼
            BRT (Bank Reconciliation Tool):
              user picks Bank Account
              → default Bank Integration loaded from BA hidden field
              → user confirms / changes the select
              → selects BTs → "Create & Reconcile"
                                  │
                                  ▼
              mapping_resolver returns mappings
              (from Bank Integration record OR Kapital Bank Settings,
               based on Bank Account.bank_integration_type)
                                  │
                                  ▼
              Payment Entry / Journal Entry created and reconciled with BT

---

File Structure

jey_erp/
├── jey_erp/
│   ├── doctype/                          # standard Frappe location
│   │   ├── bank_integration/
│   │   │   ├── bank_integration.json
│   │   │   ├── bank_integration.py
│   │   │   └── bank_integration.js
│   │   ├── bank_integration_account_mapping/
│   │   ├── bank_integration_customer_mapping/
│   │   ├── bank_integration_supplier_mapping/
│   │   ├── bank_integration_transaction_mapping/
│   │   ├── bank_integration_customer/      # registry
│   │   ├── bank_integration_supplier/      # registry
│   │   ├── bank_integration_purpose/       # registry
│   │   ├── bank_integration_excel_preset/
│   │   └── bank_integration_excel_column_mapping/
│   │
│   ├── bank_integration/                 # python logic (separate from doctype/)
│   │   ├── __init__.py
│   │   ├── excel_parser.py               # parse xlsx by preset
│   │   ├── import_api.py                 # whitelisted: bulk_import_bt + helpers
│   │   ├── matching.py                   # match_similar_customers/suppliers/accounts
│   │   ├── creation.py                   # create_unmapped_customers/suppliers/accounts
│   │   ├── mapping_resolver.py           # abstract: resolve mappings by Bank Account
│   │   ├── create_reconcile.py           # universal "Create & Reconcile" backend
│   │   └── doc_events.py                 # on_trash/on_cancel hooks for PE/JE/BT
│   │
│   ├── public/js/
│   │   ├── bank_integration_form.js      # buttons on Bank Integration form
│   │   ├── bank_transaction_list.js      # NEW: "Import From..." dropdown with "Load from Excel"
│   │   └── bank_reconciliation_tool.js   # ENHANCED: select + Create&Reconcile toolbar
│   │
│   ├── fixtures/
│   │   └── bank_integration_excel_preset.json   # ABB / Pasha / Custom presets
│   │
│   ├── custom_fields.py                  # ADD: bank_integration_type/bank_integration on BA
│   └── hooks.py                          # ADD: doctype_js, doctype_list_js, doc_events

---

Doctype Schemas

Bank Integration (main, multi-record)

A close mirror of Kapital Bank Settings but as a regular DocType (one record per bank), and without is_active / default_login / card mappings.

Fields:

• bank_name (Data, reqd) — display name e.g. "Pasha Bank", "ABB"
• Tab "Settings":
  • default_company (Link → Company)
  • default_bank (Link → Bank)
  • similarity_threshold_customers (Percent, default 90)
  • similarity_threshold_suppliers (Percent, default 80)
  • similarity_threshold_purpose (Percent, default 70)
  • consider_azeri_chars (Check, default 1)
  • default_supplier_group / default_customer_group / default_payment_terms / default_territory (Link)
• Tab "Data" → subtabs Accounts / Customers / Suppliers / Purposes (HTML render with pagination, like kb)
• Tab "Mappings" → subtabs Account / Customer / Supplier / Transaction Mappings (child tables)

Child tables (mirror of kb-side)

Bank Integration Account Mapping: iban, account_label, currency (Link → Currency), bank_account (Link → Bank Account), gl_account (Link → Account)

Bank Integration Customer Mapping: bi_customer_name (Link → Bank Integration Customer), tax_id, erp_customer (Link → Customer), customer_group, territory, payment_terms, mapping_type (Select: Manual/Automatic)

Bank Integration Supplier Mapping: analogous to customer

Bank Integration Transaction Mapping: document_type (Select: Payment Entry/Journal Entry, default PE), purpose_keyword (Link → Bank Integration Purpose), payment_type (Select: Pay/Receive), counterparty_type (Select: Customer/Supplier), counterparty (Dynamic Link → counterparty_type), paid_from (Link → Account), paid_to (Link → Account), cost_center, multi_currency (Check), currency (Link → Currency), notes

Registries (top-level doctypes)

Bank Integration Customer: customer_name (Data, reqd), tax_id, iban, bank_code, status (Select: New/Mapped, default New), mapped_customer (Link → Customer), parent_bank_integration (Link → Bank Integration, makes registries per-bank)

Bank Integration Supplier: analogous

Bank Integration Purpose: purpose_keyword (Data, reqd), direction (Select: Pay/Receive/Both), status (Select: New/Mapped), parent_bank_integration (Link → Bank Integration)

Excel preset

Bank Integration Excel Preset:

• preset_name (Data, reqd, unique)
• description (Small Text)
• header_row (Int, default 1) — 1-indexed row containing column headers
• skip_rows_after_header (Int, default 0)
• date_format (Data, default "%Y-%m-%d") — Python strftime format
• amount_mode (Select: "Single column with sign" / "Separate debit/credit columns" / "Single column + direction column", default "Separate debit/credit columns")
• column_mappings (Table → Bank Integration Excel Column Mapping)

Bank Integration Excel Column Mapping: excel_column (Data — header text in Excel), standard_field (Select: date/reference_number/counterparty/counterparty_tax_id/counterparty_iban/amount/debit/credit/direction/currency/purpose/description), notes

---

Custom Fields on Bank Account

Added in jey_erp/custom_fields.py (existing create_custom_fields()):

{
    "doctype": "Bank Account",
    "fieldname": "bank_integration_type",
    "fieldtype": "Select",
    "label": "Bank Integration Type",
    "options": "\nBank Integration\nKapital Bank Settings",
    "hidden": 1, "read_only": 1,
    "insert_after": "bank_account_no"
},
{
    "doctype": "Bank Account",
    "fieldname": "bank_integration",
    "fieldtype": "Dynamic Link",
    "label": "Bank Integration",
    "options": "bank_integration_type",
    "hidden": 1, "read_only": 1,
    "insert_after": "bank_integration_type"
}

The Dynamic Link resolves to either Bank Integration records or the Kapital Bank Settings Single (whose name is just "Kapital Bank Settings").

---

Backend Modules

bank_integration/excel_parser.py

• parse_excel(file_url, preset_name) → list[dict]
• Uses openpyxl (already a Frappe dependency)
• Returns dicts in standard form: {ref_no, date, counterparty, contr_voen, contr_iban, amount, currency, drcr, purpose, description}
• Handles 3 amount modes: single signed column / separate debit+credit / amount+direction column

bank_integration/import_api.py

• @whitelist parse_excel_for_preview(file_url, preset_name, bank_integration) — parses and returns preview, also dedupes against existing BT by reference_number + bank_account
• @whitelist import_bulk_bt(txn_list, bank_integration, bank_account) → enqueues background job
• _process_bulk_bt_import(txn_list, bank_integration, bank_account, user):
  • For each txn:
    1. Upsert Bank Integration Customer/Supplier (status=New if new) — only when counterparty present
    2. Upsert Bank Integration Purpose
    3. Insert + submit Bank Transaction (party left empty — filled later via Create&Reconcile)
  • frappe.publish_realtime events: bi_bt_import_progress / bi_bt_import_complete

bank_integration/matching.py

Mirrors kb: match_similar_customers, match_similar_suppliers, match_similar_accounts. Uses SequenceMatcher + a normalize helper (Azerbaijani translit). Consider extracting _normalize and _AZERI_MAP to a shared place if both apps end up using it.

bank_integration/creation.py

Mirrors kb: create_unmapped_customers, create_unmapped_suppliers, create_unmapped_accounts — creates ERPNext entities and updates the corresponding mapping rows.

bank_integration/mapping_resolver.py (key abstraction)

def resolve_mappings_for_bank_account(bank_account_name):
    """Returns dict: {transaction_mappings, customer_mappings,
    supplier_mappings, settings} or None if not configured.
    Reads from Bank Integration record or Kapital Bank Settings,
    depending on bank_integration_type field on Bank Account."""
    ba = frappe.get_doc("Bank Account", bank_account_name)
    if ba.bank_integration_type == "Bank Integration" and ba.bank_integration:
        bi = frappe.get_doc("Bank Integration", ba.bank_integration)
        return {...build from bi child tables...}
    elif ba.bank_integration_type == "Kapital Bank Settings":
        kb = frappe.get_single("Kapital Bank Settings")
        return {...build from kb child tables...}
    return None

bank_integration/create_reconcile.py

Backend for the universal "Create & Reconcile" button. Pulls logic from kb's existing _import_one_transaction (the PE/JE creation arm) but generalised through mapping_resolver. Operates on already-existing Bank Transactions selected by user.

bank_integration/doc_events.py

• on_trash_bank_transaction / on_cancel_bank_transaction — initial implementation: log + sanity checks. No tracking doctype yet, so no cascading deletes.

---

Frontend Modules

public/js/bank_transaction_list.js (new)

• Hooks into frappe.listview_settings['Bank Transaction'].onload
• Creates Import From... ▼ button group with item Load from Excel
• Click flow: dialog → choose Excel Preset → choose Bank Account → upload file → call parse_excel_for_preview → preview table (mirror of kb's showTransactionSelection) → click Import → enqueue + realtime progress + final summary

public/js/bank_reconciliation_tool.js (extend existing)

• Keep current "Bank Transaction column" patch (already in jey_erp)
• Add checkboxColumn: true to DataTable rebuild (port from kb)
• Add a Bank Integration select to the BRT header that fills from all Bank Integration records, and additionally adds a Kapital Bank option if kb is installed
• Default value: read from Bank Account.bank_integration_type / .bank_integration via frappe.db.get_value when user picks a Bank Account
• On change: call whitelist update_bank_account_default_bi(bank_account, type, name) to persist to BA's hidden fields
• Floating toolbar Create & Reconcile button → calls create_reconcile.create_documents_for_bts(bt_names, bank_account)

public/js/bank_integration_form.js (new)

A near-copy of kapital_bank_settings.js button setup — Match similar / Create matching / Add unmapped buttons under Accounts / Customers / Suppliers / Purposes groups.

---

Fixtures

jey_erp/fixtures/bank_integration_excel_preset.json — initial set:

• ABB Standard Statement (column mappings for ABB internet-banking export)
• Pasha Business Statement
• Custom Template (empty column_mappings — user fills in dialog)

Register in hooks.py:

fixtures = [
    {"doctype": "Print Format"},
    {"doctype": "Bank Integration Excel Preset"},
    {"doctype": "Bank Integration Excel Column Mapping"},
]

---

hooks.py Changes (jey_erp)

doctype_list_js = {
    ...
    "Bank Transaction": "public/js/bank_transaction_list.js",
}

# bank_reconciliation_tool.js already registered — file is extended in place

doc_events = {
    ...
    "Bank Transaction": {
        "on_trash": "jey_erp.bank_integration.doc_events.on_trash_bank_transaction",
        "on_cancel": "jey_erp.bank_integration.doc_events.on_cancel_bank_transaction",
    },
}

In custom_fields.py add the Bank Account block above. Ensure after_migrate_combined() calls the field creator (already does, via existing create_custom_fields).

---

Changes in kapital_bank (minimal)

1. client/bank_transaction.js — replace:

   listview.page.add_inner_button(__('Import'), ..., __('Kapital Bank'));
   

   with:

   listview.page.add_inner_button(__('Load from Kapital Bank'), ..., __('Import From...'));
   

2. client/payment_entry.js — same change.

3. client/bank_reconciliation_tool.js — delete the file.

4. hooks.py — remove "Bank Reconciliation Tool": "client/bank_reconciliation_tool.js" from doctype_js.

5. No setup.py change in kb itself. Migration of existing kb-mapped Bank Accounts (proper bank_integration_type/bank_integration values for them) happens in jey_erp's after_migrate: walk Kapital Bank Account Mapping.bank_account rows and set the BA's hidden fields.

---

Implementation Phases

#	Phase	Depends on
1	Create all 10 Doctypes — JSON schemas + __init__.py + empty .py controllers	—
2	Custom fields on Bank Account via custom_fields.py	—
3	excel_parser.py + smoke test with one sample.xlsx	1
4	import_api.py + JS preview dialog + bulk import → creates BT	1, 2, 3
5	bank_transaction_list.js — "Load from Excel" item under "Import From..."	4
6	bank_integration_form.js + matching.py + creation.py (Settings buttons)	1
7	mapping_resolver.py — abstraction over the two mapping sources	1
8	create_reconcile.py — backend for universal Create&Reconcile	7
9	bank_reconciliation_tool.js — extension: checkboxColumn + select + toolbar	8
10	Migration script in jey_erp after_migrate: walk Kapital Bank Account Mappings → set bank_integration_type/name on their Bank Accounts	2
11	Minimal kapital_bank edits: rename buttons + delete BRT file + drop hook	1–9 ready
12	Fixtures: bank_integration_excel_preset.json (ABB + Pasha + Custom)	1
13	End-to-end test: ABB Excel → BT → Settings (mappings) → BRT (Create&Reconcile) → PE/JE	all done

---

Out of Scope (Iteration 1)

• Tracking doctype (Bank Integration Transaction) — dedupe via direct Bank Transaction.reference_number + bank_account query. Add later if cleanup-cascade cases emerge.
• Outbound payments (Send/Check/Cancel) — explicitly excluded; outbound stays kb-specific.
• Card statements support — Excel banks usually only provide account statements. Can be added as a preset mode if needed.
• Other BRT enhancements beyond Create & Reconcile.
• Sharing _normalize / _AZERI_MAP between apps — duplicate for now, refactor if both apps continue to evolve in this area.

---

Notes for Future Self

• kapital_bank has TWO import paths (visible in bank_api.py): import_bulk_transactions (creates PE/JE/BT through full mappings, called from client/payment_entry.js) and import_bulk_bank_transactions (creates only BT, called from client/bank_transaction.js). Both are active and not commented out. Our new tool mirrors the BT-only path.
• BRT in BRT — the data table is rebuilt by kapital_bank/client/bank_reconciliation_tool.js to add a checkbox column. The current jey_erp/public/js/bank_reconciliation_tool.js adds the "Bank Transaction" column. After this work, jey_erp owns the entire BRT extension, and kb owns nothing on BRT.
• The Dynamic Link trick on Bank Account avoids needing kb to be aware of jey_erp at all. The mapping resolver in jey_erp reads from kb Settings directly when the type is Kapital Bank Settings.
• Fixture-driven Excel presets means adding support for a new bank = editing one JSON file in jey_erp + running migrate.