Register sales / purchase / VAT bulk imports with the background-tasks
registry: emit per-item progress, honour cancellation requests between
items, and finish/cancel the job so the global widget can track them.
Add a Redis-backed, per-user registry of in-flight background jobs and a
desk-wide "Background Tasks" widget (bottom-right) that reads progress and
can request best-effort cancellation. Wire the widget bundle via
app_include_js.
Enable show_title_field_in_link so E-Taxes Presented Certificate renders by
its certificate number (e.g. "Az1730537") in link/multi-select controls
instead of the raw oid docname. Naming (field:oid) is unchanged so the
e-taxes upsert (keyed by oid) keeps working.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "short" queue caused the import to run concurrently with the
post-setup desk load and deadlock on the tabSessions row. Document that
default is intentional; the queue arg stays for callers that need it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- import_bulk_employees gains a `queue` arg so callers can route the job
off the contended default queue. The bar that hangs on "Preparing
import..." for ~a minute post-setup is queue wait: a single `bench
worker` drains short→default→long, so the import sat behind the
post-setup backlog. The wizard now enqueues on "short".
- Emit a "connecting" progress tick the instant the worker starts, so the
bar leaves the "queued" seed and shows it's actually running (vs still
waiting) before the first detail fetch.
- Org selection: drop the Role column ("Hüquqi şəxs" for every row) — the
name + VÖEN are enough. The role value is still sent to the API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bar refilled 0->100% once per phase: the parallel fetch phase (the
slow, network-bound one) reported 0..N, then the sequential save phase
reported 0..N again. The client's monotonic guard then pinned the bar at
100% for the whole save phase, so real per-employee progress (with names)
was only visible at the very end, and the bar looked stuck at 0% during
the long fetch.
Report one overall 0-100 percent across both phases (fetch = 0-50, save =
50-100) from the worker and drive the bar from it; show per-phase "N of M"
+ a clear phase label ("Fetching details" / "Saving employees") in the
detail line. Poll every 1s for snappier updates. Bar now advances
continuously and is meaningful from the first tick.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the alarming "Reconnecting to import in progress..." fallback with
"Importing employees, please wait...". It showed whenever the bar had no
count yet — briefly before the first tick, and for the whole run when the
server returns no progress (e.g. a worker still on old code). The wording
made a working import look broken.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Employee list-view import bar relied solely on transient
publish_realtime("amas_import_progress") events. After the setup wizard
redirects to /app and the list reattaches, those in-flight events are
already gone — so the bar froze at "Reconnecting..." (0%) and employees
appeared all at once on the final refresh.
Persist each progress tick to Redis (_set_import_progress, exposed via
get_amas_import_status) and add a 2s polling fallback in employee.js that
drives the bar monotonically and detects completion even when the
realtime amas_import_complete is missed. Closing the bar still never
cancels the enqueued worker job.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "select records to import" dialogs (VAT operations, sales/purchase invoices,
AMAS employees) wasted vertical space: a fixed 500px table plus a context alert
pinned to the bottom forced scrolling to reach the content and the buttons.
Rework them to the journal_entry pattern: the records table is now the single
scroll area sized to the viewport (max-height: calc(100vh - 360px)) with a
sticky header, and the top count alert + bottom context alert are merged into
one compact header row. Affects journal_entry, sales_order, purchase_order,
sales_invoice and employee (AMAS) selection dialogs. No logic/handler changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The required Link fields default_item_group / default_uom had no field-level
default and nothing ever populated them, so every fresh base started with them
empty (had to be filled by hand). Add setup_defaults.ensure_settings_defaults()
that sets default_item_group="All Item Groups" and default_uom="Nos" — but only
when the field is blank AND the target record exists. Idempotent; wired into
after_install and after_migrate so it never overwrites an admin's choice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under jey_theme/Bootstrap, soft .hide() / frappe.hide_progress() does NOT
reliably remove a modal — it sticks and the next dialog stacks on top of it
(e.g. the ASAN auth "success" message stayed under the date/org picker).
The loading-spinner Cancel button only changed its label, never closing.
- dialogs.hide(): force modal('hide') + remove() + orphaned-backdrop cleanup
(guarded by .modal:visible); no longer resets cancelLoading there so in-flight
pagination callbacks still see the cancel and bail out
- Cancel button now actually closes the spinner
- add import._closeProgress() and route every frappe.hide_progress() through a
robust force-close: purchase/sales order, sales invoice, journal entry,
employee/AMAS (closeAmasProgress), and the E-Taxes unit list
- E-Taxes list-view loading dialogs (items/customers/suppliers/units) and their
success paths force-close the same way
Applies the fix uniformly across all ETaxes JS modules and list views.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Realtime import progress modal could hang when a late 'progress' event
arrived after 'complete' (hide_progress nulls cur_progress, show_progress
then recreates an unmanaged modal). Guard every handler with an
importFinished flag and force-close via modal('hide')+remove()+backdrop
cleanup across company/employee/journal_entry/purchase_order/sales_*.
Also: show VAT operation İzahat (explanation) column in the selection
table and on the Journal Entry list; auto-create missing "Classification
code" records from taxCodeInfo so the Link field always resolves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hash fast-path compared the JSON file hash to a stored value and exited
early when unchanged. But it only tracked the *file*, not the *table*: if rows
went missing from the DB while the JSON stayed the same, the sync skipped
entirely and never restored them.
Always run the full diff now. It costs one SELECT plus an in-memory compare —
~0.1s for 13k E-Taxes Item Group rows — so there's no reason to skip it. The
diff is driven by DB state, making the sync self-healing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These doctypes are now populated exclusively from master_data JSON via the
hash-diff sync, so block create/write/delete in the UI and API by dropping
those permissions. The sync itself writes at the DB level (db_insert /
set_value / delete_doc with ignore_permissions) so it is unaffected. Also
marks classification_name read_only and drops allow_import/allow_rename.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Frappe imports every .json under app/fixtures/ on every migrate via
DELETE+INSERT (force=True in import_doc), so 13,269 E-Taxes Item Group rows +
116 Classification code rows added ~35s to each migrate even though the
fixtures hook was empty. Replace with a hash-diff sync that fast-paths via a
stored file hash and otherwise applies only the delta (insert new with
db_insert() bulk path for first-run, update changed via set_value, smart-delete
removed with disabled fallback). Also removes the now-redundant
master_data_loader.py (install-only bulk insert that the sync supersedes) and
the original fixture files so they stop being imported on migrate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the obl-pact-list integration entirely: the doctype directory,
loader/bulk-loader entries, REFERENCE_SECTIONS row, reference-list
allow-list entry, and the existence-check guard on Presented
Certificate's sazis_code. sazis_code is now a plain Data field —
no Link target to validate against.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The data loading dialog was opened via frappe.msgprint, which reuses the
shared frappe.msg_dialog. We then rewrote .modal-body via $.html(...) on
each progress tick, destroying the DOM that frappe.show_progress also
relies on. After hide(), Bootstrap left a modal-backdrop behind and the
whole page became unclickable until reload.
Switch to the dedicated show_loading_dialog_settings /
update_loading_message_settings / hide_loading_dialog_settings dialog
that already exists in this file, so msg_dialog is never touched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces "Azeri" / "Azerbaijani" / "Azerbaijan" with the native form
"Azərbaycan" in labels, descriptions, comments and docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Recent ERPNext rejects assigning a group-type Customer Group to a Customer
("Cannot select a Group type Customer Group"); customer_group is also no
longer mandatory. The integration fell back to "All Customer Groups" (a
group node), so every auto-created party failed.
- add invoice_az/utils.py:resolve_customer_group() — returns the configured
group only if it exists and is not a group node, else None
- api.py / vat_api.py: skip customer_group when no valid non-group value;
create the Customer without a group instead of forcing "All Customer Groups"
- add link_filters (is_group=0) to all Customer/Supplier Group pickers in
E-Taxes Settings and the mapping child tables
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Convert add_menu_item to add_inner_button on E-Taxes Customer/Supplier/
Item/Unit lists, Purchase Order, Sales Order, and Journal Entry list
views so the load/import actions are visible directly instead of being
hidden under the "..." menu.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bulk import is now two-phase:
Phase A — parallel network. Up to BULK_IMPORT_PARALLELISM (=10)
ThreadPoolExecutor workers call get_employee_detail() concurrently.
Workers run in stateless mode via the new _post_amas helper and the
_state= parameter threaded through make_amas_request and the seven
detail helpers (get_edit_form_data, get_staff_data, get_person_data,
get_address_data, get_doc_main_data, get_contract_attachments,
download_contract_file). They share an in-memory snapshot of the
ƏMAS session/csrf/cookies and never touch the DB. If any worker is
rejected with HTTP 419 or a response-level CSRF code the whole
import aborts cleanly with a hint to lower BULK_IMPORT_PARALLELISM.
Phase B — sequential DB writes. create_single_employee_from_amas
takes a new prefetched_detail kwarg; the worker loop passes the
Phase-A payload through so the per-employee 7-call fetch is skipped.
Frappe ORM is not thread-safe, so writes stay sequential — that's
where the previous flow spent most of its time anyway.
Concurrency control:
- Asan Login picks up two new hidden Check fields,
amas_import_running and amas_import_cancel_requested.
- import_bulk_employees refuses to enqueue if the running flag is
already set, returning {success: False, already_running: True,
message}. The flag is set BEFORE frappe.enqueue (atomic guard
against fast double-clicks) and cleared in the worker's finally
block no matter how it exits.
- New whitelisted endpoints get_amas_import_status and
cancel_amas_import expose the flag to the UI. The worker checks
the cancel flag between Phase A futures and between Phase B
iterations.
Realtime events upgraded:
- amas_import_progress now carries phase: "fetch" | "save".
- amas_import_complete carries cancelled and aborted flags.
Frontend (employee.js):
- Bootstrap-4.6 stacked-modal fix re-applies modal-open class on
body when a nested modal closes (cancel-confirm over progress)
so the underlying backdrop isn't orphaned.
- reattach_amas_import_if_running re-binds the realtime listeners
on listview onload by polling get_connected_asan_logins +
get_amas_import_status, so refreshing /app/employee while an
import is in flight still shows the progress bar + Cancel button.
Plus the side fixes from the same session: connect_amas now opens
the org-picker dialog on success, error humanizer for common ƏMAS
codes, skip the dashboard prefetch when the cached CSRF is still
good. Schema changes need bench migrate.
- Always show "Select ƏMAS Organization" dialog after MyGovID re-auth in
the Employee → Load from AMAS flow. Previously auto-picked the first
cert, which on multi-cert users was the personal cert without rights
to the employee report — caused permission_error.
- humanize_amas_error() translates raw codes ('permission_error',
unknown) into user-facing messages. Permission failures return
permission_error: True so the frontend can offer a Reconnect button
that runs full re-auth + org selection.
- make_amas_request: drop the eager /core.dashboard fetch before every
call. Use the cached CSRF token + cookies (already updated from
Set-Cookie on each response). On real CSRF rejection (HTTP 419 or
response code CSRF/TOKENS_ARE_NOT_SAME/CSRF_TOKEN_MISMATCH), refresh
once and retry. Same simplification in get_amas_accounts.
- _process_bulk_employees_import: commit once after the loop instead of
per row, per CLAUDE.md guidance.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move E-Taxes Item Group (~13k rows) and Classification code (~116 rows)
out of the `fixtures` hook. Re-syncing these on every `bench migrate` was
the dominant cost of the fixture sync step.
Add invoice_az.master_data_loader.load_master_data() that bulk-inserts
from the existing JSON files; it runs once via after_install and is
idempotent (skips rows that already exist by primary key) so it can be
re-invoked from the bench console to refresh master data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
load_company_bank_accounts now makes every non-closed e-taxes bank
record usable inside ERPNext, not just a read-only E-Taxes Bank Account
cache row:
- Bank (global, one per unique bankName) — idempotent insert.
- GL Account under parent "223 Bank hesablaşma hesabları", named
"<CUR> <IBAN>", account_type=Bank, account_currency from e-taxes.
Parent discovered by account_number=223 with a name-prefix fallback.
- Bank Account linking Bank + GL Account + Company, is_company_account=1,
iban + bank_account_no = the AZ IBAN.
Closed accounts (status=C) are left at cache-row only; no native
records are created for them.
After the loop, unused AZ CoA placeholder accounts (names like
"AZN AZXXXXXXXXXXXXXXXXXXXXXXXXXX" under the bank group) are deleted
so the chart isn't cluttered with unmatched templates. Per-record
failures go to the "E-Taxes Bank Native Materialize" error log and
don't abort the overall load.
Adds a "Data Loading" dialog on Company form that pulls 6 types of
taxpayer reference data from E-Taxes into dedicated doctypes, plus
in-form summary tables inside the Tax Policy tab.
New DocTypes (all with company link, Accounts User read-only,
System Manager full CRUD):
- E-Taxes Object (taxpayer registered locations)
- E-Taxes Cash Register (kassa aparatları with object FK)
- E-Taxes POS Terminal (with auto-derived object from registration number)
- E-Taxes Bank Account
- E-Taxes Obligation Pact (sazişlər / oil fields)
- E-Taxes Presented Certificate (with sazis_code FK to Obligation Pact)
Backend (invoice_az/company_api.py):
- 6 per-doctype loaders + shared _auth_headers / _etaxes_request helpers
implementing CLAUDE.md §10 401-retry-once pattern
- Cash Register / POS Terminal loaders walk the hasMore pagination
- load_company_data_bulk enqueues a background job that runs selected
loaders sequentially and streams progress over Socket.IO
(company_data_loading_progress / _complete events)
- get_company_reference_list exposes first N rows per doctype for the
Tax Policy tab tables
Frontend (invoice_az/client/company.js):
- Full ETaxes auth module inlined (CLAUDE.md §2) so ASAN Imza re-auth
works from Company form without relying on other forms' state
- "Data Loading" button opens a dialog of checkboxes (DATA_LOADERS
declarative array) — one bulk call, one floating show_progress bar
- Six reference tables rendered inside Tax Policy tab sections,
pattern copied from e-taxes_settings (card-section + Refresh +
"View All" only when total > rows)
CLAUDE.md:
- Rewrote §3 backend template with 401 retry + explicit commit
- Added §10–§16: 401 auto-refresh, two response shapes,
db.commit() standard, log title convention, frontend unauthorized
handling, field naming for reference vs fixture doctypes,
Socket.IO background job pattern
hooks.py:
- Register client/company.js under doctype_js["Company"]
.gitignore:
- Exclude *.har (may contain Bearer tokens / PII)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vat_18_percent_with_amount stores the total INCLUDING VAT (e.g. 118),
but the e-taxes API vat18 field expects just the tax amount (e.g. 18).
Also calculates net cost/pricePerUnit for VAT-inclusive pricing (ƏDV daxil 18%).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Checkbox values (load_items, load_units, load_customers, load_suppliers)
were collected in the dialog but never passed through the loading pipeline,
causing all data types to always be created regardless of user selection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ali