Justifactu — Technical Documentation¶

Contents¶

1. Overview
2. Architecture
3. Components
4. Data flow
5. Identifiers & naming conventions
6. Configuration
7. Running it
8. Testing
9. Known limitations
10. Glossary

1. Overview¶

• Justifactu automates matching and merging bills (facturas) with their corresponding payments (remeses) stored in SharePoint. Payments are renamed to the SAP ID found in their PDF content; bills, already named with their SAP ID, are matched against that renamed set and combined into a single justification PDF per pair.

• The pipeline runs every few days, keeping the volume of unprocessed documents manageable between runs. It operates independently of Finances or any other department's systems or schedule.

• Justifactu replaces a manual, repetitive process that previously took significant time each month from the people responsible for these justifications. Automating the matching and merging leaves only the final review as manual work.

2. Architecture¶

flowchart LR
    SP[(SharePoint)]
    OD["OneDrive-for-Linux
    (download-only sync)"]
    LC[(Local cache)]
    APP["Pipeline
    (app container)"]
    GAPI["Microsoft Graph API"]

    SP -- "sync: read-only" --> OD
    OD -- "mirrors to disk" --> LC
    LC -- "reads input" --> APP
    APP -- "writes: rename / delete / upload" --> GAPI
    GAPI -- "mutates" --> SP

• SharePoint serves as the only source of truth, and we use OneDrive-for-Linux to sync the relevant directories in download-only mode. This produces a local cache that serves as a temporal reference to do the main process. All of this runs in a Jenkins pipeline that manages the automation process, and it connects to various APIs throughout the run to do each step.

[!IMPORTANT] Core design principle: local files are a disposable cache, and only SharePoint changes are durable.

• This automation has two main containers: app and onedrive.
• app contains the installation of Justifactu itself. It's a one-shot script, meaning it doesn't run continuously.
• onedrive is a long-running service that contains the sync process with SharePoint, and every 5 minutes mirrors the contents of the SharePoint directory and applies any changes. Since SharePoint is the source of truth of this whole project, this process is download-only to avoid local filesystem interference.

3. Components¶

4. Data flow¶

flowchart TD
    A[Input arrives from SharePoint into local cache via OneDrive sync]
    B[Payments renamed to match inner SAP ID]
    C[Bills matched with payments sharing SAP ID]
    D[New PDF created merging paired files]
    E[Payment marked as processed]
    F[Processed bill deleted on SharePoint and local]
    G[Outputs and QA report uploaded to SharePoint]
    H[Confirmation mail sent with logs attached]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

5. Identifiers & naming conventions¶

• SAP ID: formed by the corresponding year and a 6-digit number. Bills are already named based on it, while payments receive theirs from their PDF content.

• File naming stages for payments:

flowchart LR
    A[Original bank filename]
    B[sap_id-P]
    C[sap_id-P_merged]

    A -->|rename_payments| B
    B -->|cleanup_processed_files| C

• SharePoint folder layout:
• _input, where all input files are. These files never
  • FACTURES, containing bills already properly named after their SAP ID.
  • Remeses, where the original payment files are and where they are renamed according to their SAP ID.
• _output
  • FACTURES+PAGAMENTS
  • 2025_FACTURES+PAGAMENTS
  • 2026_FACTURES+PAGAMENTS
  • ...
  • QA

5.1 Extracting the SAP ID from a bank payment PDF¶

Inside the payment PDF, the SAP ID sits next to a fixed marker: Fra. <SAP ID> (e.g. Fra. 2025003695). The parser (bills.py::parse_sap_id_from_bill) looks for that marker with the regex (Fra\.?\s+) followed by a 4-digit year and a 6-digit sequence — the period is optional and the whitespace is \s+ (matches spaces and newlines), which is what lets a single pattern cover both formats seen in practice:

The resulting output naming ({sap_id}-P.pdf, hyphen not underscore) is already covered by the "Payment file renaming" row in the table above.

5.2 Bill naming (Docuware)¶

Bills are downloaded from Docuware already named F {sap_id} (note the space after F) — bills.py::parse_bill_filename matches that exact shape. Since 20/05/2025, bills entering Docuware get the SAP document number written into the filename directly when the document is accounted for, replacing the older manual process (scan the paper bill, split the PDF, name it by hand). Bills logged before that date were retroactively renamed to the same convention as a one-off cleanup.

[!NOTE] Downloading bills from Docuware is not automated yet — it's currently a manual, roughly-weekly step. Automating it (nightly, or timed to coincide with the bank payment download) is a planned follow-up, not yet scheduled.

5.3 Matching and merging¶

A bill (F {sap_id}) and a payment ({sap_id}-P) are matched purely on a shared SAP ID — merge_bills_and_payments builds a payment map keyed by SAP ID and looks up each bill against it. When both exist, they're merged bill first, then payment into the single fused PDF named per the "Fused bill and payment pdf" row above.

[!NOTE] The original process description calls for deleting both source files after a successful merge — the bill and the bank payment PDF. The current implementation deletes the bill but only renames the payment (appending _merged) rather than deleting it, keeping it as a lightweight audit trail of which payment file produced which merge. Flagging this as a divergence from the original spec worth confirming is intentional, not an oversight.

[!WARNING] The original process notes describe payments being consolidated into a distinct _input/-stages/1/Pagaments folder after renaming, before the merge step. That doesn't match the SharePoint folder layout documented above, where renaming happens in place under Remeses with no separate consolidation folder. Flagging the discrepancy rather than resolving it here — worth checking which one is current.

5.4 What doesn't get matched¶

A bill with no matching payment isn't necessarily an error — it likely belongs to a payment method handled outside this flow: card payments, travel expenses, direct debits (domiciliacions), etc. Those go through a separate "own treatment" process (Phase 3 — see phase3_credit_cards.md) rather than the bank-remittance matching described here.

6. Configuration¶

Secrets¶

service/onedrive/conf/config¶

This file controls how OneDrive-For-Linux behaves, what it syncs, how often, and how the local mirror is kept in line with SharePoint. Its contents are the following:

[!WARNING] drive_id has to match exactly the real drive — do not modify manually without care.

compose.yml¶

It defines two services, which have healthchecks to ensure they are working properly, and with one dependency between them:

7. Running it¶

Local dev setup¶

1. Install dependencies:

make install

1. Run the pipeline:

make run CMD=""

[!TIP] make run alone just prints --help, since CMD defaults to that — always override it explicitly.

Common invocations¶

8. Testing¶

• In order to run the test suite, simply run the command make test on console inside the environment.
• Instead of using remote calls, we use mock calls to the different services in order to do relevant tests, since local files are not the source of truth of the project. The objective of this testing strategy is to follow the architecture structure.

9. Known limitations¶

[!CAUTION]

• Bill deletion after merge is irreversible. Depending on future needs, we might move deleted items to the trash instead of a direct deletion.
• Local mirror lag relative to SharePoint. Since OneDrive-For-Linux takes minutes to read, check, and download new content and erase the ones not mirrored in SharePoint, the process could incur some syncing lag. However, this pipeline is designed to be run once every few days, so it should have ample time to sync before each full run.
• No network timeouts on HTTP calls yet.
• Bill ingestion from Docuware is still a manual, weekly step (see §5.2) — not yet part of the automated pipeline.

10. Glossary¶