Skip to content

Historical Data Modelling⚓︎

Historical Data Modelling⚓︎

The analysis data model used by FIUs has changed significantly in the past 20 years as data volumes have increased and new technology solutions have become available. Historically financial intelligence data has been modelling using a cross reference schema, whereby a central cross reference table links keys in the following objects/tables: reports, names, addresses, accounts and identifications. The below diagram illustrates how these database objects have historically been linked.

Cross Reference Schema

There are a number of challenges with this approach, including:

  • To ensure the same database key is allocated to a; name, address, account or identification the objects/tables need to be serially loaded to ensure duplicates are not added. Serially loading data is obviously slower than modern parallel data loading approaches.
  • The process to analyse the data model will generally require mulitple (potentially expensive/slow and recursive) joins.
  • The processes required to load and unload data needs to maintain data integrity across the data model. For example, when deleting a report the system needs to check to see if the report's names, addresses, accounts and identifications are no longer used on any other report before they can be deleted.

Basically this type of solution works very well for smaller datasets which can be run on a single server, however begin to show performance degradation with high volumes of transactions.

Contemporary Data Modelling⚓︎

More recently with the advent of; highly parallelised processing and document based data stores, generic data structures have started to be used to model financial intelligence data. The below document identifies a simplified financial intelligence document structure (this is the data model used in FI-Comp). It provides a generic mechanism to store and search multiple names, addresses, accounts and identifiers for each party to the report and allows multiple parties to be associated to the same role. 

{
 "role": [
   {
     "roleType": "",
     "party": [
       {
         "account": [],
         "name": [],
         "address": [],
         "identification": []
       }
     ]
   }
 ],
 "transaction": {
   "amount": "",
   "direction": "",
   "transactionDatetime": ""
   "reference": ""   
 },
 "report": {
   "reportType": "",
   "processedDatetime": ""
   "reportNumber": ""
   "reporter": ""   
 }
}

The above data model is generic enough to support all financial intelligence report types, obviously the party roles are different across the report types. The below table provides a mapping to the roles on each report type.

Report Type Roles
Suspicious Matters suspiciousParty, otherParty
Threshold Transactions customer, agent
Cross Border Movements 💰 carryingPerson
Suspected Unregistered Entity 🚓 SuspectedUnregisteredParty
InternationalFundTransfer orderingCustomer, beneficiaryCustomer