Requirements in Context Part 8: Derived Data
It’s the job of Business Information Systems to know stuff. They are able to know stuff either because they have been told the stuff or because they have been told how to derive stuff from the stuff they know.
This article is about deriving stuff within the context of detail functional requirements.
Contexts For Derived Data
In parts 5 and 7 of this series we saw three detail requirements contexts that can involve derived data:
- Interfaces
- Reports
- On-Line User Interfaces
There is a fourth context not yet discussed – Internal Processes.
Content Detail for Internal Process Requirements
An internal process is a cross between a report and an input interface. It’s like a report in that it runs at a scheduled time and has selection criteria that finds all of the records it needs to work with. Having been initiated and with the selected records the internal process behaves like an input interface, working its way through those records with the result being new records inserted into the database and/or existing database records updated.
An example of an internal process for a Banking Information System might be “Credit Customer Accounts With Earned Interest.” Scheduled to execute daily, the process would:
- Find all of the Customer accounts that are due to be paid interest.
- For each account found
- Look up the rate of interest that applies to the account type
- Calculate the amount of interest earned
- Create a credit transaction for the customer containing this amount
- Post the transaction (resulting in the account balance being updated with the interest amount)
A detail requirement for the “Earned Interest” internal process would follow the same pattern as for reports and interfaces described in Part 5 – a single detail requirement that names the process and references its details recorded in an Internal Process Spec. The drafting of this requirement either as a User Story or Shall Statement is left as an exercise for the reader (pick your preferred format). Extra credit will be given for including acceptance criteria.
As indicated in the previous articles, the detail requirement does not mention each field involved. Rather the detail, including identifying individual fields involved, is relegated to a template-based specification appropriate each of the context types. In cases where a field represents derived data, a description of the derivation should be included as a comment associated with the field. We will see examples of these below.
Data Derivation Descriptions
I like to think of derived data, and their corresponding descriptions as being one of the following three types:
- Grown Locally
- Phone a Friend
- Build an Ark
Grown Locally – With this type of derivation, the values needed for the derivation can be found amongst the other fields in the spec for the given context. So when you are describing the derivation, you are referencing fields that are listed right there in the spec.
Example 1 – Producing an Invoice, the query has returned all of the line items being included in the report. Each Line Item includes the “Total Amount” for that line. So when it comes time to derive the overall “Invoice Total” field value the necessary items to sum up are all available. If the invoice can involve additional fees or charges, then the derivation would involve them as well.
Given fields called “Line Item Total” and “Shipping Charge” listed in the report spec, the derivation description in the comments field for “Invoice Total” might be:
Sum of Line Item Total values plus Shipping Charge
This description is a verbal version of a mathematical formula. The equivalent mathematical version would look something like this:
Invoice Total = Sum(Line Item Total) + Shipping Charge
In cases where a derivation formula involves lots of fields and/or more complex calculations it can be clearer to use abbreviations:
NOTE: I’ve kept the example that uses abbreviations the same as the previous two forms. It would probably not be necessary to resort to this style until the number of fields or the complexity of the formula gets larger. When you encounter these, you’ll recognize the benefit of using both the mathematical form and abbreviations.
Example 2 – Presenting Customer Name on a report or screen as a single field derived by stringing together the locally available non-derived fields Title, First Name and Last Name. The derivation description in the comments field might be one or the other of the following:
Derivation descriptions should include examples wherever possible. Examples help confirm what the business is wanting and provide guidance for development and testing. Note that both description versions in Example 2 expect there to be spaces between the concatenated fields. The first description didn’t mention anything about spaces, but the included example showed that spaces were part of the derived result. The second description indicated spaces explicitly. Even so, an example is worth including.
Both Example 1 and 2 above involve simple formulas describing how to derive a field from existing data using arithmetic operators or via string manipulation. Derivations can also involve logical operators such as greater than, equal to and less than either as simple pairs or combined with And, Or and Not. The following variation on Example 2 involves a single condition. The additional fields are assumed to be ‘grown locally’ as well:
Example 3 – Presenting either a Person Name or an Organisation Name depending on Customer Type:
Phone a Friend – This type of derivation involves one or more values that are not amongst the fields in the context. They need to be sourced from the information system’s database.
Example – Grocery item purchases being captured using an on-line POS terminal. Scanning the barcode of each item identifies the product but the “Current Unit Price” needs to be looked up. The derivation description in the comment for this field might be:
Use the product’s BarCode and the current date/time to find the Product’s Current Unit Price
Note that his description is very ‘what, not how.’ No mention is made of what table contains the field that will have the value needed. Those details should be left to the solution designers. The derivation description is best left to what the business knows at the time the derived value is needed. If it turns out that additional details are required to find a result, then the SME should provide the correct information for the BA to include in the description.
Data derivations can be a mixture of Grown Locally and Phone a Friend. If a derivation formula involves one or more fields that require lookup, then the description might best be split into multiple steps. The Phone a Friend steps should be described first so that the values become available. They can then be used as if they were ‘local’ in the remainder of the derivation.
Build an Ark – When derivations go much beyond a simple formula, a small number of database lookups and/or conditions the form of the derivation is called an algorithm. The comment field of a detail requirement spec is not an appropriate place to describe an algorithm. It really needs its own separate specification [template] and its own [single] requirement.
Content Detail For Algorithm Requirements
Algorithms will typically have a business activity name such as “Determine Price” or “Determine Eligibility”. This activity would very likely appear as its own ‘box’ in at least one business workflow. Descriptions of algorithms fall into the category of ‘A picture is worth a thousand words.’ In this case, that means providing something like a UML Activity diagram. Those types of diagrams are designed to show sequences of steps and branches based on decision points (i.e. conditions). The pre-UML name for an Activity diagram is “Flow Chart.” The diagram (and supporting text and definition of fields used) represents the details of the functional requirement.
Like the contexts discussed previously, a single requirement should be used to represent the detail. An example based on our ‘pricing’ activity would be as follows:
User Story Format:
As a Customer, I want my contracted rates used when pricing purchases because those rates are legally binding during the term of my contract.
Shall Statement Format:
The system shall use the currently contracted rates for a customer when calculating the price for purchases made by that customer.
Both requirements should include acceptance criteria:
Acceptance Criteria
- Given the current contracted rates for a customer
- When the customer purchases a product
- Then the price charged for that product is the result of those rates being applied to the “Customer Pricing” algorithm described in Appendix C.
The activity diagram and accompanying description are produced by the BA in conjunction with the SME that understands the algorithm. Some of the fields involved will be ‘locally grown’ and some will involve ‘phone a friend’.
Requirements In Context – Just Know It
This series has been about business information systems and how it is their responsibility to know what users need to know to support their business processes and activities. What the system ultimately knows is field-level detail. This detail is sourced from input interfaces, on-line interfaces, internal processes, and algorithms. Providing that detail to users and other systems is through reports, output interfaces, and on-line user interfaces. Detail requirements, at the end of the day, is about field level details in context.
The next and final article will summarise the key ‘take-away’ points from the series. Following that, there will be a quiz.