What makes a good clinical calculator?

A clinical calculator uses a patient’s data to a ‘calculation’. The result is used by clinical staff in delivering patient care, and by patients to help them understand their condition.

This blog explores some of the benefits and risks of using clinical calculators and makes a number of recommendations for designing and integrating calculators with the electronic patient record (EPR).

What are clinical calculators?

Clinical calculators can be found on a variety of medical web sites and an increasing number are now available as free mobile apps.

Clinical systems may also include medical calculators and some EPR vendors provide the ability for organisations to create their own clinical forms, calculations and decision support to assist in patient care.

All clinical calculators follow a similar format. They are designed to take input of one or more clinical parameters (age, sex, ethnicity, height, weight, temperature, test results etc) and use a calculation (algorithm) to produce a result or score.

Examples include:

  • BMI (body mass index)
  • eGFR (estimated glomerular filtration rate)
  • AKI (acute kidney injury score)
  • NEWS2 (national early warning score)
  • AMTS (abbreviated mental test score)
  • PURPOSE T (pressure ulcer risk primary and secondary evaluation tool)
  • P-POSSUM (physiological and operative severity score for the enumeration of mortality and morbidity)

Some calculations combine clinical coding with a prognostic indicator score

  • TNM/AJCC (cancer staging)
  • ACE27 (comorbidity score)

Why use clinical calculators?

Clinical scoring systems are well established in medical practice and have been shown to add value in developing clinical guidelines, monitoring population health and evaluating clinical outcomes. They have been adopted in areas where an aggregated score has been shown to better predict prognosis than the clinical parameters alone.

For an individual patient, the calculated score may indicate a level of risk. This risk is usually based on evidence from clinical trials and population studies. Professional standards bodies provide tailored guidance and clinical decision support appropriate to the level of risk.

Traditionally, scoring systems have been implemented using paper charts and forms. The standard NEWS2 paper chart requires users to plot each of the observations (pulse, temperature, blood pressure etc) on a scale and manually add up the scores to calculate the total. The user will then look up that total score in the guidelines to determine the appropriate action.

Studies have shown that there can be errors in data entry using paper charts and the scores can be calculated incorrectly in up to 30% of cases. As we might expect, these studies also show that well designed electronic calculators can improve clinical safety and efficiency by automatically calculating the score and providing the appropriate clinical guidance support.

What are the risks with using calculators?

The challenge arises in working out how to implement medical calculators safely in clinical practice. There are clinical governance considerations and also safety regulations that may require some calculators to be classed as medical devices.

Clinical risk management training provides a framework for identifying potential hazards and the contributing factors that may potentially result in patient harm. In this context, a number of risks identified relate to using standalone calculators that are not integrated with the clinical systems and patient record.

These risks include:

  • The clinical data may not be entered correctly resulting in a false reading
  • It is possible the calculation may be incorrect, producing a false result.
  • The result may not be captured routinely as part of the clinical patient record
  • There is often no record kept of how the calculated value/score was derived
  • No audit trail is maintained of who performed the calculation and when
  • There may be no permanent record of the decision support given at the time
  • Standalone calculations often provide no longitudinal history and charting of results
  • There is limited ability to integrate the calculation with clinical workflow and trigger subsequent actions.
  • The help, support and training provided is variable and may not be tailored to local needs
  • Third party apps may not be properly CE marked for clinical use or provide a manufacturer’s clinical safety report to support a local DCB0160 clinical safety assessment

Any risk assessment should also take into account the risks of not implementing and using traditional paper based methods to calculate scores. Whilst we should aim to mitigate all risks, there is a degree of pragmatism required and weighing up of the relative risks involved. This should be clinically led as it requires good clinical judgement. In modern development practices it is important to decide what is the ‘minimum viable product’ and have the ability to iterate and improve on that based on user feedback and evaluation.

In practice these risks can be addressed through proper clinical governance processes that cover both the design and the implementation of the product. In view of the known risks with standalone calculators, however, we should give strong consideration to incorporating all clinical calculators within the electronic patient record (EPR).

Design Principles

In the course of our work we have developed the following design principles to help support the integration and implementation of clinical calculators within the healthcare EPR system.

These principles are intended to support the local clinical governance and assurance processes by providing a checklist of requirements to consider during the development, testing and clinical sign off of a clinical calculator, prior to its deployment and use in clinical practice.

Data and Audit

  • All clinical parameters used in the calculation should be captured in patient context as part of the structured electronic patient record  (e.g. height, weight, temperature, age, sex, ethnicity, performance status, lab results etc.)
  • The calculated value/score should also be captured and time stamped as part of the structured clinical record within the EPR
  • It should be possible to retrospectively derive, reproduce, validate and audit the calculated score from the source data captured and recorded at the time within the EPR
  • Clinical data should be recorded using standard (SI) units of measurement (UoM) but consideration should be given to performing on the fly conversion from non-standard units at data entry where these are in common use (eg height ft/metres and weight stone/kg)
  • It should be possible to display and individually chart in patient context both the the source parameters and the calculated score over time
  • If a clinical parameter or calculation is recorded in error it must possible to redact and correct the entry and calculated score in the patient record
  • There should be a full audit trail of who performed the calculation and when, and any subsequent amendment or revocation made
  • There should be a version history of any changes/updates made to the calculation algorithm or methodology
  • For maintainability, the calculator should ideally be implemented once but be able to be used and accessed from more than one area of the EPR if required.
  • It should be possible to unit test the calculation to ensure it remains consistent with changes to the system or browser/device type and configuration

Validation and Safety

  • Clinical staff should be sufficiently trained and help instructions provided to understand the purpose of the calculation and how it should be used.
  • Clinical data (parameters) should be pre-populated from the system and IoT connected devices where appropriate to reduce risk of manual data transcribing errors
  • It should be possible to configure any data pre-populated and set limits on persistence to avoid performing calculations and basing decision support on outdated clinical information
  • There should be inbuilt validation of manually entered clinical parameters to mitigate the risk of mistyping and errors
  • Any changes made to clinical parameters at entry should automatically recalculate the score and provide a clear indication of the change.
  • It should be possible to RAG rate calculated values/scores or flag those outside the normal range
  • Where possible the system should show which contributing parameters were out of range in respect of an abnormal calculation
  • Clinical decision support should be provided based on the calculated value/score
  • All calculations should undergo DCB 0129 and 0160 clinical safety assessment in design and implementation and assessed against the design principles.

Clinical Workflow

  • Calculations and decision support should be designed to support specific use cases and clinical workflows.
  • The calculator should only be accessible to those required to perform the calculation and at the appropriate step in their workflow.
  • A clinical calculation may form part of a wider clinical assessment process and should be displayed in the context of that workflow.
  • It should be possible to configure the clinical calculation to drive onward system workflows (e.g. create a task, send a message, add to a worklist, make a referral, recommend an action, trigger an assessment, inform a care plan, send a notification, add an alert, set a repeat interval for reassessment etc.)
  • It should be possible for the appropriate clinical staff to acknowledge the calculated value/score/result to indicate that the information has been received and actioned
  • It should be possible to record a calculated value within a clinical form as a ‘read only’ field either through calculating directly within the form or pulling /pre-populating the calculated value into the form
  • It should be possible to script automated clinical rules based on the clinical decision support relevant to the calculated value score
  • It should be possible for authorised users to override the clinical guidance where appropriate and record a reason for varying from any established protocol.
  • It should be possible to include the calculated value on a configurable clinical work-list with the relevant help and context instructions.
  • Complex clinical workflow should be process mapped and documented to ensure all parties are aware of their respective roles and responsibilities.
  • It should be possible to display and feed the calculated value into business reports and dashboards to ensure compliance with the clinical workflow and support assessment of clinical outcomes.

Software development and quality management

In addition to the design principles above, clinical calculators should follow the local agreed quality management system (QMS) processes and guidelines for software development and testing. These should factor in the NHS standard user interface (UI) guidance, relevant common user interface (CUI) safety measures and local/national data standards to meet regulatory requirements and accessibility guidelines.

Every effort should also be made to optimise the user experience (UX) and minimise the number of clicks involved. User interaction should be tested with the clinical users on each of the clinical devices to be used in practice. Post implementation reviews and feedback mechanisms should be in place to identify any comments and issues, and use these to iteratively improve the UI design and UX.

Sharing code and design is a great way to help validate and improve your implementation and also benefit others. The open source initiatives support sharing of code and you can also blog and include screenshots to show some of the design principles that prove popular with the users.

Standards based open application programme interfaces (APIs) provide a means to share a calculation within and between clinical applications. These can also help separate the application logic form the visual display.

All calculations should undergo a formal clinical safety assessment and clinical safety sign off according to local policies and legislative requirements.

Examples

The following examples show sample implementations of clinical calculators within our local EPR. It is not possible to show all aspects of the guidance in the context of these simple screen shots but it is hoped these may serve to illustrate how calculations may be displayed in clinical practice.

Web form BMI calculator

This body mass index calculator is incorporated in several areas of the EPR where clinical practice determines we record height and weight, and the calculated BMI may contribute the the risk assessment and decision support. Example implementations are in our nutrition and dietetics assessment form, moving and handling nursing assessment form, and our standard outpatient clinical assessment form.

The data are captured as part of the form and underlying clinical data record (model/repository). This enables the (eg weight) trend to be viewed in a popup chart when entering the data, and wherever the data and BMI are displayed in the EPR.

Users have fed back that they like the ability to see both metric and imperial values at data entry, and that they use either the slider or keypad to enter the data. Dieticians value the ability to chart the data and see the trend and progress over time.

NEWS2 calculator

The national early warning score takes input from a range of clinical observations, each of which generates its own score. The overall score determines the care plan and escalation.

A number of dedicated NEWS2 apps are available that can calculate the score and provide decision support. The risks arise in using standalone applications that do not meet the standards for charting the observations, or provide a way to interface with an EPR. By integrating or developing a calculator with the EPR it is possible to record and chart the data as per the national guidelines. This enables healthcare staff to see the trend and relations between the observations, and support earlier detection and response to the deteriorating patient. Escalations can be automated and the observations can be linked to support other clinical pathways (sepsis, transfusion etc.)

NEWS2 form clinical calculator

NEWS2 chart and guidance

End note

I have created this blog to help share our collective experience gained in developing and assuring clinical applications. Please feel free to copy and use all or some of these principles in your local governance processes. I’d be grateful for any comments to help improve these. Please feedback either here or on Twitter and I will look to update this blog with any comments received.

©️Dr Rhidian Bramley 2019

Author: Rhidian Bramley

Chief Clinical Information Officer (CCIO) and Associate Medical Director at the Christie NHS trust. Clinical directorate consultant at NHS Wales Informatics Service (NWIS).

2 thoughts on “What makes a good clinical calculator?”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s