Our assessment process

We use a set of Digital Assessment Questions to make sure only safe and secure apps and digital tools are published on the NHS Apps Library

Our Digital Assessment Questions (DAQ) have been designed by a group of experts across a range of technical and policy backgrounds and incorporate national standards, regulations and industry best practice.

They are used to see how a product performs in areas such as clinical safety, data protection, security and usability.

How we assess

Step 1: Eligibility

Developers must answer our eligibility questions before putting their product forward for full assessment. It must meet the following criteria to be considered for publication on the Apps Library:

  • the product is already available to the public in the App Stores or by other means
  • the developer can be contacted directly by users of their product
  • the product doesn’t use any form of NHS branding unless permitted to do so
  • if the product connects to any NHS services, the developer must have evidence that a proper interoperability review has been undertaken
  • the product must be registered and have the relevant certification if it is a medical device
  • the product must be registered with the General Pharmaceutical Council if it provides a pharmacy service
  • if the product requires registered healthcare professionals to operate it, the developer must provide the healthcare professionals' registration status and names
  • developers are registered with the Care Quality Commission if required
  • developers can provide a guest login for use by those assessing their product
  • the developer's organisation is registered as one of the following:

    eligible types of organisation
    • Public Limited Company (Plc)
    • Private Company Limited By Shares (Ltd)
    • Company Limited By Guarantee
    • Unlimited Company (Unltd)
    • Limited Liability Partnership (Llp)
    • Community Interest Company
    • Industrial and Provident Society (Ips)
    • Royal Charter
    • Public Body
    • Charitable Organisation

Step 2: Registration

After satisfying our eligibility questions, developers will be asked to provide information about their organisation and the product they are submitting for assessment.

We will need to know the developers' organisation’s registered address, contact details for the individual that will be managing their assessment and details of the developer's Care Quality Commission registration if the developer required to have one.

We will then ask for information about the product, such as what health theme it addresses, who the intended users are and how much it costs. This section also covers the developers business model, device registration and use of branding.

Step 3: Technical assessment

When answering these questions, developers may need to show evidence that their product passes our tests in these areas, and how anyone using them could see benefits to their health and wellbeing. Any major updates made to the product will require it to be reassessed to make sure it still meets the necessary standards following the changes.

Our technical assessment examines how a developer's product performs in seven key areas:

Evidence of Outcomes

These questions make sure all products are doing what they are supposed to do, and we will ask developers to show us how their product improves health and wellbeing. For example, if an app is designed to help patients with their mental health, developers must give examples of how it could help - or already has helped - people.

We also ask if there is any evidence of the clinical, economic or behavioural benefits of using a developer's product, such as how it has helped improve symptom control, clinical outcomes or patient reported outcomes.

Clinical Safety

Our clinical safety questions make sure that developers have taken all appropriate action to keep safe any patients using their product. For example, with an app that reminds patients to take their medication, developers must give evidence that shows that any risk of these reminders being incorrect has been completely removed or made as low as possible.

Developers of any product that could put a user at risk must meet the safety standards required by the Health and Social Care Act 2012. This would mean producing Hazard Logs and Safety Case Reports, which would be reviewed by experts at NHS Digital.

Data Protection

Our data protection questions are designed to make sure that any personal information collected or shared by an app or digital tool is handled in a safe, fair and lawful way. This would include health information recorded by the user, such as diabetes readings or health information available via the product if it uses the internet to connect to an individual’s health record

The UK Data Protection Act 2018 gives people rights and control over their information and places greater responsibilities on organisations to use people’s information appropriately and securely.

The developer must give details of how where the data collected is stored and tell users what rights they have to control how their information is used.

Security

This section is used to assess the security assurance of an app or digital tool. The questions make sure a user’s data has been correctly categorised taking account of data protection regulations and clinical impact.

They also ask for confirmation that a security assessment against applicable Open Web Application Security Project standards has been carried out.

Usability & Accessibility

Our usability and accessibility questions are designed to make sure a person can understand and use an app or digital tool effectively. Text must be clear and easy to read and action buttons big enough, easy to press and marked with commands that make sense to users. Functions the product carries out must do what the user expects and not perform any extra actions that are not made clear.

All products are assessed against Web Content Accessibility Guidelines 2.1, the agreed international standards for digital accessibility that all web content must satisfy. This is to ensure they provide access to as many people as possible, including older users, younger users and those with disabilities. This might involve being able to increase text size where needed and work with voice software to help visually impaired people.

The usability of an app or digital tool must satisfy the International Organization for Standardization’s requirements and recommendations for human-centred design principles and activities throughout its life cycle.

Interoperability

Our interoperability questions test how well a product exchanges data with other systems. For example, how it connects with a patient’s medical record or collects readings from another device such as a smart watch or blood pressure monitor. This process helps developers use data within their products to build new functions, benefiting users.

To do this, developers use Application Programming Interfaces (APIs) – a service that allows third parties to view a product’s data in a more digestible format. Not all apps exchange data, but those that do must adhere to NHS England’s Open API policy. These rules make the sharing process simple while also keeping it safe and secure.

Technical Stability

The technical stability questions are used to understand how an app or digital tool has been tested and how testing will continue over time. Developers must show how patients can report any problems with a product and how the developer will work to correct them.

These questions also cover what happens to any patient information a product has collected if the patient stops using it or it is shut down by the developer.

Technical assessment diagram

The number of questions developers will be required to answer depends on the complexity of their product (see diagram, above). For more basic apps, they will need to answer approximately 60 questions. More complex apps may require up to 180 additional questions to be answered.

View the DAQ

You can see an example of the questions we ask so that you know what to expect from the assessment. This document is for information purposes only and cannot be completed and submitted for assessment.


How the DAQ are updated

As technology constantly moves forward and improves, so do the standards and regulations. It is therefore very important that the DAQ are frequently reviewed and updated to make sure that the products on the library maintain the expected high standards. We will be working closely with our team of experts and partner organisations to review and update the DAQ periodically.

For example, in December 2018, the National Institute for Health and Care Excellence (NICE) published new standards for evidence of clinical and cost effectiveness in digital health technologies, including health apps. NHS Digital is working to include these standards in the next version of the DAQ to help developers understand what level of evidence is required.

To help developers keep up with the very latest developments, we have published a roadmap of planned changes to the DAQ based on relevant national standards, policy and regulations for apps and digital tools in healthcare.

The latest version of the roadmap can be found here.