User needs from consumer price inflation item indices and price quote microdata

Between 6 August and 23 October 2024, we engaged with users of our consumer price inflation item indices data and price quotes microdataset to better understand how they are used. These datasets are published on the same day as our inflation figures and are supplementary to our consumer price inflation, UK bulletin and reference tables.

With the planned introduction of groceries scanner data, we will no longer be able to publish the item indices data and price quote microdataset in their current forms. Our data-sharing agreements with retailers providing scanner data will lead to the discontinuation of price quote microdata for Divisions 1 and 2 (food and non-alcoholic beverages, and alcohol and tobacco). While the coverage of the analytical microdata will be reduced, the introduction of large scanner datasets will greatly improve the quality of our accredited official inflation statistics.

There may be scope to publish alternative data products which meet the current needs of existing microdata users. To better understand those needs, we ran a short user engagement questionnaire to capture more information about how these data are currently being used.

We received 15 external responses from a variety of users. Below is a table that shows a breakdown of the type of respondents who replied to the questionnaire.

Table 1: Respondent count by sector

All respondents were in analytical roles.

Respondents were also asked to share how they make use of both datasets, as outlined in the below table.

Table 2: Use count of each dataset

For both datasets, the frequency of the data use was split evenly between irregular or occasional usage, and regular monthly or quarterly usage.

Many of the price quote microdataset users indicated that they try and understand how the price quote data have influenced higher-level index movements. This is often in the context of improving inflation forecasting models. Users indicated they wish to understand the extent to which price movements have been influenced by one-off events, compared with broader movements across the price sample. They also indicated interest in the prevalence of discounting across the price sample, as well as making use of other marker codes such as temporary missingness or sample replacements.

A further area of common interest is in understanding regional variation in prices. In some cases, users are seeking to construct higher-level regional aggregates. In other cases, exploratory analysis of the regional data is undertaken.

Some users highlighted that having a long-run, continuous dataset of comparable price quote data is particularly valuable to the wider research community. One user suggested copying the approach used to release census extracts, through a “sample of anonymised records”, drawing a sample in a way which mimics the CPI price collection design.

The item indices dataset is commonly used for the construction of alternative aggregates of inflation data, whether as a cross-check for elementary aggregation from the price quote data, or to supplement special aggregates where the price quote data for a particular item are unavailable.

As detailed in our blog, we will not be introducing scanner data into our consumer price inflation statistics until March 2026. This means that the price quote microdataset will continue to be available in its current form up until this date, with some structural changes resulting from the processing improvements planned for March 2025 (more information will be provided in the Consumer price inflation, UK bulletin published on 19 February 2025).

However, from March 2026, our data sharing agreements with scanner data retailers mean we will have to discontinue the price quote microdataset for Divisions 1 and 2 (food and non-alcoholic beverages, and alcohol and tobacco). Price quote data for the remaining divisions will continue to be available as per the current publication.

The size and structure of the scanner data is intrinsically different to the current data, making them unsuitable for some existing uses. For example, whilst a price quote in the locally collected dataset can be identified as “on-sale”, scanner data unit prices are averaged across three weeks, during which time products may have been both on and off sale at different points of the month.

Nonetheless, the questionnaire responses suggest that there are aggregate statistics that could be produced to meet some of the existing needs of users of the price quote microdataset. Before March 2026, we aim to share example aggregate statistics with users, and would welcome further user feedback about them.

From March 2026, groceries data (Divisions 1 and 2) will continue to be available as part of the item indices dataset. However, the move from sample items to broader, exhaustive “consumption segments” (see the article, Introducing alternative data into consumer price statistics - aggregation and weights) means that, from March 2025, disaggregate groceries indices will have less specific definitions. Changes to the classification structure (as described in the article, Classification of new data in UK consumer price statistics) mean that specific item indices for Divisions 1 and 2 will no longer be calculated as part of the aggregation process. For all other divisions, the consumption segment level will be equivalent to the existing item level, so these indices will continue to be available in their current form.

We would like to thank all of those who took part in the user engagement questionnaire and provided us with invaluable feedback that will guide future changes to the dataset. We intend to provide further opportunities to engage and inform users with updates before March 2026.

If you have any further questions or would like to request additional information, please email cpi@ons.gov.uk.