The Delphi method: conducting a multi-stage survey

The Delphi method is a multi-stage process used to assess future developments. Experts in a particular field are asked over several rounds, usually by online questionnaire, to give their assessment of a topic or complex problem. The method is based on the assumption that several specialists in a field can produce more robust assessments of future developments than a single individual. The systematic survey process brings expert knowledge together to support more realistic assessments of future events and more informed forecasts.

Delphi method: definition and objectives

A Delphi survey is a research method in which several experts are asked about complex problems in order to develop well-founded forecasts about the future. It is a multi-stage process, meaning that the experts are questioned more than once about the same topic or issue. The Delphi method brings the knowledge of these specialists together so that more robust forecasts can be made and relevant trends identified. Systematic round-based Delphi studies are usually conducted using online questionnaires.

The method involves repeatedly questioning a predefined panel of experts on a particular topic. It is based on the assumption that several experts can produce more robust assessments of complex future developments than one individual. The experts analyse the problem independently of one another and are surveyed in several stages. Feedback is provided between rounds, so each expert receives information about the preceding round. The exact form of this feedback depends on the Delphi variant being used.

In addition to the standard Delphi survey, other variants include the wideband Delphi and real-time Delphi methods. Their differences and similarities are explained below.

Typical objectives of Delphi surveys include:

• consolidating expert knowledge
• producing more realistic estimates
• developing solutions to complex problems
• creating forecasts about future developments
• finding consensus or achieving convergence between individual opinions
• generating ideas for solving a problem

The Delphi method is used to develop forecasts about the future. Its name is derived from the Oracle of Delphi, an ancient Greek site associated with prophecy and predictions about the future.

Create a Delphi questionnaire

Standard Delphi survey: process and procedure

Before the Delphi study begins, the project manager or project team defines the panel of experts and provides detailed information about the content and objective of the survey. The choice of experts can have a considerable influence on the results. In the first round, all experts receive the same online questionnaire containing statements or questions that they assess or answer anonymously. A key feature of the standard Delphi method is that the experts do not communicate with one another and usually do not know who else belongs to the panel. This is intended to reduce group dynamics and mutual influence.

After the first round, the project manager analyses and consolidates the results and prepares a summary for the second round. This may involve calculating means and ranges or adding descriptive statistical analysis. The experts then receive the questionnaire again, supplemented with comments or statistical measures. They can reconsider their estimates and assessments in light of the anonymised group results and adjust or refine them where necessary. This creates a form of indirect discussion between the experts with a feedback-based revision process.

The process is repeated over several rounds. Each subsequent questionnaire is supplemented with group results from the previous round. The number of rounds depends on the topic and research objective. It may be fixed in advance at two or more rounds, or it may depend on the degree of consensus reached during the study.

The aim of a round-based Delphi study is generally to achieve a degree of convergence, meaning that expert assessments move closer together and reach a high level of agreement. The process concludes with a final report documenting the results and presenting the group view. Means calculated from the experts’ assessments after the final round can then support decision-making.

Wideband Delphi method: direct exchange between experts

Unlike the standard Delphi method, the wideband Delphi method involves direct communication between the experts. This usually takes place before the first survey round in the form of a group discussion, although views and approaches may also be exchanged in writing. After discussing the topic together, the experts usually receive an online questionnaire in which they provide their own assessments of the subject.

Once the questionnaire has been completed, the project manager analyses the responses and prepares the results for the experts. The interim findings are presented to the panel and discussed again in a meeting. The discussion often focuses on extreme values, large differences and discrepancies between individual estimates. Each expert then completes the questionnaire or form independently again and can revise their assessment in light of the group discussion. This cycle is repeated until a degree of consensus develops and the estimates fall within an agreed tolerance range.

The final result is based on the last survey round. Means are calculated from the estimates and presented as the final estimate.

Agile software development uses a variant of wideband Delphi known as “Planning Poker”. Planning Poker, also called Scrum Poker, is a game-based technique for estimating effort.

Real-time Delphi survey: feedback in real time

Unlike the conventional round-based Delphi method, a real-time Delphi survey does not use repeated, separate rounds. Instead, its defining features are the calculation and provision of results in real time. Real-time Delphi studies are usually conducted through an online portal. Participating experts receive access for a limited period. During this period, they can log in at any time, review and edit their previous estimates, and observe the current estimates of the other experts in real time.

The portal also displays means and measures of dispersion for the expert group, allowing each participant to see whether and to what extent their own estimate differs from the group estimate. The main difference from the Delphi variants described above is therefore the real-time calculation and provision of results.

LamaPoll can be used to conduct standard Delphi surveys securely and in compliance with data protection requirements. Free questionnaire templates for these and other market research projects, as well as employee and customer surveys, are available here: questionnaire templates and examples