A quantum algorithm is not meaningfully better because it produces an attractive result on one simulator run. This guide presents a reproducible workflow for comparing quantum and classical methods using the same problem instances, quality targets, runtime accounting, resource limits, and uncertainty analysis. Use it to decide whether a quantum approach deserves further engineering, hardware testing, or technical review.
Overview
Quantum algorithm benchmarking is a comparison problem, not simply a circuit-timing exercise. A fair evaluation asks whether a quantum workflow provides a useful trade-off against a credible classical baseline under clearly stated conditions. Those conditions include input size, train-test splits, optimizer settings, simulator or hardware selection, shot budgets, compilation overhead, and the definition of success.
The comparison should cover more than answer quality. Record at least:
- Quality: accuracy, objective value, approximation ratio, energy estimate, feasibility rate, or another task-specific metric.
- Runtime: classical preprocessing, circuit construction, compilation, queue or service time when relevant, execution, measurement, and classical post-processing.
- Cost: provider charges or internal compute accounting, reported with the pricing assumptions and currency or unit used.
- Resources: logical qubits, physical qubits where available, circuit depth, two-qubit gate count, shots, parameter evaluations, and memory requirements.
- Reliability: failure rate, invalid outputs, constraint violations, convergence failures, and variability across repeated runs.
A quantum result should be compared with a classical method that is appropriate for the same problem, not with an intentionally weak implementation. For example, a variational quantum optimizer might be compared with a conventional local optimizer, a greedy heuristic, a simulated annealing implementation, or an exact solver on small instances. The best baseline depends on the task and the size range being tested.
For implementation context, review the resource-estimation workflow in Quantum Circuit Resource Estimation, then use the transpilation guidance in Quantum Circuit Transpilation Optimization before collecting execution metrics.
How to estimate
1. Define the decision before running experiments
State what the benchmark is intended to decide. Examples include whether a quantum classifier reaches a target accuracy, whether QAOA improves an objective under a fixed execution budget, or whether a quantum kernel is competitive for a constrained dataset size. A benchmark without a decision rule can accumulate measurements without producing an actionable conclusion.
Write a primary metric and acceptable trade-offs in advance. A useful rule might be: “The quantum workflow is worth further testing if its quality is statistically compatible with the best baseline while staying within the allowed runtime and execution budget.” This does not claim a speedup; it defines a transparent gate for the next stage.
2. Create matched problem instances
Use the same instances, labels, constraints, and stopping criteria for every method. For supervised learning, fix the train-validation-test split or use identical cross-validation folds. For optimization, generate or select instances across several sizes and difficulty levels rather than reporting only the easiest case. Keep instance generation separate from the algorithm code so that a later run can reproduce the exact inputs.
3. Establish classical baselines
Include at least one simple baseline and one credible competitive baseline. The simple method reveals whether the quantum method is adding value beyond a minimal approach. The stronger method tests whether it is competitive with the tools a practitioner would reasonably choose.
Give baselines comparable tuning effort. Record software versions, hardware, thread counts, random seeds, stopping tolerances, initialization, and parameter-search budgets. If the quantum workflow receives extensive hyperparameter tuning but the classical baseline uses defaults, the comparison is not balanced.
4. Separate ideal, noisy, and hardware conditions
Do not combine an ideal statevector result with a hardware result in one undifferentiated ranking. Treat them as separate experimental conditions:
- Ideal simulation: useful for checking algorithmic behavior without sampling or device noise.
- Shot-based simulation: exposes measurement uncertainty while keeping the execution environment controlled.
- Noisy simulation: tests a specified noise model, but should not be presented as a substitute for every hardware behavior.
- Hardware execution: includes device-specific compilation, calibration conditions, connectivity, queue behavior, and measurement effects.
When moving between SDKs or quantum cloud platforms, preserve the experiment definition while adapting backend configuration. Capture the selected backend, transpilation or compilation settings, shots, observable definitions, and job identifiers where available. Backend limits and timing fields can change, so treat provider metadata as part of the record rather than assuming every platform reports the same quantities.
5. Calculate end-to-end runtime and cost
Report both component times and total wall-clock time. A practical decomposition is:
T_total = T_prepare + T_compile + T_submit + T_queue + T_execute + T_postprocess
Not every environment exposes every term. If queue time is unavailable or excluded, say so. Also report a second metric when useful: active execution time excluding queue delays. This prevents a provider scheduling artifact from being mistaken for an algorithmic property, while still showing the operational time a user experienced.
Cost estimation follows the same principle. Use the actual billing fields available to the platform, or label the result as an internal estimate. Include the number of jobs, shots per job, parameter evaluations, retries, and any classical compute used alongside the quantum execution. Avoid comparing a measured quantum charge with an unmeasured classical cost.
Inputs and assumptions
Before running a benchmark, create a configuration file or run manifest containing the following inputs:
| Category | Record | Why it matters |
|---|---|---|
| Problem | Instance IDs, sizes, encoding, constraints, and objective definition | Prevents hidden changes to the workload |
| Data | Split method, preprocessing, normalization, and labels | Controls leakage and makes quality metrics comparable |
| Algorithm | SDK version, circuit family, ansatz, optimizer, initialization, and stopping rule | Defines the implementation being evaluated |
| Execution | Simulator or hardware, compilation settings, shots, retries, and seeds | Determines noise, sampling, and runtime conditions |
| Resources | Qubits, depth, gate counts, parameter evaluations, memory, and CPU threads | Shows what each method consumed |
| Outcome | Quality, confidence interval, failure count, runtime, and cost | Supports an evidence-based decision |
Use repeated trials when randomness affects initialization, sampling, instance selection, or hardware execution. Report the mean and spread, but do not stop at an average. Include the number of trials and a confidence interval or another clearly explained uncertainty measure. For a binary success metric, record the numerator and denominator; a success rate without the number of attempts is difficult to interpret.
For measurement-heavy algorithms, distinguish between circuit executions and classical iterations. One optimizer iteration may require many circuits, observables, parameter shifts, or measurement groups. Reporting only the number of optimizer iterations can substantially understate the workload.
Worked examples
Example 1: A variational optimizer for Max-Cut
Suppose a team compares QAOA with a classical heuristic on graph instances containing several sizes. The primary metric is the best feasible cut value found within a fixed wall-clock budget. The secondary metrics are approximation ratio, number of circuit evaluations, depth after compilation, two-qubit gate count, shots, total runtime, and failure rate.
The team should use identical graph instances and define whether the budget includes parameter optimization and compilation. A useful comparison table might look like this:
| Method | Quality | Runtime | Cost | Depth | Qubits | Failure rate |
|---|---|---|---|---|---|---|
| Classical heuristic | Objective value and spread | End-to-end CPU time | Compute estimate | Not applicable | Not applicable | Invalid or infeasible outputs |
| QAOA, simulator | Objective value and spread | Compilation plus execution | Local compute estimate | Compiled depth | Logical qubits | Constraint violations |
| QAOA, hardware | Objective value and spread | Active and wall-clock time | Reported provider or internal cost | Compiled depth | Logical and physical counts if available | Invalid or infeasible outputs |
If the hardware result is lower quality but uses a different budget, the conclusion should not be “hardware is worse.” The correct conclusion may be that the tested circuit and shot budget did not reach the target under those conditions. The next experiment could test transpilation, parameter initialization, error mitigation, or a smaller instance range. See QAOA in Python for Max-Cut for the algorithm-specific workflow.
Example 2: A quantum classifier
For a hybrid classifier, compare the quantum model with a classical model that receives the same features and training split. Fix the number of training attempts or define a fair tuning budget. Report test performance only after model selection is complete. Record training runtime, inference runtime, circuit evaluations, qubits, depth, shots, and the number of failed or non-converged runs.
A quantum model that matches accuracy but requires substantially more evaluations may still be useful for a research prototype, but it has not demonstrated a practical advantage. Conversely, a model with lower accuracy might be valuable if it meets a domain-specific constraint; that constraint must be stated before comparing results. Related workflows include Quantum Neural Networks in Python and Quantum Kernel Methods in Python.
Benchmark record template
Store one machine-readable record per run with fields such as:
run_id, date, code_commit, environment, sdk_version, problem_seed, instance_ids, split, method, backend, execution_mode, shots, random_seed, optimizer, stopping_rule, qubits, depth, two_qubit_gates, circuit_evaluations, quality, uncertainty, failures, compile_time, execution_time, queue_time, total_time, cost, notes
Keep raw counts, probabilities, quasi-distributions, or expectation values where applicable rather than saving only a final score. That makes it possible to audit post-processing and rerun analysis when the reporting requirement changes.
When to recalculate
Recalculate the benchmark whenever an input changes the meaning of the comparison. The most important triggers are:
- A dataset, graph generator, encoding, train-test split, or instance-size range changes.
- The classical baseline, optimizer, stopping rule, or tuning budget changes.
- The SDK, compiler, transpiler, simulator, backend, device, or error-mitigation method changes.
- The shot budget, number of repetitions, circuit depth, qubit mapping, or parameter-evaluation strategy changes.
- Provider pricing, billing fields, backend limits, or available timing metadata changes.
- A new noise model, calibration condition, hardware generation, or execution location is introduced.
As a maintenance practice, review the benchmark record and execution scripts after major framework or provider changes, and perform a full comparison at least every six months if the results guide an active project. Do not overwrite earlier results. Version the benchmark so that a new outcome can be attributed to a changed input.
Before publishing or approving a result, use this checklist:
- Is the problem definition identical across methods?
- Are the classical baselines credible, tuned fairly, and fully documented?
- Are ideal simulation, noisy simulation, and hardware results separated?
- Are compilation, shots, parameter evaluations, retries, and post-processing included in the workload?
- Are runtime and cost definitions explicit?
- Are uncertainty, repeated trials, failures, and invalid outputs reported?
- Are qubit count, depth, two-qubit gates, and other resource measures included?
- Can another developer reproduce the run from the code, configuration, environment, and recorded seeds?
A fair benchmark rarely produces a single universal winner. Its value is more practical: it shows which method meets a defined quality target under a defined resource budget, and it makes the next engineering decision defensible. That is the standard to apply when testing a new quantum SDK, moving from simulation to hardware, or deciding whether a hybrid quantum-classical application merits further investment.