Tutorial 5: Benchmark solvers

Benchmark

This tutorial follows closely the tutorial OptimizationProblems.jl. We show how to use SolverBenchmark.jl to benchmark solvers that take a NLPStopping as input.

using LinearAlgebra, NLPModels, Stopping
using JSOSolvers, StoppingInterface
using DataFrames, Printf, SolverBenchmark
using ADNLPModels, OptimizationProblems

We select the problems from OptimizationProblems.jl that are unconstrained and scalable.

df = OptimizationProblems.meta
names_pb_vars = df[(df.variable_nvar .== true) .& (df.ncon .== 0), :name]

ad_problems = (
  OptimizationProblems.ADNLPProblems.eval(Symbol(problem))(n = 31) for problem ∈ names_pb_vars
)

Then, we prepare the solvers we will benchmark. Here, we use JSOSolvers.jl that are made Stopping-compatible using StoppingInterface.jl.

solvers = Dict(
  :lbfgs => model -> stopping_to_stats(StoppingInterface.lbfgs(NLPStopping(model); mem=5, atol=1e-5, rtol=0.0, max_time = 5.)),
  :trunk => model -> stopping_to_stats(StoppingInterface.trunk(NLPStopping(model); atol=1e-5, rtol=0.0, max_time = 5.)),
)

The main function used from SolverBenchmark.jl.

stats = bmark_solvers(
  solvers, ad_problems
)

The output of bmark_solvers can then be analyzed for the results as a table

cols = [:id, :name, :nvar, :objective, :dual_feas, :neval_obj, :neval_grad, :neval_hess, :iter, :elapsed_time, :status]
header = Dict(
  :nvar => "n",
  :objective => "f(x)",
  :dual_feas => "‖∇f(x)‖",
  :neval_obj => "# f",
  :neval_grad => "# ∇f",
  :neval_hess => "# ∇²f",
  :elapsed_time => "t",
)

for solver ∈ keys(solvers)
  pretty_stats(stats[solver][!, cols], hdr_override=header)
end

first_order(df) = df.status .== :first_order
unbounded(df) = df.status .== :unbounded
solved(df) = first_order(df) .| unbounded(df)
costnames = ["time", "obj + grad + hess"]
costs = [
  df -> .!solved(df) .* Inf .+ df.elapsed_time,
  df -> .!solved(df) .* Inf .+ df.neval_obj .+ df.neval_grad .+ df.neval_hess,
]

or as a performance profile

using Plots
gr()

profile_solvers(stats, costs, costnames)

This page was generated using Literate.jl.