Benchmarking Results

Results are split by distance measure and dataset. In the bottom, you can find an overview of an algorithm's performance on all datasets. Each dataset is annoted by (k = ...), the number of nearest neighbors an algorithm was supposed to return. The plot shown depicts Recall (the fraction of true nearest neighbors found, on average over all queries) against Queries per second. Clicking on a plot reveils detailled interactive plots, including approximate recall, index size, and build time.

Machine Details

All experiments were run in Docker containers on 2x 14-core Intel Xeon E5-2690v4 (2.60GHz) with 512GB of RAM using Ubuntu 16.10 with kernel 4.4.0.

Raw Data & Configuration

Please find the raw experimental data here. The query sets and ground truth is available with each dataset. The algorithms used the following parameter choices in the experiments.

Benchmarks for Single Queries

Results by Dataset

Distance: Angular

glove-100-angular-diverse-expansion (k = 10)

---

glove-100-angular-diverse-lid (k = 10)

---

glove-100-angular-diverse-lrc (k = 10)

---

glove-100-angular-easy-expansion (k = 10)

---

glove-100-angular-easy-lid (k = 10)

---

glove-100-angular-easy-lrc (k = 10)

---

glove-100-angular-hard-expansion (k = 10)

---

glove-100-angular-hard-lid (k = 10)

---

glove-100-angular-hard-lrc (k = 10)

---

glove-100-angular-middle-expansion (k = 10)

---

glove-100-angular-middle-lid (k = 10)

---

glove-100-angular-middle-lrc (k = 10)

---

glove-2m-300-angular-diverse-expansion (k = 10)

---

glove-2m-300-angular-diverse-lid (k = 10)

---

glove-2m-300-angular-diverse-lrc (k = 10)

---

glove-2m-300-angular-easy-expansion (k = 10)

---

glove-2m-300-angular-easy-lid (k = 10)

---

glove-2m-300-angular-easy-lrc (k = 10)

---

glove-2m-300-angular-hard-expansion (k = 10)

---

glove-2m-300-angular-hard-lid (k = 10)

---

glove-2m-300-angular-hard-lrc (k = 10)

---

glove-2m-300-angular-middle-expansion (k = 10)

---

glove-2m-300-angular-middle-lid (k = 10)

---

glove-2m-300-angular-middle-lrc (k = 10)

---

gnews-300-angular-diverse-expansion (k = 10)

---

gnews-300-angular-diverse-lid (k = 10)

---

gnews-300-angular-diverse-lrc (k = 10)

---

gnews-300-angular-easy-expansion (k = 10)

---

gnews-300-angular-easy-lid (k = 10)

---

gnews-300-angular-easy-lrc (k = 10)

---

gnews-300-angular-hard-expansion (k = 10)

---

gnews-300-angular-hard-lid (k = 10)

---

gnews-300-angular-hard-lrc (k = 10)

---

gnews-300-angular-middle-expansion (k = 10)

---

gnews-300-angular-middle-lid (k = 10)

---

gnews-300-angular-middle-lrc (k = 10)

---

Distance: Euclidean

fashion-mnist-784-euclidean-diverse-expansion (k = 10)

---

fashion-mnist-784-euclidean-diverse-lid (k = 10)

---

fashion-mnist-784-euclidean-diverse-lrc (k = 10)

---

fashion-mnist-784-euclidean-easy-expansion (k = 10)

---

fashion-mnist-784-euclidean-easy-lid (k = 10)

---

fashion-mnist-784-euclidean-easy-lrc (k = 10)

---

fashion-mnist-784-euclidean-hard-expansion (k = 10)

---

fashion-mnist-784-euclidean-hard-lid (k = 10)

---

fashion-mnist-784-euclidean-hard-lrc (k = 10)

---

fashion-mnist-784-euclidean-middle-expansion (k = 10)

---

fashion-mnist-784-euclidean-middle-lid (k = 10)

---

fashion-mnist-784-euclidean-middle-lrc (k = 10)

---

mnist-784-euclidean-diverse-expansion (k = 10)

---

mnist-784-euclidean-diverse-lid (k = 10)

---

mnist-784-euclidean-diverse-lrc (k = 10)

---

mnist-784-euclidean-easy-expansion (k = 10)

---

mnist-784-euclidean-easy-lid (k = 10)

---

mnist-784-euclidean-easy-lrc (k = 10)

---

mnist-784-euclidean-hard-expansion (k = 10)

---

mnist-784-euclidean-hard-lid (k = 10)

---

mnist-784-euclidean-hard-lrc (k = 10)

---

mnist-784-euclidean-middle-expansion (k = 10)

---

mnist-784-euclidean-middle-lid (k = 10)

---

mnist-784-euclidean-middle-lrc (k = 10)

---

random-xs-20-euclidean-diverse (k = 10)

---

sift-128-euclidean-diverse-expansion (k = 10)

---

sift-128-euclidean-diverse-lid (k = 10)

---

sift-128-euclidean-diverse-lrc (k = 10)

---

sift-128-euclidean-easy-expansion (k = 10)

---

sift-128-euclidean-easy-lid (k = 10)

---

sift-128-euclidean-easy-lrc (k = 10)

---

sift-128-euclidean-hard-expansion (k = 10)

---

sift-128-euclidean-hard-lid (k = 10)

---

sift-128-euclidean-hard-lrc (k = 10)

---

sift-128-euclidean-middle-expansion (k = 10)

---

sift-128-euclidean-middle-lid (k = 10)

---

sift-128-euclidean-middle-lrc (k = 10)

---

Results by Algorithm

Algorithms:
• faiss-ivf
• NGT-onng
• hnsw(faiss)
• puffinn
• annoy

faiss-ivf

---

NGT-onng

---

hnsw(faiss)

---

puffinn

---

annoy

---