Theory

The answer to lesson 6

Lesson 6's question. The same heap, the same work — push all 10,000 in and drain it — two ways.

$ algo pq -in s10k.jsonl -rounds 20
n = 10000, push-all then drain, 20 rounds

                              per round
  hand-written heap             1.469ms
  container/heap                2.395ms

  container/heap is 1.63x slower

And properly, with testing.B, five runs each:

$ go test -run XXX -bench 'Heap' -benchtime=20x -count=5

BenchmarkHeapMine-32      20    1194035 ns/op
BenchmarkHeapMine-32      20    1216500 ns/op
BenchmarkHeapMine-32      20    1184885 ns/op
BenchmarkHeapMine-32      20    1251305 ns/op
BenchmarkHeapMine-32      20    1172580 ns/op
BenchmarkHeapStdlib-32    20    1964010 ns/op
BenchmarkHeapStdlib-32    20    2018915 ns/op
BenchmarkHeapStdlib-32    20    2039655 ns/op
BenchmarkHeapStdlib-32    20    2015920 ns/op
BenchmarkHeapStdlib-32    20    2013920 ns/op

1.17–1.25 ms against 1.96–2.04 ms. The ranges do not touch. container/heap is 1.66× slower.

In lesson 6 the standard library won. Here it loses, and not narrowly.

But WHY — that is the lesson

In lesson 6 the reason was one line: the closure is known at compile time, so Go inlines it straight into the loop, and after optimisation there is no call left.

What stops that happening here? Look at the interface signatures again:

func (s *stdItems) Push(x any)  { *s = append(*s, x.(Item)) }
func (s *stdItems) Pop() any    { ... }

any. Every Item is boxed into an interface value on the way in and unboxed on the way out. An Item is 56 bytes and an interface value is 16, so the struct does not fit inside one: it is copied to the heap and the interface holds a pointer to it.

Copying to the heap is an allocation. Add -benchmem:

$ go test -run XXX -bench 'HeapMine|HeapStdlib' -benchtime=20x -count=3 -benchmem

BenchmarkHeapMine-32      20   1256900 ns/op    565248 B/op        1 allocs/op
BenchmarkHeapStdlib-32    20   2026515 ns/op   3934086 B/op    20020 allocs/op

One against 20,020

Your heap: one allocation. NewMinHeap(len(items)) takes the whole slice it needs up front, and memory is never touched again.

container/heap: 20,020. Of which

  • 10,000 are the Pushes, each boxing an Item into any;
  • 10,000 are the Pops, each returning an any;
  • and 19 more while the slice grows from nothing (lesson 2's append, measured there).

There is a second suspect too. Less and Swap are interface methods, called through a table: the compiler does not know which type will turn up, so it cannot inline them. Less is one comparison. Swap is one swap. The cost of such a call is the same order as the work it wraps.

Two candidate causes, then — boxing and dispatch — and this measurement does not separate them, because container/heap imposes both at once. Drill 4 separates them, and the answer is sharper than you might expect.

What actually differs

lesson 6, sort.Search lesson 13, container/heap
what is passed a closure, known right there a type, hidden behind an interface
what the compiler does inlines it — no call remains cannot; calls through a table
is the data boxed no yes, 20,000 times
result the abstraction is free 1.66× slower

Lesson 6 said:

There is a rule that reconciles both cases, but it is not about the standard library. It is about what the compiler can see through. In this lesson it could see everything. Next time it will not.

This is next time. The rule is step 7.

And another boundary for the counter

Notice what did not appear once in this step: comparisons.

Both heaps do exactly the same number of them — it is the same algorithm. The counter would report a tie and would be right. The whole 1.66× lives in 20,000 allocations and a table of calls, and there is not one comparison among them.

lesson what the counter could not see
4 memory
6 itself — it moved the quantity it was measuring
11 memory layout — the same 5000.5, 4.5× different time
12 rotations — 23% fewer comparisons, 1.65× slower
13 boxing and dispatch — equal comparisons, 1.66× slower

The fifth time. And the conclusion is the same every time: the counter measures the algorithm, the clock measures the program. You need both.