The HeapSort algorithm is based upon interpreting an array A[0..N) (or an initial segment thereof, A[0..n) (n≤N)) as storing the values in the nodes of a complete binary tree1 whose nodes are numbered corresponding to the locations in the array. Specifically, the root of the tree is node zero and, in general, node k's left and right children are, respectively, nodes 2k+1 and 2k+2. (Of course, assuming that the tree extends up to, but not including, location n, node k has a left child only if 2k+1 < n and a right child only if 2k+2 < n.)
As an example, the ten-element array
0 1 2 3 4 5 6 7 8 9 +----+----+----+----+----+----+----+----+----+----+ | 4 | 23 | 18 | 2 | 27 | 15 | 10 | 9 | 32 | 5 | +----+----+----+----+----+----+----+----+----+----+ |
would be interpreted as representing the following complete binary tree (where the node IDs are in parentheses):
(0)
+----+
| 4 |
+----+
/ \
/ \
/ \
/ \
/ \
/ \
(1) / \ (2)
+----+ +----+
| 23 | | 18 |
+----+ +----+
/ \ / \
/ \ / \
(3) / \ (4) (5) / \ (6)
+----+ +----+ +----+ +----+
| 2 | | 27 | | 15 | | 10 |
+----+ +----+ +----+ +----+
/ \ /
(7) / (8) \ / (9)
+----+ +----+ +----+
| 9 | | 32 | | 5 |
+----+ +----+ +----+
|
A max-heap is a complete binary tree in which, for every node other than the root, the value stored therein is no greater than that stored in its parent. In general, then, along any "downward" path from a node into its subtree the values in the nodes encountered will be in descending order. In particular, the greatest value in a max-heap is at its root.
HeapSort has two phases, the first of which rearranges the elements in the given array so that the complete binary tree that it represents is a max-heap. This is achieved by applying the siftDown operation to every non-leaf node2 and to perform those applications in a correct order. Specifically, siftDown should be applied to a node only after it has been applied to the node's children. This ensures that the subtree rooted at that node has itself become a max-heap. When, at last, siftDown is applied to the root of the tree, the whole tree is then a max-heap!
Because of how the nodes are numbered, it is easy to ensure that a node's children have had siftDown applied to them before it is applied to the node itself: we simply perform the applications in decreasing order by node number. That is, we start with the highest-numbered non-leaf node, which is node ⌊N/2⌋−1, and work our way back to node zero, the root.
But what does siftDown do? Actually, it's quite simple. It takes the value in a node and "sifts" it downward into the node's subtree until it reaches a node where that value is in proper relationship to the values in its children (if any). (Recall that, in a max-heap, the value in a node must not exceed the value in its parent.)
As the node's original value is being moved downward along a path to its final destination, the values in the nodes encountered along the way are being placed into their parents. In other words, the manner in which the value originally in the node where we started reaches its final destination is via a sequence of swap operations, each of which exchanges the values in a node and one of its children.
To illustrate the siftDown process, consider the figure below, which shows what happens, as a first step, when the operation is applied to the node (on the left) containing 8. Because its left child contains 14, which is greater than 8 (thereby violating the condition defining a max-heap), we swap their values, the result of which is seen on the right. The operation is not necessarily finished yet, as the node now containing 8 may have a child in which is stored a value larger than 8. In that case, a swap between those two nodes would be carried out and the operation would continue even farther "downwards" towards the leaves of the tree.
.
.
|
|
+----+
| 8 |
+----+
/ \
/ \
/ \
+----+ +----+
| 14 | | 10 |
+----+ +----+
/ \ / \
/ \ / \
. . . .
. . . .
|
⟶ |
.
.
|
|
+----+
| 14 |
+----+
/ \
/ \
/ \
+----+ +----+
| 8 | | 10 |
+----+ +----+
/ \ / \
/ \ / \
. . . .
. . . .
|
Note that, in swapping the values in a node and its child, it is vital that the swap be with the child having the larger value among the two. In our diagram above, for example, it would have been incorrect to swap the values in the nodes containing 8 and 10 because it would have left us with a node containing 10 and one of its children containing a larger value, 14.
Here is a pseudocode version of siftDown(). Its first parameter, k, identifies the node whose value is to be "sifted downward" in the tree, if warranted. Its second parameter, n, identifies the boundary of the tree. That is, it is assumed that A[0..n) is being interpreted as representing a complete binary tree. (Locations n and above, if any, are "off limits".)
// precondition: A[0..n) represents a complete binary tree.
siftDown(int k, int n)
leftChild = 2*k + 1;
if leftChild < n then // node k has at least a left child
// assign to m the location of node k's "larger" child
if (leftChild == n-1 || A[leftChild] > A[leftChild+1]
m := leftChild;
else
m := leftChild + 1;
fi
if A[k] < A[m] then
swap(k,m); // Swap the values in nodes k and its child m.
siftDown(m,n); // Recursively sift down from child node m.
fi
fi
//postcondition: Assuming that, beforehand, the subtrees rooted at
// node k's children (if any) were max-heaps, now the subtree rooted
// at node k is a max-heap.
|
HeapSort is carried out, in place3, as follows:
|
Input: Array A[0..N), whose elements are comparable using the < operator Output: A[0..N), the elements of which are in ascending order Phase 1: Heapify the array
// Sift down from every non-leaf node, from the // highest numbered (N/2 - 1) to lowest (0, the root). k := N/2; // Loop invariant: Every node numbered ≥ k is the root of a max-heap. do while (k != 0) k := k-1; siftDown(k,N); od // postcondition: A[0..N) is a max-heap. Phase 2: Deconstruct the heap
// During each iteration, swap the largest value in the // heap (which is at the root, location 0) with the value // in the last location still part of the heap (k-1). // Then sift down from the root. k := N; // Loop invariant: // A[0..k) is a max-heap and // A[k..N)'s elements are in ascending order and // no element in A[0..k) is > than any element in A[k..N) do while (k != 0 ) k := k-1; swap(0,k); siftDown(0,k); od // postcondition: A[0..N) is in ascending order |
As discussed in the previous section, the purpose of Phase 1 is to transform the tree (represented by A[0..N)) into a max-heap. What each iteration of Phase 2 does is to place the correct value into A[k], with k counting downward from N−1 to zero.
For example, consider the first loop iteration. At this point, A[0..N) represents a max-heap and therefore its largest element is at the root, location zero. So we swap the values in locations 0 and N−1, thereby placing the largest array element where it belongs, at the last location of the array! At this point, we no longer consider location N−1 as being part of the tree and it is never touched again.
Before trying to place the next-to-largest element into location N−2, we must restore the tree to being a max-heap, a property that was almost certainly lost by having swapped the value occupying location N−1 into the root. But doing that is easy, as it requires only that we apply siftDown to the root.
Generalizing this reasoning to apply to not just the first iteration of Phase 2 but to any of them, consider that, as each iteration begins, we have that A[0..k) is a max-heap and that A[k..N) already contains the N−k largest values in A[] where they belong. So we take the largest value in what remains of the max-heap, at location zero, and put it into location k−1, where it belongs. The value that had been at location k−1 is placed into the root and then sifted down so as to restore the max-heap property and thereby to restore the truth of the loop invariant in preparation for the next loop iteration.
0 1 2 3 4 5 6 7 8 9 +----+----+----+----+----+----+----+----+----+----+ | 4 | 23 | 18 | 2 | 27 | 15 | 10 | 9 | 32 | 5 | +----+----+----+----+----+----+----+----+----+----+ |
(0)
+----+
| 4 |
+----+
/ \
/ \
/ \
/ \
/ \
/ \
(1) / \ (2)
+----+ +----+
| 23 | | 18 |
+----+ +----+
/ \ / \
/ \ / \
(3) / \ (4) (5) / \ (6)
+----+ +----+ +----+ +----+
| 2 | | 27 | | 15 | | 10 |
+----+ +----+ +----+ +----+
/ \ /
(7) / (8) \ / (9)
+----+ +----+ +----+
| 9 | | 32 | | 5 |
+----+ +----+ +----+
|
During Phase 1, siftDown is applied to nodes 4 down to 0. The following shows how the array changes as a result of each application. (It is left to the reader to draw the pictures of the corresponding trees.)
| siftDown applied to | Effect | Resulting array |
|---|---|---|
| Node 4 | None | ⟨ 4 23 18 2 27 15 10 9 32 5 ⟩ |
| Node 3 | swap with node 8 | ⟨ 4 23 18 32 27 15 10 9 2 5 ⟩ |
| Node 2 | None | ⟨ 4 23 18 32 27 15 10 9 2 5 ⟩ |
| Node 1 | swap with node 3 | ⟨ 4 32 18 23 27 15 10 9 2 5 ⟩ |
| Node 0 | swaps with nodes 1,4,9 | ⟨ 32 27 18 23 5 15 10 9 2 4 ⟩ |
How Phase 2 modifies the array, from iteration to iteration, is shown next. (The vertical bar indicates the boundary between the max-heap in A[0..k) and the rest of the array.) Again, it is left to the reader to draw the pictures of the trees.
| When | Array's contents |
|---|---|
| Initially | ⟨ 32 27 18 23 5 15 10 9 2 4 |⟩ |
| After k=10 iteration | ⟨ 27 23 18 9 5 15 10 4 2 | 32 ⟩ |
| After k=9 iteration | ⟨ 23 9 18 4 5 15 10 2 | 27 32 ⟩ |
| After k=8 iteration | ⟨ 18 9 15 4 5 2 10 | 23 27 32 ⟩ |
| After k=7 iteration | ⟨ 15 9 10 4 5 2 | 18 23 27 32 ⟩ |
| After k=6 iteration | ⟨ 10 9 2 4 5 | 15 18 23 27 32 ⟩ |
| After k=5 iteration | ⟨ 9 5 2 4 | 10 15 18 23 27 32 ⟩ |
| After k=4 iteration | ⟨ 5 4 2 | 9 10 15 18 23 27 32 ⟩ |
| After k=3 iteration | ⟨ 4 2 | 5 9 10 15 18 23 27 32 ⟩ |
| After k=2 iteration | ⟨ 2 | 4 5 9 10 15 18 23 27 32 ⟩ |
| After k=1 iteration | ⟨ | 2 4 5 9 10 15 18 23 27 32 ⟩ |
In terms of space (i.e., quantity of memory used, aside from that used to store the input data, which here is an array), HeapSort uses only a constant amount (i.e., O(1)). This is superior to QuickSort, which uses O(log N) space (to store on a stack the "left" and "right" boundaries of the array segments yet to be sorted)4 and MergeSort, which uses O(N) space.5
As for time, Phase 2 applies siftDown N times, each time to the root of a complete binary tree. An application of siftDown takes time proportional to the number of nodes in the path that is followed while moving a value downward in the tree. In the worst case, that downward path goes all the way to a leaf. (And, indeed, in HeapSort's Phase 2, that is likely.) Assuming a complete binary tree with N nodes, that worst-case distance is essentially lg N. Thus, during each of the N iterations, O(log N) time is taken, for a total of O(N·log N)6 time.
A cursory analysis of Phase 1 also leads us to characterize it as taking O(N·log N) time. We note that siftDown is applied N/2 times and that a given application of it can take O(log N) time. This produces the result O(N·log N).
However, a more careful analysis reveals that Phase 1 takes only O(N) time. (This doesn't affect the overall asymptotic running time of HeapSort, because O(N + N·log N) = O(N·log N). It is interesting in it own right, however.)
For purposes of this discussion, let us take the simplest case, which is when the array's length is 2k-1 for some k. That means that the tree is perfect in that every level is completely full of nodes, including the bottommost. (The number of nodes in the levels would be, respectively, 1, 2, 4, ..., 2k−1.)
The worst-case running-time of applying siftDown to a node is proportional to the number of nodes along a path from it to a leaf. In Phase 1, we apply siftDown to every non-leaf node. For purposes of this discussion, suppose that we applied it to leaf nodes, too. (That could only increase the running time, of course.) So we charge every node in the tree with the cost of applying siftDown to it, and add up all those charges.
But half the nodes in the tree are leaves (and get charged 1 unit of time), a quarter of the nodes are parents of leaves (and get charged 2 units of time), an eighth of the nodes are grandparents of leaves (and get charged 3 units of time), etc., etc. Indeed, in general (N+1)/2i nodes in the tree get charged i units of time, where i goes from 1 to k (recall that N = 2k − 1). If you add those up, you get a result less than 2N (see below for the calculation). Thus, the running time of Phase 1 is O(N).
| Σ
i·((N+1)/2i))
1≤i≤lg(N+1) |
||
| = | Σ
((N+1)·(i/2i))
1≤i≤lg(N+1) |
(algebra) |
| = | ( Σ
(i/2i)) · (N+1)
1≤i≤lg(N+1) |
(property of sums) |
| ≤ | ( Σ
(i/2i)) · (N+1)
1≤i≤∞ |
(each term is positive) |
| = | 2(N+1) | (well-known sum) |
| Original Tree | After Sifting Down from Nodes 4 and 3 |
After Sifting Down from Nodes 2 and 1 |
After Sifting Down from Node 0 |
|---|---|---|---|
4
/ \
/ \
/ \
/ \
/ \
23 18
/ \ / \
/ \ / \
/ \ / \
2 27 15 10
/ \ /
/ \ /
9 32 5
|
4
/ \
/ \
/ \
/ \
/ \
23 18
/ \ / \
/ \ / \
/ \ / \
32 27 15 10
/ \ /
/ \ /
9 2 5
|
4
/ \
/ \
/ \
/ \
/ \
32 18
/ \ / \
/ \ / \
/ \ / \
23 27 15 10
/ \ /
/ \ /
9 2 5
|
32
/ \
/ \
/ \
/ \
/ \
27 18
/ \ / \
/ \ / \
/ \ / \
23 5 15 10
/ \ /
/ \ /
9 2 4
|
| Before 1st Iteration | After 1st iteration | After 2nd iteration |
|---|---|---|
32
/ \
/ \
/ \
/ \
/ \
27 18
/ \ / \
/ \ / \
/ \ / \
23 5 15 10
/ \ /
/ \ /
9 2 4
|
27
/ \
/ \
/ \
/ \
/ \
23 18
/ \ / \
/ \ / \
/ \ / \
9 5 15 10
/ \ /
/ \ /
4 2 (32)
|
23
/ \
/ \
/ \
/ \
/ \
9 18
/ \ / \
/ \ / \
/ \ / \
4 5 15 10
/ \ /
/ \ /
2 (27)(32)
|
| After 3rd iteration | After 4th iteration | After 5th iteration |
18
/ \
/ \
/ \
/ \
/ \
9 15
/ \ / \
/ \ / \
/ \ / \
4 5 2 10
/ \ /
/ \ /
(23) (27)(32)
|
15
/ \
/ \
/ \
/ \
/ \
9 10
/ \ / \
/ \ / \
/ \ / \
4 5 2 (18)
/ \ /
/ \ /
(23) (27)(32)
|
10
/ \
/ \
/ \
/ \
/ \
9 2
/ \ / \
/ \ / \
/ \ / \
4 5 (15) (18)
/ \ /
/ \ /
(23) (27)(32)
|
| After 6th iteration | After 7th iteration | After 8th iteration |
9
/ \
/ \
/ \
/ \
/ \
5 2
/ \ / \
/ \ / \
/ \ / \
4 (10) (15) (18)
/ \ /
/ \ /
(23) (27)(32)
|
5
/ \
/ \
/ \
/ \
/ \
4 2
/ \ / \
/ \ / \
/ \ / \
(9) (10) (15) (18)
/ \ /
/ \ /
(23) (27)(32)
|
4
/ \
/ \
/ \
/ \
/ \
2 (5)
/ \ / \
/ \ / \
/ \ / \
(9) (10) (15) (18)
/ \ /
/ \ /
(23) (27)(32)
|
| After 9th iteration | ||
2
/ \
/ \
/ \
/ \
/ \
(4) (5)
/ \ / \
/ \ / \
/ \ / \
(9) (10) (15) (18)
/ \ /
/ \ /
(23) (27)(32)
|
[1] Recall that a complete binary tree is one in which, at every "level" of the tree, the maximum number of nodes is present (e.g., 1, 2, 4, 8, etc.), with the possible exception of the bottommost level, where all nodes present must be "as far to the left" as possible.
[2] siftDown has no effect when applied to a leaf node and therefore there is no point in doing so.
[3] An algorithm that processes an array is said to work "in place" if the quantity of space (i.e., memory) that it uses is O(1) (i.e., a constant), as opposed to being dependent upon (i.e., a function of) the array's length.
[4] To achieve a O(log n) space bound requires a slightly clever implementation of QuickSort. A straightforward approach could result in O(n) space being used.
[5] An extremely clever and complicated version of MergeSort achieves a O(1) space bound.
[6] It is true that each iteration in Phase 2 makes the tree get smaller, and thus its height decreases over time, but consider that its height remains the same for up to N/2 iterations, and decreases by at most one during those iterations. Thus, each of the first N/2 iterations could take up to O(log N) time, which gives us O(N·log N) time already.