Explore possibilities for storing speed statistics in higher fidelity

[bmander]

The plan is to digest GPS point updates into speed statistics for way segments. The statistics will be keyed to:

way_id, node_start_index, node_end_index, hour_of_week

The easiest way to store speed statistics as a rolling average, with fields

count, mean

In this case, every time a new speed is detected which matches the key, count = count+1 and mean=((mean*count)+newSpeed)/(count+1). That's simple enough.

But what if we want to know the distribution of speeds? How can we update such a distribution incrementally?

[abyrd]

One straightforward way is to just store histograms in arrays, with each element in the array being a bin of 1 kph, 5kph or whatever. When a new measurement comes in you increment the corresponding bin. If this turns out to be too voluminous the usual combination of delta coding / variable byte coding / run length coding should compress this down nicely.