Spatiotemporal Short-term Traffic Forecasting using the Network Weight Matrix and Systematic Detrending

Recent working paper:

LookBackWindowsThis study examines the dependency between traffic links using a three-dimensional data detrending algorithm to build a network weight matrix in a real-world example. The network weight matrix reveals how links are spatially dependent in a complex network and detects the competitive and complementary nature of traffic links. We model the traffic flow of 140 traffic links in a sub-network of the Minneapolis – St. Paul highway system for both rush hour and non-rush hour time intervals, and validate the extracted network weight matrix. The results of the modeling indi- cate: (1) the spatial weight matrix is unstable over time-of-day, while the network weight matrix is robust in all cases and (2) the performance of the network weight matrix in non-rush hour traffic regimes is significantly better than rush hour traffic regimes. The results of the validation show the network weight matrix outperforms the traditional way of capturing spatial dependency between traffic links. Averaging over all traffic links and time, this superiority is about 13.2% in rush hour and 15.3% in non-rush hour, when only the 1st -order neighboring links are embedded in modeling. Aside from the superiority in forecasting, a remarkable capability of the network weight matrix is its stability and robustness over time, which is not observed in spatial weight matrix. In addition, this study proposes a naïve two-step algorithm to search and identify the best look-back time win- dow for upstream links. We indicate the best look-back time window depends on the travel time between two study detectors, and it varies by time-of-day and traffic link.

The End of Driving — Money and Politics Podcast

My podcast on the Ricochet Podcast: Money and Politics with Jim Pethokoukis: Episode 21: The End of Driving is now up

Opens the player in a new window so you can listen uninterrupted while continuing to browse the site in this window.  Please allow to open a pop-up window for this to work.

Direct link to MP3 file

It’s not just driverless cars. In the latest Ricochet Money & Politics podcast, economist, transportation expert, and blogger David Levinsonargues traffic is declining and will continue to decline dramatically in the coming decades. And that decline is not only the result of some deeper economic and technological trends, but will itself cause a radical restructuring of American society.

What happened to traffic? — The Transportationist

Traffic on Washington Avenue – Raw data edition |

Cross-posted from  Traffic on Washington Avenue – Raw data edition.

“Why does this matter? By being “conservative” and adjusting traffic counts up, they are over-estimating the need for roadway capacity, that is, they are being “liberal” with the number of lanes required to ensure a particular level of service.”

Traffic on Washington Avenue – Raw data edition


Prompted by Brendon’s recent post (and a Twitter conversation with Ethan and Janne), I got interested in some of the traffic counts and engineering on Washington Avenue. I looked at the Washington Avenue Traffic Operation Analysis by Alliant Engineering for Hennepin County. On p.11 of this document are traffic “counts” the report said were collected in Spring of 2011. The traffic added up perfectly, and with some experience looking at traffic counts in a previous life (some people’s previous lives were as Cleopatra or the King of England, like Bill Gates I was a traffic counter), I had to believe some massaging was going on, data never comes out that clean, especially when it is collected on multiple dates. For instance, there is conservation of flow in traffic, every car that enters and intersection must leave it (unless it is raptured). A traffic count at one site on one date will ensure this. Similarly every car that leaves an upstream intersection must arrive at the downstream intersection, after controlling for driveways. A traffic count on one date is likely inconsistent with another date. Other reasons for massaging include inconsistent peak periods (the peak time at intersection A may differ from downstream intersection B).

In one of the great blessings of open data, Minneapolis makes its raw traffic counts available online. So I went to their website and looked for myself, under turning movement counts [TMC] on Washington Avenue South. These counts are summarized in Table 1 for Eastbound traffic (and the first 5 columns of Table 2 for Westbound traffic), along with the numbers from Alliant’s report. As you can see Alliant’s numbers (data column 3) are 10 to 35% higher than the counts in the City of Minneapolis database for the same period (data columns 1 and 2) (and I assume these are the source of Alliant’s resulting numbers, though the report is vague on the traffic count details). The ratios are given in data column 4.

Why does this matter? By being “conservative” and adjusting traffic counts up, they are over-estimating the need for roadway capacity, that is, they are being “liberal” with the number of lanes required to ensure a particular level of service.

Table 1: Eastbound AM flows on Washington from Hennepin to 11th Avenue

Cross-street Count Inflow Count Outflow Alliant Inflow (Fig 5) Alliant/Count
Hennepin 1061 906 1172 1.10
Nicollet 880 887 972 1.10
Marquette 831 860 988 1.19
2nd Ave 812 773 1044 1.29
3rd Ave 753 890 952 1.26
4th Ave 923 561 1130 1.22
5th Ave 537 642 620 1.15
Portland 640 462 744 1.16
Park Ave 474 578 532 1.12
Chicago 702 666 724 1.03
11th Ave 593 776 656 1.11


Table 2: Westbound AM flows on Washington from 11th Avenue to Hennepin

Cross-street Count Inflow Count Outflow Alliant Inflow Alliant/Count Dominant Dir. WB/EB WB/ln w/ Rev. EB/ln w/ Rev. Lane Split
11th Ave 1046 1107 1268 1.21 W 1.764 349 593 3/1
Chicago 1120 870 1262 1.13 W 1.595 373 702 3/1
Park Ave 875 998 1040 1.19 W 1.846 292 474 3/1
Portland 875 874 1184 1.35 W 1.367 437.5 320 2/2
5th Ave 921 1131 1068 1.16 W 1.715 460.5 268.5 2/2
4th Ave 1155 783 1356 1.17 W 1.251 577.5 461.5 2/2
3rd Ave 786 667 928 1.18 W 1.044 393 376.5 2/2
2nd Ave 607 522 756 1.25 E 0.748 303.5 406 2/2
Marquette 595 612 662 1.11 E 0.716 595 277 1/3
Nicollet 555 535 702 1.26 E 0.631 555 293 1/3
Hennepin 594 573 686 1.15 E 0.560 594 354 1/3


Reversible Lanes

I was also interested in some other aspect of traffic. Ethan said the traffic was balanced on Washington after I posited that it was unbalanced, and we could consider reversible lanes. In fact it is both, depending on where you are looking. The final columns of Table 2 identify the dominant direction, the directional ratio (WB/EB flow), and what flows would be with the lane split given in the final column. At 3rd Avenue, traffic is balanced, to the East there is much higher westbound traffic in the morning, to the West there is much higher eastbound traffic in the morning. Along Washington Avenue, the midpoint of downtown traffic is 3rd Avenue (not Nicollet as I would have supposed before looking at the numbers).

Is the imbalance sufficient to justify reversible lanes? The case is marginal. In general, with two lanes in each direction, left turn lanes, and good signal timing, I think a 2/2 split should work well enough. Near I-35W a 3/1 split (3 lanes westbound, 1 lane eastbound in the AM) is plausible. Similarly on the westside of downtown, a 1/3 split is also plausible in the reverse direction.

I am leaving the PM analysis as an exercise for the reader.

The Minneapolis Turning Movement Counts can be found here: WashAveAMFlows.pdf andWashAvePMFlows.pdf.


Atlanta traffic bad but predictable

I get interviewed about the reliability measures of the new Urban Mobility Report by Ariel Hart of the Atlanta Journal-Constitution: Atlanta traffic bad but predictable :

“‘People care about this,’ said David Levinson, a professor of civil engineering at the University of Minnesota who researches traffic psychology surrounding reliability. People will even accept more congestion to get more reliability, he said, and he has a mathematical formula to calculate how much.
‘It’s the surprises, the inconsistency of the delay that makes it difficult,’ costing people social capital with colleagues, clients and friends when they are unexpectedly late, he said.”

The reliability ratio, the ratio of the value of reliability to the value of time us about 1, depending on how it is measured. See

Does TTI underestimate historic congestion levels?

To read the Texas Transportation Institute’s Urban Mobility Report is to believe congestion has more than doubled since 1982 (really between 1982 and 2000). From one perspective, of course congestion must have risen, demand (Vehicle Miles Traveled, Population, etc.) increased significantly over this period while supply (Lane Miles of Road Capacity) did not increase at nearly the same rate.
But I was alive in 1982, I was in cars at that age (and driving myself the next year) (in Central Maryland). I remember congestion in the 1980s. To misquote Lloyd Bentsen, “Congestion was a friend of mine”, and TTI seems to be saying to 1982 “You’re no congestion”. But congestion doesn’t seem appreciably different from today. People complained about it then as much as now. Some bottlenecks have been fixed, new ones have emerged.
So I wonder whether congestion did, in fact, “double”.
Some hypotheses:
1. Measurement issues. Continuous roadway travel time measurements were a lot scarcer in the 1980s than today. Freeways now have loop detectors on every segment, whereas there might have been a permanent recording station every 5 or 10 miles in the 1980s, so a lot more had to be estimated and approximated. There are still no good arterial measurements, the most recent Urban Mobility Report uses GPS data from Inrix, and this will clearly come to dominate congestion measures. Notably, including this measurement forced TTI to re-estimate downward their historical congestion measurements.
2. Definition: As noted by Joe Cortright’s report Driven Apart, mobility is not accessibility. A city where I can reach everything in 10 minutes, but travel at 30 MPH (when freeflow is 60 MPH) is more congested than one where I can reach everything in 30 minutes, but can travel at freeflow conditions. The TTI in a sense penalizes efficient land uses.
3. Induced Demand: Highway expansion tends to get used up (this is not a bad thing of itself, just a thing), so much of road expansion gets eaten up in more traffic. Similarly highway reduction reduces travel. Duranton and Turner write “We conclude that an increased provision of roads or public transit is unlikely to relieve congestion.”
This does not explain why congestion is under-estimated in the past though.
4. Congestion vs. Speed: Travel times on journey to work increased only marginally over this period. Average distances for trips rose faster than travel times, indicating average travel speeds increased. So even with increasing congestion, if travelers shifted to relatively faster (e.g. suburb to suburb freeways) from slower (e.g. suburb to city arterials), congestion can rise on each link, but travel speeds still increase. See The Rational Locator for an example of this.
5. Perspective: This previous point about perception can be refamed as one of perspective. There are differences between spatial averages (which TTI uses) and person-based averages (which individual observers perceive). So the person based average for any metropolitan resident may be the same, but the amount of space (network) covered by congestion may increase if the total amount of space which is developed increases. Similarly, if there is peak spreading, congestion occurs over a longer duration.
However, TTI is not simply saying that the amount of area that is congested increased, they are claiming, for Washington DC the delay per person increased from 20 hours per year in 1982 to 74 hours in 2010.
I am willing to believe that with recent measurements, 74 hours per year for an average commuter in DC is plausible in 2010, since that is just under 10 minutes each way each day for 225 work days per year. 10 minutes of delay on a 30 minute commute means the freeflow time on that commute (un-delayed, e.g. Sunday morning) was 20 minutes. This seems about right for the “average” commuter. Rush hour is when everyone has to slow down.
But this implies in 1982 that delay was less than 3 minutes a day per commuter each way. That seems unreasonably small when you think about it, I could have spent 3 minutes at a traffic light in DC at the time, and that certainly constitutes delay. They are saying for every person who had a 10 minute delay, 2 people had 0 delay to get an average 3 minute delay, and that is not the metropolitan Washington I was familiar with. Congestion was sufficiently important than that radio stations had regular traffic reports, and traffic helicopters, it was not something insignificant.
Of course this is impossible to fully validate, as we cannot go back in time and accurately measure speed. The best I could think of was using the Google NGram feature to track mention of some keywords in books. This proves nothing unfortunately, and suggests a small uptick in the word “traffic” in the 1990s, but is interesting none-the-less.
One however can imagine the motivation for wanting congestion to appear lower in the past than it actually was. This means congestion is rising faster, and thus creates a greater claim on the public weal than if congestion were always with us at roughly the same level.

Why does the Institute of Transportation Engineers exist? 10 Ideas for Big Changes – Mike on Traffic

Mike Spack vs. ITE: Why does the Institute of Transportation Engineers exist? 10 Ideas for Big Changes. See his post for the list.

I am disappointed Mike took down his spreadsheet, though I understand why. If he were at a University, they wouldn’t dare. Frankly, the ITE trip generation data is mostly like a telephone book and can’t be copyrighted, though its specific presentation (and maybe the regressions, though those seem pretty damn uncreative to me) can be. An analysis of that data is certainly fair game. An alternative though would be to set up a Trip Generation Wiki or Google Docs which is open, letting people upload their own data and updating the regressions automatically (since it is a pretty trivial spreadsheet operation).
I am thinking of unjoining ITE, my last professional organization (I quit APA a long time ago due to their profit-maximizing behavior since I gained nothing from the organization and they wanted a non-trivial share of my salary) over their heavy-handed, anti-public, guild-like behavior. If Mike were President, I would reconsider. The backwardness of ITE is one of many reasons Traffic Engineers are becoming increasingly unpopular.

SMART Signal

UM News reports on my colleague Henry Liu’s new SMART Signal Technologies startup: University of Minnesota startup to improve traffic flow on congested roads:

“Based on research from the University of Minnesota, SMART Signal Technologies, Inc., will commercialize a system to better predict and manage the flow of traffic on roads controlled by traffic lights. The system could potentially cut down on traffic congestion and help drivers save both time and fuel.
Using data from existing traffic signal equipment, the system accurately calculates queue length at signalized intersections. These data, collected in real time and archived in a database, will allow cities across the state to better mediate the flow of traffic at peak times using real time performance measures provided by the system.”

I have talked about this before. I hope it gets widely deployed, what we don’t know about travel times arterials in real-time is embarrassing.

Gridlock as China begins its ‘Golden Week’ holidays


From the Telegraph (via AE): Gridlock as China begins its ‘Golden Week’ holidays :

“When 1.3 billion people all go on holiday at the same time, a little chaos is perhaps to be expected.
But it was a generous decision by Chinese politicians to grant free road travel, by suspending motorway tolls, that saw hundreds of thousands of drivers spend the first day of the Mid-Autumn Festival on Sunday in gridlock.”

Marked Crosswalks Considered Harmful


In 1968 there was a famous Computer Science article Go To Statement Considered Harmful by Edsger W. Dijkstra (of algorithm fame). It says in part:

My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible.

In early 21st Century America, pedestrian crosswalks may be marked or unmarked. Whether a crosswalk is marked is functionally based on the whim of the traffic department. A fuller discussion of issues about “how” to use crosswalks (from the Town of Brookline, Massachusetts) is here, but not “when” to use them, hence my use of the term “whim”, which says engineering studies are required, but does not have hard and fast rules about application.
Interesting the Brookline document asserts:

Marked crosswalks are viewed widely as “safety devices,” and most municipalities give the pedestrian the right-of-way when within them. However, there is strong evidence that these facts prompt many pedestrians to feel overly secure when using a marked crosswalk. As a result, pedestrians will often place themselves in a hazardous position by believing that motorists can and will stop in all cases, even when it may be impossible to do so. It is not unusual for this type of aggressive pedestrian behavior to contribute to a higher incidence of pedestrian accidents and cause a greater number of rear-end collisions. In contrast, a pedestrian using an unmarked crosswalk generally feels less secure and less certain that the motorist will stop and thereby exercise more caution and waiting for safe gaps in the traffic stream before crossing. The end result is fewer accidents at unmarked crosswalks.

Implicitly the document blames pedestrians for asserting their rights, rather than drivers for violating them.
I posit that if you are a trained, but human driver, whose “intellectual powers are rather geared to master static relations” you will generally respect crosswalks. You will believe, just as all stop signs are marked, all legal crosswalks are marked. As “our powers to visualize processes evolving in time are relatively poorly developed” you will disrespect unmarked crosswalks, since if they were legitimate, you reason, they would be marked. You may not even notice them if they come from side streets for which you have no stop sign of traffic signal. They only appear relevant when there is a person surprising you in the road. Hence you will be aggressive to pedestrians trying to cross at unmarked crosswalks, as you will (wrongly) believe you have the right-of-way. Pedestrians will in turn be intimidated as suggested by the Brookline document above. Research about driver and pedestrian behavior can be found in this paper by Mitman et al. It notes:

Driver yielding behavior was a statistically significant variable at all six observation sites. For all road types, pedestrians in the marked crosswalk were more likely than pedestrians in the unmarked crosswalk to have drivers immediately yield the right-of-way to them.


Average gap acceptance was a statistically significant variable at five of the observation sites. At all five locations, pedestrians in the unmarked crosswalk were more likely than pedestrians in the marked crosswalk to wait for larger gaps in traffic before crossing. This finding was consistent across all road types.

The empirical findings are sound as far as they go. I disagree with the recommendations.
The problem is inconsistent ambiguity.
Solution A. Mark all crosswalks.
If we were completely consistent about where pedestrians might be found, (i.e. crosswalks) that would be acceptable, drivers and pedestrians would both understand the law. It would be clearly spelled out to drivers where pedestrians might be, including smaller intersections that might otherwise be raced by. It would be bad from a pedestrian rights perspective, as it over channelizes walkers and gives too much power to cars.
By implication, it requires pedestrians to use only marked crosswalks. It in a sense delegitimizes jaywalking. It increases pedestrian travel times. As Peter Norton notes in Fighting Traffic:

“Before the American city could be physically reconstructed to accommodate automobiles, its streets had to be socially reconstructed as places where cars belong.” “Until then, streets were regarded as public spaces.” [Quoted in Planning Pool]

In practice, we will not mark all crosswalks. The vast majority of intersections in the US are unmarked, and no one wants to spend the money to mark them all. Hence if we claim to adopt solution A, we will in fact resign ourselves to inconsistent ambiguity (false certainty) or crosswalk markings.
Solution B. Unmark all crosswalks.
In contrast, if we were completely (i.e. consistently) ambiguous about where pedestrians would be, that would be good from both a safety perspective, and in the long run, a pedestrian rights perspective. While in the mixed environment, pedestrian might wait more, in the no crosswalks environment, pedestrians will be cautious where they are now reckless. But pedestrians would also be more assertive in more places (those without crosswalks now) as they would know that drivers would be also be more cautious. This strategy will make both drivers and pedestrians more aware of their surroundings since pedestrians might be anywhere. (See shared space.)
In addition to unmarking all crosswalks, we should put up periodic reminder signs/messages to drivers when entering new districts, leaving freeways, etc. that pedestrians have the right-of-way. We might put up markers where pedestrians have died to somber-up drivers. (Further, we ought to develop some hand-signal communication protocol so pedestrians can signal drivers they are about to enter the roadway. Reuben Collins has a nice discussion here.).

It is the false expectation of consistency that causes many of the 4,280 pedestrian deaths per year in the United States.
I strongly prefer Solution B. Do we have any examples of this in the United States over a widespread area? A single street with shared space would be insufficient to draw conclusions.
Comment: this is the same argument as about Class III Bikeways. Since Class III Bikeways give bicyclists no advantage, they imply to drivers that on any unmarked road, they have rights over bikes (when they don’t).
Comment: Yes I did see a driver yell at a pedestrian for crossing an unmarked crosswalk again today, and the intimidated pedestrian ran after trying to yield the road.