evaluation

Transportist: April 2022

| David Levinson

Welcome to the latest issue of The Transportist, especially to our new readers. As always you can follow along at the  transportist.org or on Twitter.

Follow-up

  • Sydneysiders will get  12 days of free travel across the public transport network after “the NSW Rail, Tram and Bus Union threatened to strike every Friday in June unless the government instituted a period of free travel for commuters as an apology for last month’s fiasco when trains shut down for 24 hours.” This occurs during the school holidays when demand is lower.

Evaluating Evaluation

We need a way of assessing strategic (i.e. UTPS-like ‘four-step’) transport models. 

My first real job was working as a transport modeler for the Montgomery County, Maryland Planning Department. At the time, models were about a third of a century old, dating from the mid-1950s, deriving from the work of Douglas Carroll et al. in Detroit and then Chicago. I worked on developing a new model for the Washington DC region, (as we didn’t really trust the model from MWCOG, and wanted our own), Travel/2, using then current data and ideas in transport modeling. It had some good features which are yet to be mainstream like ensuring travel time consistency between trip distribution and route assignment, and used then au courant logit mode choice models and detailed trip generation. It wasn’t an agent (activity)-based model, the world wasn’t quite ready, TranSims was just being developed, though we played with the idea some and I wrote not very efficient Fortran code for population synthesis. Route assignment was still static (though multi-class), DTA was not an off-the-shelf product. We tried to document everything, (which is probably sitting in WordPerfect files on my computer) though there was no internet or GitHub to host our code or macros, and of course the source code for the modeling platform (EMME/2) was and remains proprietary, not open access.

Models have not advanced as much in the past third of a century since I started as we would have hoped at the time. Certainly they have moved in the direction we anticipated, just very slowly. Ideally we would assess their accuracy, there is some literature on ex-post analysis of course, and we have done that, but that takes too long, years are required to know whether the forecast was accurate, and by then decisions have been made and the models have evolved, so anything that was wrong was obviously obsolete and cannot be used to criticise the new models.

It should however be possible to assess the models (and each component model) themselves on a number of criteria, e.g. (but not limited to):

  • Transparency,
  • Replicability,
  • Methodological Consistency,
  • Methodological Quality (without prescribing a particular modeling technique, but definitely proscribing those that are known inferior),
  • Internal Consistency (the input travel times to travel demand equal the outputs from route choice, e.g.),
  • Calibration and Validity (how does the base model compare with observed data of various kinds, has backcasting been done, are the results accurate, … is this done systematically, following standard documented procedures?),
  • Recency of Estimation Data,
  • etc.

And develop some kind of scorecard that individuals or teams can apply to strategic transport models, roughly similar to how the ITDP BRT scorecard works. The score should not be a weighted average, but a product of the scores of the components. A zero on any attribute would result in a 0 for the whole thing. Models could then be rated as Gold, Silver, Bronze, or Tin. This can be used to argue for model improvements by external comparison in some sort of systematic (rather than ad hoc) way, perhaps using league tables to compare models, as well as providing a means to discredit bad forecasts, which is what scares away modellers from this approach. 

The Zephyr Foundation aims to improve transport modeling, but it is largely organised by modelers themselves, and thus subject to political sensitivities.

Similarly, we need a way of assessing Business Cases (Benefit-Cost Analyses) (BCA) based on how good the methodology is.

Obviously many BCA in Australia often fail on the criteria of transparency and replicability, as they are tightly held as “Cabinet in Confidence” or “Commercial in Confidence”, so no one knows how the results were actually established. By the product rule, if they fail on that criteria, they fail on everything else, as none of the other criteria can be properly evaluated.

We can ask for some of the same kind of criteria as above. There are various industry committees (TRB, ATAP) that aim to improve and standardise methods, but I am not aware of anyone from outside the sector who is actually scoring the quality of the evaluation. Let me know of any.

Research

  • Lahoorpoor, Bahman, and David Levinson. 2022. “In Search of Lost Trams: Comparing 1925 and 2020 Transit Isochrones in Sydney.” Findings, March. https://doi.org/10.32866/001c.33040. Has Sydney lost access by removing its extensive tram network? We compare the 1925 tram network with today’s bus network, and conclude that the access provided today exceeds what would have been provided by just trams. The Sydney CBD would have had better access if 1925’s central tram lines were still in operation.
  • Access by Trams and Trains in Sydney and Melbourne in the 1920s. Sydney map by Bahman Lahoorpoor inspired by map of MelbourneMap of Sydney and Suburbs showing railway lines including trams. (1925)Map of Sydney and Suburbs showing railway lines including trams, buffered. (1925)Metropolitan Town Planning Commission Map of Melbourne and Environs: Minimum Railway and Tramway Time Zones.[Cam Booth sells a restored version of the Melbourne map, suitable for framing]
  • The latter image is on the Cover of Elements of Access,

Elements of Access: Transport Planning for Engineers, Transport Engineering for Planners. By David M. Levinson, Wes Marshall, Kay Axhausen.

A Pattern Language.jpg

Research by Others

News and Opinion

Substack App

You can now read Transportist in the new Substack app for iPhone.

Now available for iOS

Get the app

With the app, you’ll have a dedicated Inbox for my Substack and any others you subscribe to. New posts will never get lost in your email filters, or stuck in spam. Longer posts will never cut-off by your email app. Comments and rich media will all work seamlessly. Overall, it’s a big upgrade to the reading experience.

The Substack app is currently available for iOS. If you don’t have an Apple device, you can join the Android waitlist here.

Link-based Internal and Full Cost Analysis

| David Levinson

Full cost of travel on the Twin Cities Road Network ($/veh-km)

Recent working paper:

This paper develops a link-based full cost model, which identifies the key cost components of travel, including both internal and external versions of cost, and gives a link-based cost estimate. The key cost components for travelers are categorized as time cost, emission cost, crash cost, user monetary cost, and infrastructure cost. Selecting the Minneapolis – St. Paul (Twin Cities) Metropolitan region as the study area, the estimates show that the average full cost of travel is $0.68/veh-km, in which the time and user monetary costs account for approximately 85% of the total. Except for the infrastructure cost, highways are more cost-effective than other surface roadways considering all the other cost components, as well as the internal and full costs.

Evaluation in a Time of Uncertainty

| David Levinson

I was recently in Copenhagen, where I gave a keynote talk at the UNITE Conference (Uncertainty in Transport Evaluation) at DTU. My thanks to the organizers for inviting me. (I really liked Copenhagen, more on that in a later post).

My talk was titled: Evaluation in a Time of Uncertainty

Abstract: We live in a world where the future is increasingly unpredictable. How should we evaluate transportation investments? Should we even try to evaluate investments in advance? What do we know about what people value? This talk will consider directions in transportation evaluation, and suggestions for better decision-making given uncertainty.

The slide deck is here.

The essential tension in my presentation is the hypothesis that essentially “Long-term forecasting is impossible” vs. my argument that forecasting can only be improved if we “Make forecasters responsible for forecasts.”

Are EU Cohesion Policy funds well spent on roads?

| David Levinson

An interesting report by the European Court of Auditors (who knew Europe had a “court of auditors”?): Are EU Cohesion Policy funds well spent on roads?.

The EU spends a lot of money transferring resources between member states (cohesion funds). Much of that is spent on roads (a surprisingly large amount in Poland). This report compares proposals with results. Most roads under-performed traffic forecasts.

From the report:

Observations

  • The road projects audited partly achieved the intended results
  • … but their impact on economic development could not be assessed …
  • … and most of them delivered less than the planned return on investment …
  • … while clearly improving road safety and helping to save travelling time
  • Enhanced transport capacity could have been achieved at lower cost
  • The road projects mainly followed the most economical road alignment
  • Traffic forecasts were in the majority of cases not in line with actual road use
  • Costs per user vary significantly
  • Express roads were clearly less costly to build than motorways
  • The best possible price was not obtained for all audited projects
  • Significant cost differences for road accessories and bridge decks
  • Cost overruns of more than 20 % for 11 projects

(Via KA)

Transportation Benefits Too Little

| David Levinson

Highway_Chart_Graph_1
InterstateProductivity We are not building much new transportation in the US not just because the costs are too high, but because the benefits are too low.
My 2011 post Transportation Costs Too Much made the claim that new projects were too expensive, and listed a series of hypotheses as to why that might be the case. The whole list is collected here.

When we were much younger as a nation, say 1956, and growing fast, with relatively poor connectivity, you could do almost anything and it would have a benefit/cost ratio above 1. Very little of the interstate has been reversed. But the productivity of new investments has declined over time (as shown by Nadiri, M.I. and Mamuneas, T.P. (1996) Contribution of Highway Capital to Industry and National Productivity Growth Federal Highway Administration. Office of Policy Development.)

The first 3 data points on the graph are rates of return from their work, the last is an extrapolation. This is illustrative only,but someone should update their study.

Updated Jan 10, 2012. Matt Logan points me to a study that shows the authors updated their study in 2006 (Mamuneas, T and Nadiri, M.I. (2006) Production, Consumption and the Rates of Return to Highway Infrastructure Capital)

They report net rates of return are as follows (Table 5)

Period Net Rate of Return
1949-1950 : 0.554
1960-1969 : 0.480
1970-1979 : 0.298
1980-1989 : 0.212
1990-2000 : 0.136

My guess is that the 2000s are significantly lower.

A value of approximately zero returns in recent years is consistent with recent work by Noland and his colleagues. This is what you expect with life-cycle theory, and it applies well to existing modes.
The reason we don’t draw new lines on the map is that the net benefits are not perceived to outweigh the net costs. The costs have risen as land has gotten scarcer and for all the reasons linked above, and the benefits of additional lines drop. Here we explore the second point.

  1. Diminishing marginal returns to new roads due to diminishing distance reductions as the network is increasingly complete.
    TTE-F27-4
    This is a spatial argument, illustrated in the Third Figure (Source: Figure 27.4 from The Transportation Experience). Imagine you have a network with a 1 mile grid (typical for much of the US). With development of farms, you add roads in between, say at 1/2 mile spacing, this reduces travel costs some, as people don’t need to back-track as much, and this might be a significant share of the distance for short trips. At most, you are saving someone 1 mile (1/2 mile at the beginning of the trip, and 1/2 mile at the end of the trip). Now add additional links to diminish spacing to 1/4, This requires twice as many links, but only reduces travel costs by at most 1/4 mile at each end of the trip (1/2 mile total). New links do less and less to reduce distances. Distances, along with speed, determine travel time.
  2. Diminishing speed savings for new roads
    Recognition that transport links become congested, and thus slower over time due to:

      1. induced demand
      2. induced development
      3. induced driveways and interchanges, which increase friction on roadways and slow them down over time. While access management addresses this on arterials, very few interchanges are removed to speed freeways, they are almost irreversible

    Thus we have to discount the opening year forecast travel time benefits to account for the fact that the travel time savings of any expansion will in part (if not in whole) be eaten up by more travel. While this is not of itself a bad (travel is a measure of people doing something that they value), it is not perceived to be a good thing (because it creates congestion and pollution externalities which existing travelers bear).
    There is some compensation for induced demand and induced development, as more travelers may lead to more service (induced expansion of existing links and construction of new links), but this is a longer term process, and only works up to a point (and more easily with transit services than roads).

  3. Diminishing demands for new roads
    1. peak travel
    2. slowed population growth, particularly in un-roaded or under-roaded exurban areas.

    Implies the benefits from new construction are falling.

  4. Disillusionment with the quality of travel and transportation facilities
    The saying goes: Fast, Cheap, or Good, pick any two. It sometimes seems we have none of these. We know transport costs too much. We know transport takes too long to build. Surely if we are paying a premium, it should be of high quality. It seems not. Maybe the few facilities which are brand new, expensive, and took too long to build are, but the rest of it is not. Even then, I have doubts. Big Dig Ceiling Collapse is one of many notable failures, this one all the worse because the facility was new and expensive.An earlier post, It Just Makes You Feel Poor describes the poor quality environment around some local new LRT stations.

The arguments about benefits apply to mature systems in general, and are modally independent. If a new mode comes about that is better than existing modes for some market, it has lots of room to run while providing benefits in excess of costs. But if the new mode is inferior to existing modes, than it has little in the way of prospects.
What constitutes better or inferior is in the eyes of the customer. Certainly customers care about time and price, but they do also consider quality, and may be willing to sacrifice one for the other. But as experience with zeppelins and cruise ships (and conventional intercity rail in most of the US) show, high comfort at slow speed will not trump low comfort at high speed.
This also has implications for the Preservation vs. Expansion argument (Fix It First). If the old projects had high benefit/cost ratios, and without proper maintenance are at risk of disappearing (either failing catastrophically, or being closed to prevent such an end), it is incumbent to maintain them. In fact, given that land uses and resultant activity patterns that have evolved around transportation networks, the benefit/cost ratio of preserving those links is probably much higher than it was originally, and certainly higher than the B/C of new links.

Peak congestion

| David Levinson

Percentage of Congested Miles on Twin Cities Freeway System
Percentage of  Miles on Twin Cities Urban Freeway System Congested

MinnPost Twin Cities traffic congestion slowed a tiny bit in 2011. Though MinnPost doesn’t seem to believe it, they should be following the discussion of Peak Traffic. The measure is percentage of freeway miles flowing at 45MPH or less. The full report is here.

Accessibility Dynamics and Location Premia: Do Land Values Follow Accessibility Changes?

| David Levinson

Twin Cities Regional Land Use Map
Twin Cities Regional Land Use Map

Working paper:

The structure of transportation networks and the patterns of accessibility they give rise to are an important
determinant of land prices, and hence urban spatial structure. While there is ample evidence on the cross-sectional
relationship between location and land value (usually measured from the value of improved property), there is much
less evidence available on the changes in this relationship over time, especially where location is represented using
a disaggregate measure of urban accessibility. This paper provides evidence of this dynamic relationship using data
on home sales in the Minneapolis-St. Paul (MN) metropolitan area, coupled with disaggregate measures of urban
accessibility for multiple modes, for the period from 2000 to 2005. Our investigation seeks to track the effects of
marginal changes in accessibility over time, as opposed to static, cross-sectional relationships, by using an
unconventional approach in which the unit of observation is a “representative house” for each transportation
analysis zone in the region. This approach allows us to control for changes in structural attributes of houses over
time, while also isolating the effect of changes in accessibility levels. Results of this approach are compared to a
cross-sectional model using the same variables for a single year to illustrate important differences. These
differences are discussed in terms of their implications for practitioners and for further investigations of the
relationship between transportation, location and land value.

The abstract buries the lede. The average cross-section effect (that is the long – term accessibility) shows access is related to land value. The marginal effect (that is the change in land value due to the change in accessibility) is much, much weaker if not nonexistent (depending on the model). This suggests there are diminishing returns to access in mature networks.

The Demand Curve for Life

| David Levinson

At the 2009 International Transport Economics Conference Bruno De Borger, Erik Verhoef, and I were having dinner, Erik raised an interesting question about the use of statistical value of life in evaluation studies. Suppose there is a road improvement which will save 1 life per year, reducing the number of fatalities from 2 to 1 per year (out of 1000 people using the road). Assume all travelers are identical. What value of life should be used in the analysis?
Normally, we would do the equivalent of trying to compute for each traveler what is the willingness to pay for a 50% reduction in the chance of death by driving (from 2 in 1000 to 1 in 1000), and multiply that by the 1000 people whose chance of dying is reduced.
An alternative approach is to figure out the willingness to pay for the driver whose life is saved. So how much would you pay to avoid dying (with certainty) (i.e. what is your Willingness to Pay)? The answer to the first question is usually taken to be all of your resources (you would pay you everything so I won’t kill you).
Alternatively how much can I pay you to allow you to let me kill you (Willingness to Accept)? The answer to this second question is: I would have to pay you an infinite amount of money in order for you to let me kill you.
Both of those sums of money (everything or infinity) likely exceed the willingness to pay to reduce the likelihood of dying with some probability, multiplied by the number of people experiencing it.
In economic terms, we are comparing the area under the demand curve (the consumer’s surplus) for life (which has a value asymptotically approaching infinity as the amount of life approaches 0 (death approaches certainty) for a single individual, with the marginal change in the likelihood of survival multiplied by all individuals (i.e. the the quadrilateral between the y-axis of price and the same demand curve, between Pb and Pa) which describes the change in price for a change in survival).
On the one hand, using the marginal change for everyone rather than total change for the one person whose life is saved, we will give a lower value to safety improvements. On the other hand, the value of life to the individual himself is much higher than the value of life of that individual to society at large.

Discount rates on human lives

| David Levinson

From Tyler Cowan: Don’t apply positive discount rates to human lives:

“Ben Trachtenberg writes:

This Article presents two new arguments against “discounting” future human lives during cost-benefit analysis, arguing that even absent ethical objections to the disparate treatment of present and future humanity, the economic calculations of cost-benefit analysis itself – if properly calculated – counsel against discounting lives at anything close to current rates. In other words, even if society sets aside all concerns with the discounting of future generations in principle, current discounting of future human lives cannot be justified even on the discounters’ own terms. First, because cost-benefit analysis has thus far ignored evidence of rising health care expenditures, it underestimates the “willingness to pay” for health and safety that future citizens will likely exhibit, thereby undervaluing their lives. Second, cost-benefit analysis ignores the trend of improved material conditions in developed countries. As time advances, residents of rich countries tend to live better and spend more, meaning that a strict economic monetization of future persons values the lives of our expected descendents above those of present citizens. These two factors justify “inflation” of future lives that would offset, perhaps completely, the discount rate used for human life. Until regulators correct their method of discounting the benefits of saving human lives in the future, the United States will continue to suffer the fatal costs of underregulation, and agencies will remain in violation of legal requirements to maximize net benefits.”

I think in practice we have to discount future lives, if the discount rate were zero, then we should do nothing for the present as the infinite future would dominate any calculation. I am dubious health inflation will continue unabated. The discussion on the article is interesting and worth reading.