Urban planning community

  • Automated Analysis: How Accurate are the Results?

    By Michael Stumpf, AICP, CEcD

    Two homes of a similar size and age, on equally-sized lots in the same neighborhood, should have more or less equal value. On the face of it, this seems like a reliable statement and is the basis for several home value websites such as Realtor.com or Zillow.com. These services rely on such basic assumptions in developing algorithms to calculate expected home values based only on data from sources such as assessment roles (providing information such as lot and building size) and recent home sales in the vicinity. But what about other factors not apparent in the data? What if one home is a beautifully restored 1900’s Craftsman bungalow in a quiet neighborhood across from a park, while the other is a poorly maintained and nondescript ranch house on a four-lane arterial surrounded by industry? While obvious to us, the computer cannot “see” these differences and will suggest similar values for the homes, significantly over- or under-estimating their value.

    A recent article in the Wall Street Journal (The Fuzzy Math of Home Values, November 14, 2011) tackled this issue, noting that the home values these sites determine could vary significantly from true values, and could also swing widely from month to month depending on factors such as new sales. Assumptions on which the models rely are only a part of the problem. Data may also be unreliable, particularly at the local level where there may be few recent sales, or where some variables may not be reported. Unfortunately, the typical person visiting one of these websites does not understand the methodology and its inherent flaws. The result is that pricing decisions for both buyers and sellers relying on these estimates are often out of line with the market. A clear conclusion that may be drawn from this is that when we make decisions regarding home values, we need to hire realtors and appraisers with local market expertise.

    But what about retail market analysis? Several companies have been providing data-driven market reports for many years. At a cost of only a couple hundred dollars, developers, planners, economic developers and others can purchase a report that claims to offer analysis of local demographics and growth, market potential, estimated sales within the market, and demand for retail by sector. As these reports became widely available, every planning firm suddenly thought itself capable of providing “market analysis” services by simply repackaging data in the report and spitting it back to the client – who may not always recognize how little they have received for their fees.

    These market reports have been around for so long that they are seldom questioned. But like the home value websites, they should be. While each provider has their methodology and may use differing raw data sources, the analysis ultimately rests on assumptions that are untested against the realities of the local marketplace. Consider a few examples:

    ● Demographics. Are population estimates and projections accurate? This is an especially important question in areas experiencing changes to long-term trends, such as areas of the south and west where previous rapid growth came to a halt and has even reversed itself due to the recession and housing crisis. Consider another area. Western North Dakota has been losing population for decades, and most projections will show this trend continuing. But oil drilling in the region has created a tremendous economic boom and has drawn an estimated 32,000 new residents to the region. These oilfield workers also earn hefty wages that have significantly lifted area incomes. These sudden changes – occurring in only the last three or four years – will not be reflected in many of the automated market analyses.

    ● Population Anomalies. Prisons, military installations, and colleges all generate populations with atypical income and expenditure patterns. Still, they are counted in the trade area population and treated in the same manner with regard to calculations of market potential.
    ○ Prisoners, of course, earn very little income and do not contribute to a trade area’s market potential or expenditures. Counted with the rest of the trade area residents, they tend to reflect demographics not in the resident population and will also skew area income downward.
    ○ Military installations can have a range of effects. Rotations may increase or decrease the population available to make purchases, and household expenditures spike when people move into a new home or apartment. Personnel living on base will have expenditure patterns differing from those off-post. Also on base, a Post Exchange can be significant competition capturing military personnel expenditures.
    ○ Colleges are well known for altering the composition of a market and business districts, and not only for a proliferation of drinking establishments. Again, students living on campus have expenditure patterns differing from those living off campus, and both are significantly different than the base population. Timing of purchases may also be important if students are not present during the important Christmas season.

    ● Market Demand. Computer-generated market analyses will only estimate demand from the resident market. In many communities, tourism provides a substantial boost that should not be ignored, yet it receives no notice in automated analyses.

    ● Estimated Sales. Average sales by store type and average sales per employee are the most common statistics used by most market data services in their estimation of sales. The basic idea is that if a store has a given number of employees, and you multiply that by the average sales per employee in the respective retail sector, the result will provide a close approximation of actual sales at the establishment. Again, the opportunities for error are substantial:
    ○ Walmart and Kmart are both discount stores (not including supercenters), yet Walmart averages about $45 million per store while Kmart barely manages sales of $10 million.
    ○ Operating practices also make a difference. Costco tends to have a greater share of full-time employees than Sam’s Club, which favors part-time staff. A 2004 Business Week study determined sales of $795 per Costco employee, versus only $516 per employee at Sam’s Club.

    These two examples demonstrate the fallacies associated with applying averages to a diverse population of stores, but yet another error factor is introduced in determining the number of stores in a market. Market data services use directories to identify the stores located in a trade area. These often miss a great number of smaller establishments, and may take a year or two to include information about new stores openings or closings. In some cases, stores that have changed names or relocated may be counted twice.

    If the data from online market analyses is so poor, then why is it so widely used? Part of the answer is simply cost. Dedicating staff or hiring a qualified consultant can cost thousands. A “canned” report will cost $100-$200. The bigger part, though, is that most people do not know how poorly these analyses reflect actual market conditions. Most users do not have the market or industry knowledge to question numbers they see in computer-generated reports, and do not have access to the finer-detailed estimates (such as sales estimates for a particular store) that might make them question the results. Unfortunately, this lack of understanding extends to many consultants as well.

    So how can you be sure to get a true picture of the market? If you do not have the technical background to perform the analysis in-house, the only answer is to hire a qualified consultant. And when you do, ask them where they get their data.

    Michael Stumpf is a consultant and principal with Place Dynamics LLC, specializing in economic development, market research, and community planning. http://www.placedynamics.com

    Comments 3 Comments
    1. ColoGI's avatar
      ColoGI -
      Quote Originally posted by Dan View post
      It's all about the premise: if your premise is "I'll only use this tool" then the tool is not good enough. If your premise is "its a good place to start" then you are all set.
    1. azmodela's avatar
      azmodela -
      "Two homes of a similar size and age, on equally-sized lots in the same neighborhood, should have more or less equal value. On the face of it, this seems like a reliable statement and is the basis for several home value websites such as Realtor.com or Zillow.com. These services rely on such basic assumptions in developing algorithms to calculate expected home values based only on data from sources such as assessment roles (providing information such as lot and building size) and recent home sales in the vicinity. But what about other factors not apparent in the data? What if one home is a beautifully restored 1900’s Craftsman bungalow in a quiet neighborhood across from a park, while the other is a poorly maintained and nondescript ranch house on a four-lane arterial surrounded by industry? While obvious to us, the computer cannot “see” these differences and will suggest similar values for the homes, significantly over- or under-estimating their value. " Doesn't sound like they're in the same neighborhood. But, this article is written to justify the services you offer and Zillow or Realtor.com impede on the services that the average home owner might peruse. I would expect that Zillow or Realtor.com would pick up on the two completely different examples that you provided and would generate a good starting point for the estimation of value. Remember, starting points can go up or down.
    1. Cardinal's avatar
      Cardinal -
      Quote Originally posted by azmodela View post
      "But, this article is written to justify the services you offer and Zillow or Realtor.com impede on the services that the average home owner might peruse. I would expect that Zillow or Realtor.com would pick up on the two completely different examples that you provided and would generate a good starting point for the estimation of value. Remember, starting points can go up or down.
      True, market analysis is one of the services I provide. Still, the article is meant to caution people about relying too heavily on the kinds of numbers generated by sites like Zillow and from services like ESRI or Claritas. The opportunity for error is significant, especially as you move from trying to assess a metropolitan area to a neighborhood to a site. Planners often fail to question the numbers they receive, whether they download a report themselves or if it comes from their consultant. But the decisions they make and strategies they recommend are often influenced by those same reports. Shouldn't people seek more reliable information?