Five Reasons the AUVSI Got Its Drone Market Forecast Wrong

Since its publication in early 2013, AUVSI’s The Economic Impact of Unmanned Aircraft Systems Integration in the United States has become the gold standard forecast for the commercial drone market, garnering media attention typically reserved for celebrity weddings and babies born to royalty.  Its primary forecast is that the UAS market will reach a whopping $1.14 billion [1] in the first year after the FAA issues favorable regulations and that the precision agriculture market will “dwarf all others.”
The accuracy of these predictions is enormously important. A lot of people – tens of thousands, if not more – have been relying upon them for big decisions like, “Should I leave my job to start a drone company?” or “Which market should my company pursue?”  But commercial drones are not just cocktail party conversation–they are increasingly driving the flow of capital and labor, and impacting many lives in the process.
Inquiring Minds Want To Know
Recently, however, a growing chorus of industry observers has started to ask questions about the reliability of AUVSI’s findings. This post at Drone Analyst is a good example. These individuals, many of whom are among the true pioneers in commercial UAS usage, can best be characterized as enthusiastic but pragmatic UAS evangelists who don’t want to see unwarranted hyperbole lead to unmet expectations.  Many realize that initially overhyped industries never recover because customers, investors, and employees who were burned in the initial wave of unmet expectations are difficult—if not impossible—to ever win back.  They are passionately committed to the industry’s success and believe that rational expectations are a key part of it.
With no axe to grind or agenda to advance, I partnered with Colin Snow @droneanalyst to explore whether the skeptics and pragmatists were on to some something.  We felt our combined backgrounds in market intelligence and tech market strategy would give us a reasonable set of expertise to draw upon and would help others form a more balanced opinion of AUVSI’s forecasts.  So over the past several weeks, we’ve been carefully reviewing AUVSI’s report, as follows:

• Compared their research methodologies to what we believe to be best practices in market research based upon our own experience.
• Conducted an in-depth interview with the researchers themselves, so that we could directly ask them questions about their methods and results that were not made clear in the report.
• Initiated a follow-up discussion with AUVSI leadership to understand their perspective on the report and its origins.
• Performed intensive primary research with about 20 carefully selected professionals in the field of precision agriculture to understand their UAS adoption plans, since the report’s findings are almost entirely based upon rapid adoption by American farmers.

We then synthesized our findings into the following five conclusions about the report and its reliability.

1. Research Can Be Objective, But Don’t Assume It Is

First and foremost, every reader of AUVSI’s report needs to understand that it is not an objective piece of research.  The report was commissioned not to paint an accurate picture of how the commercial UAS market is expected to evolve, but to give the 50 states and their elected officials the data they needed to:

• lobby for funding during the now completed FAA-sponsored competition for UAS test sites, and
• push the FAA to move more quickly on the integration of UASs into the national airspace.

These are certainly worthwhile goals, and AUVSI should be commended for pursuing them.  But as a direct result, the implicit (if not explicit) mission for the two researchers who did the work was to come up with the biggest numbers – the largest market, fastest growth rates, and biggest costs of delaying integration – that they could.  An objective attempt to size, segment, and forecast the commercial UAS market (all of which the report appears to be), is something it never actually was, and we believe it’s critical that all participants in the UAS industry know this and avoid making decisions based upon it.

2. Methodology – Boring But Oh So Important

A biased agenda is only one part of the story regarding the reliability of AUVSI’s findings.  An equally important part is the quality and reliability of the research methods.  Generally speaking, strong research methods yield highly defensible results.  While presented somewhat differently in the report, the methodology used by the researchers can be summed up as:

• Studying UAS adoption in Japan
• Adjusting the Japanese experience for the US market
• Asking experts how big they think the market is / will be
• Applying research on new technology adoption to the US UAS market

As experienced researchers, it sounded pretty good to us at first.  But, unfortunately, it did not hold up very well to careful scrutiny.

3. Japan – When the Best Available Proxy Just Isn’t

We like the idea of searching for analogous markets and scenarios that can serve as the basis for forecasting the US market.  The question is: Is Japan an analogous market for the US? We believe that the US and Japan are so different, and the magnitude of the required extrapolations so enormous, that the resulting data is not useful.  Most in the industry already know that Japan’s UAS market remains dominated by one product, the Yamaha RMAX (77% market share in Japan), which is used to spray a large percentage of the country’s rice fields.  These fields tend to be small (less than five acres), are often in densely populated areas, and are located on steep hard-to-reach hillsides.  In contrast, rice represents a tiny percentage of US agricultural output.  Our farms are comparatively huge (very often running well into the thousands of acres).  No single product, much less a relatively large, unmanned helicopter from Yamaha is likely to dominate the American market.  And remote sensing, not pesticide application, is almost certain to be the dominant use of UAS for the major US crops of corn, wheat, and soy.
While we understand that Japan has been the most aggressive adopter of commercial UAS technology as a result of its rice industry, and we appreciate the resulting temptation to use Japan as a proxy for the United States, we see such a large disparity between the agricultural economies of the two countries that we find it impossible to draw any parallels that inform how the UAS market in the US will evolve.  And while no other country serves as a better proxy than Japan, the absence of a better alternative cannot justify the use of a bad one.

4. Expert Opinions or Really Just Guesses?

Another method used by the researchers is referred to as “survey results.”  In short, the researchers conducted 30 telephone interviews with industry experts and asked many questions, including those regarding two critical matters: the size of the commercial UAS market, and the relative size of key market segments.  The responses were then used to develop “reasonable estimates.”  On the surface, the approach of asking experts for their opinions seems sensible whenever you’re conducting research.  However, many of the experts that were consulted were hand-picked by AUVSI, which immediately introduces the possibility (likelihood?) of bias given its agenda.
Perhaps more important, not every question is one that experts can necessarily answer well.  Certainly UAS industry experts would generally be well prepared to share their opinion on whether fixed wing or rotor aircraft will be more useful for particular applications, or what regulations make the most sense for the small UAS market.  But the idea that you can ask experts for opinions about the size of a market and obtain meaningful results is, we believe, inherently flawed.  Unless these experts were professionals focused on sizing, segmenting, and forecasting the commercial UAS market (and nothing close to 30 such professionals exist), the opinions voiced by the “experts” are nothing more than guesses, akin to asking 30 people how many clouds there are in the sky and expecting to get the right answer.  Our experience in sizing markets, and in working with many experts across a wide variety of markets over many years, gives us considerable confidence in stating that very few people have good insights into how big a market is today, much less how big it will be years from now, even if they work directly in it.  The lack of insight is only compounded for complex, nascent markets like the one for commercial UAS.

5. A Brief Literature Search Isn’t Really a Research Method

The final method used by the researchers was a “brief search” of “literature…on rates of adoption of new technologies.”  The authors explicitly state that they could have gone deeper in investigating how this research might apply to UASs, and that a follow-up study on this subject is recommended.  That they simultaneously cite the use of the literature as one of their four methodologies, yet characterize their search of the literature as “brief” and recommend a follow-up study raises serious questions.  From our perspective, the brief use of literature on technology adoption trends is far from a true research method. It’s more akin to subject matter expertise and qualitative insight that professional researchers might use to inform or validate a forecast they developed with rigorous quantitative techniques.  How it was actually used and what value it added to the research is unclear, other than allowing the authors to make the statement that because UAS are already being used “….we reject the notion that these products will not be adopted,” a statement that even a layperson with little or no knowledge of UAS could likely have made.
In sum, we see a methodology that erroneously uses Japan as an analog; uses experts for answers that are really just guesses; and relies upon a loose, limited, and ambiguous application of prior research on new technology adoption to validate the statement that UAS will, in fact, be used in America.  As much as we want to support AUVSI, the authors, their methodology, and the research results, we simply cannot.
Sometimes You Get Lucky
As a final point, we do need to acknowledge (and quickly refute) the possibility that despite the flawed methodology, the research findings are reasonable, by pure chance.  Perhaps, as the authors assert, the US commercial UAS market actually will be at least $1.15 billion in the first year after rules are approved.  And perhaps 80% of this, or roughly $900 million will be driven by the precision agriculture market.  But at the risk of disappointing the reader, and with a view toward keeping this post a reasonable length, suffice it to say that while we have high expectations for the US commercial drone market, we do not see a billion dollar market in year one.
We base our position on the deep understanding we have developed of the precision agriculture market, which is at the heart of AUVSI’s forecast.  Indeed, the many in-depth interviews we’ve conducted with farmers, precision agriculture vendors, crop scientists, crop scouts, agriculture equipment dealers, input vendors, academic researchers, manned aircraft operators, satellite imaging providers, UAS-service providers, and many others indicate a building interest in the use of remote sensing in general, and in UASs in particular, but do not support the notion that a mad-dash by farmers and their consultants to use UASs is underway or right around the bend.  And after looking at many other vertical and application markets for UAS, we do not see any – not public safety, inspection, photography, mapping or a variety of other possibilities – that can close the resulting multi-hundred million dollar gap in the AUVSI forecast created by the much slower adoption we see in precision agriculture.

Acknowledging the Effort

Of course, it’s easy to critique the work of others, and hard to do the work yourself.  In defense of the report’s authors, we need to acknowledge that they did a lot with a little.  They had a budget to work within that was much smaller than is typical for an assignment of this complexity, and they invested much more time and effort than the budget allowed.  Like virtually almost everyone else in the brand-new (some would say still non-existent) commercial UAS industry, they had limited prior exposure to the commercial UAS market, making their learning curve steep.  And they had complex agendas to meet in order to satisfy their client, AUVSI, and its many stakeholders.  In light of the foregoing, there is much for which they should be commended.  But creating a forecast for the commercial UAS industry that participants can rely upon for critical decisions is not one their accomplishments.  Indeed, it’s not what they set out to do in the first place, so they can’t really be faulted for not accomplishing it.

Looking Forward
As we look to the future of the commercial UAS market in America, we believe the need for reliable data and insights is more acute than ever.  Critical decisions about products, markets, channels, and operational best practices are being made daily, even as we write. UAS technology vendors, service providers, and end-users are relying on intuition, gut feel, or data that is very likely misleading.  Some decisions will still turn out to be right, but many others will unnecessarily result in big missed opportunities, significant wasted time and resources, disappointed customers, angry investors, disgruntled employees, and many other negative outcomes that certainly could have been avoided.
—–
[1] AUVSI’s forecast implies a UAS market that is likely significantly greater than the $1.14 billion in 2015 shown in the report, because it does not address the large part of the market that is currently being satisfied by offshore vendors.  The $1.14 billion represents only product supplied by US manufacturers of UAS.  It may also fail to include industry profits, though further investigation would be required to confirm this.

Sourse: http://droneanalyst.com