Evaluating innovation has always been a difficult job for innovators, investors, facilitators, and managers. With increased pace of developing ideas, it becomes critical to evaluate innovations effectively and quickly. Before I begin developing an innovation evaluation framework, I will define what I think is an innovation and draw some characteristics first. Innovations are
- purposeful action and aligns with some personal or organizational vision
- developing ideas that are perceived as new and valuable
- impactful at a scale, may include financial, social, environmental, life impact
- investments that may lead to disproportionate returns
Innovations are evaluated for various purposes like
- Qualifying for investment/grant/other resources
- Quantifying impact of the innovation
- Modifying the development process for a set of ideas
“While we recognize that the American economy is changing in fundamental ways- and that most of this change related directly to innovation-our understanding remains incomplete…centrality of the need to advance innovation measurement cannot be understated” – Carl Schramm in the committee report to Secretary of Commerce 2008
At level 0, I believe the following facets have to be considered
Evaluation includes the following phases/activities around data and reporting
1. Data Collection, depending on the kind of evaluation it may include quantitative and qualitative information. Typically if data is collected from primary sources aka the field through surveys, direct interview, or secondary sources like agencies. Every collection effort should include independent variables, and dependent variables. It is useful to segregate between input variables, and outcome variables. Units of measure for all variables have to be standardized or they should be convertible. In case of comparison between different variables, you might want to consider some normalization process. Data quality standards are to be set prior to beginning the data collection and for any further analysis data has to be of some agreed minimum quality.
2. Analysis and Data representation, depending on the kind of data collected analysis methods will vary. For example representations for financials will be in spread sheets and charts, social data will be on maps, stories will be as fitness landscapes. Typically here is where any hypothesis is provided, and tested, future state predictions like forecasts based on models are put forth. Comparison with history or benchmarks will happen at this stage as well.
3. Results of evaluation, should be an action or recommendation. In most cases evaluation leads to decisions by parties other than the evaluator. If this party is not identified prior to evaluation process, the effort is most likely to go waste.
“What are we really going to do with this? Why are we doing it? What purpose is it going to serve? How are we going to use this information?” This typically gets answered casually: “We are going to use the evaluation to improve the program” — without asking the more detailed questions: “What do we mean by improve the program? What aspects of the program are we trying to improve?” So a focus develops, driven by use.” – Michael Quinn Patton
Once you have decided which facet of innovation you are trying to evaluate, we can now adopt from many of available methods for doing the actual evaluation. I will try and list some of them below, with links to external resources that I have found useful.
Impact: EPSIS provides a detailed framework and clearly distinguishes between output, impact and performance and provides a set of indicators that can be used for direct measurements or indirect impact measurements. Social Impact evaluation on philanthropy from Stanford is a good place to start.
Investment: Investments related evaluation includes both input costs and outcome returns to compare innovations. For example we use something called as the t-shirt sizing for ideas at first level, that will give a scale estimate of cost. Return on Investment as a ratio is a good measure but the underlying assumptions for predicting returns has to clear, and the other common error is around data quality when predicting returns.
I personally use value investing check for fundamentals when getting into stocks, and the factors that are checked are around stability, margin of safety, and historical dividends. Investment evaluation should be reduce the impact of any failure and enhance experiment design. In many cases ‘closed world’ resources (freely available locally, and has potential use) play a significant role in reducing investment.
Diffusion: Interdisciplinary classic work in this field Diffusion of Innovations by E Rogers lists different ways and covers a broad range of research that has already happened in diffusion. I like the stages around innovation diffusion as awareness, persuasion, decision and implementation. Data collected should focus on units of adoption (individual, community, user groups, etc), rates of adoption over time, and other social aspects of the adoption.
Model: In this facet of evaluation we only focus on what model of development was used for generating and developing the innovation, and should cover business model elements and how each of the elements are being looked at. Data collection would typically include metrics (see size, time, interface and costs worksheet below from NUS below) on needs, stages of development, partner structure, productivity, etc. For example Villgro, kickstarter, and Google ventures all operate in distinct models for developing innovations.
Development: Entire field of Developmental Evaluation is dedicated to evaluating during innovation and applicable for complex, highly social, non-linear situations. McConnel foundation’s practitioner guide is probably the best you can get for free.
I will cover a few methods for selecting innovation like PUGH matrix, decision trees, possibly in another post. This will be my last post for the year 2012, and I hope to build on the momentum covering deeper and meaningful innovation topics in 2013. Happy new year…
Although the policy has been there for quite sometime, I had to do my analysis before predicting any outcomes. I will focus on data points and the recent press release from the government, noting catches and feasibility of the stores across India. Background material that I have used is all linked here starting from the original policy circular, addendum press note, census data, and other references.
First off this policy is only an enabling policy and states are still free to take their own discretion. Considering the negligible amount of collaboration that had gone behind this policy formulation, other than the favored states mentioned in the policy, there will not be many other takers.
As per census 2011 data on urban population metro area, the policy allows the following 49 cities eligible
|City||Favorable State/Territory||Population (2011)|
|7||Srinagar||Jammu and Kashmir||1,192,792|
|City||Unfavorable State/Territory||Population (2011)|
Number of favorable states with more than 3 cities eligible is only 3 namely AP, Maharashtra and Rajasthan. I doubt if the likes of Wal-mart would commit investment in a state having just 1 city above the population threshold. Of course states can choose if they do not have cities more than 10 lakh and allow investments but it clearly does not favor the investor to make any returns from smaller cities.28/49 of those cities are in unfavorable states that are not supporting UPA directly and 21 are in UPA support states.
To open stores there are other considerations including land, and sourcing assuming demand exists.
Average time needed for land acquisition in Maharashtra (most favored state above) for large-scale development has been around 2 years, and that too after approvals from state. Many reasons exist including the political nature of the transactions itself, as has been seen in so many of the SEZ transactions, Singur, POSCO, etc. With the Land Acquisition Bill yet to be tabled there might be a window of opportunity as the bill renders most land acquisitions unviable. 50% of the minimum USD 100 million is to be invested in back-end development excluding land costs and rentals. That leaves a maximum of USD 50 million for land and development. My rough calculation assuming a nominal market rate of 4000 per sq ft leaves with around 600, 000 sq ft possibility. Obviously no retailer will invest in such a large format and a reasonable strategy could be to split and invest in many stores and distribution network.
Back end infrastructure norms is possibly the most useful part of the entire policy. Food loss is ranging from 25-35% in India before it can be consumed (for horticultural produce it is even higher), and don’t get me started on the amount of waste that happens as part of the government controlled distributions. Control on this and possibly selling these warehousing, transportation services to other retailers will definitely cut inflation and help the overall economy as efficiency improves.
Supply side issues coming from the sourcing norms listed in the policy are as follows 1. there may not be enough number of suppliers in favored states like Rajasthan 2. quality of produce may not be on par with retailer standards 3. amount of produce cannot meet regulatory norm of 30% from small industries. Hence transport costs will add to the retailer burden here especially when subsidies are reducing on fuels, this is not a major issue as most retailers always run large and efficient distribution and transportation network.
My feeling is, extremes of this policy being completely anti-farmer or anti-common man to being completely pro-investor or pro-consumer are both incorrect. Projections on outcomes including job creation, creating efficient back-end, and encouraging ‘small industries’ are all far-fetched considering favorability for investor in only few pockets. Enabling conditions for quality investments (even for domestic institutions) in retail is still a long way to go including cost of land, capital, variety of demand etc. May be with some clever network + a lot of dealing we may see a few stores around Maharashtra (favorable demand + party in power), and MP (favorable sourcing) in pockets. That said FCPA norms could be a bigger show stopper than anything else at least for US listed entities trying to invest in India. Kiranas will still thrive and their inefficiencies are solvable at their scale, without any stimulus from government.
Michael Mitchell made a presentation on Travel Industry trends recently. Key take-away from the session personally was on the fitment between cultures and systems and on RoI. Having been part of many M&As, made hard choices on systems, benchmarking systems, deciding where to invest for the travel industry, Michael is uniquely qualified and his perspectives are unique as well.
He started from his experience with a couple of mergers and how the choice of systems actually is not of systems itself but of culture. In any M&A does not necessarily mean the best systems will prevail but systems that support the culture that is conducive for business will be picked and sustained. Making the wrong choice means erosion of brand value (in industries like travel the brand development takes as much as 20 years) and service to customers which are actually closer to culture than IT systems. On a follow up question from Jas on how will a conglomerate like SITA develop and deploy across cultures, Michael reinforced the point that it was still a matter of choice on culture that is dominant. So as always systems fit culture and not the other way around.
There was this trend that IT leadership in the industry showing positive outlook on investing in initiatives that had "shorter RoI cycles", commenting on it he said it was not the right thing to do. My question to him was when does RoI cease to be a measure of impact, and what are the alternatives. His response was if an investment was being made to reduce costs (typically cost of transactions around a core service delivered) it is relevant, but the equation becomes murky when revenue is involved. Take advertising for example, it is hard to quantify revenue that came specifically from a marketing initiative and applying RoI is erroneous here. Incremental revenue analysis was suggested as an alternative, but in my opinion that will still have the issue of attribution of credit.