Evaluation of Protection and Restoration Effectiveness

In this section we present a method for evaluating the effectiveness of the various protection and restoration strategies identified and described in the preceding chapters. This evaluation method is designed to be used to make recommendations and conclusions for implementing the most ecologically and fiscally effective strategies for restoring Puget Sound ecosystem function.

The goal of this suggested evaluation process is to evaluate how likely a particular strategy, or group of strategies, will achieve its stated goal; namely, the restoration or protection of one or more desirable attributes of Puget Sound. In short, how effective is the strategy in question?

The results of this evaluation should provide decision-makers with a clear indication of the relative effectiveness of different strategies; including those strategies which have been particularly effective and those where further improvement is needed. The evaluation will assist managers in deciding which strategies are thought to be most effective, their relative costs, extent or reliability of application, protection and restoration research needs, and guidance for monitoring.

The objectives of the proposed evaluation process are to:

1. Develop an ecosystem and goal-based framework for classifying strategies to protect and restore Puget Sound (the framework should be consistent with the major topics identified in our outline);

2. Compile a list of strategies, organized by category (i.e., outline chapter) based on a review of the peer-reviewed literature. (Due to resource constraints, the compilation will not be exhaustive; rather, it is meant to enable us to broadly characterize strategies so that we can come up with a satisfactory method of organizing and evaluating them using the proposed assessment methodology.);

3. Identify performance criteria and develop a methodology (e.g., scoring procedures) for evaluating the effectiveness of individual or groups (combinations) of strategies to protect and restore Puget Sound. The approach should be rational, adaptable, easily comprehended, and capable of being applied at different scales (e.g., it can be used by PSP and local governments to assess the effectiveness of their restoration and protection strategies);

4. Apply the criteria and scoring procedures to obtain a qualitative ranking for each strategy or group of strategies;

5. Provide a foundation for effectiveness monitoring and adaptive management; the framework should be based on recommended indicators and existing governance systems (e.g., State, PSP, WRIAs) (This is beyond the scope of our current assignment, but the method and results of our evaluation should inform and integrate with future monitoring and management.); and

6. Provide practical guidance so that others can evaluate the effectiveness of restoration and protection strategies without direct assistance from us. This, too, is outside our current scope of work. However, the evaluation methodology is intended to be reapplied iteratively in the future as new goals are formulated and new information is developed.

The categorization of strategies and development of appropriate assessment criteria is proposed to proceed in two steps. Step 1 is the articulation of the basic goals of restoration and protection and Step 2 is the development of performance criteria that help us define and evaluate effectiveness, the main focus of the process described here.

We propose that the evaluation be conducted relative to three broad sets of pre-stated goals for protection and restoration:

• Perceived technical performance and scientific soundness
• As compared to the PSP Action Agenda Priorities
• Relative to the PSP Action Agenda Outcomes

Because the interaction of protection and restoration strategies with the actual implementation, as reflected in habitat outcomes, are extremely complex, dynamic, and at variable scales, a method is needed to account for as many features as possible in the evaluation. We therefore propose that each strategy that is evaluated be first categorized according to area, method, and scale of application, as follows:

• Watersheds and tributaries
• Estuarine and marine
• Fish and wildlife populations
• Overarching or general

• Preservation
• Protective retrofit actions
• Protective new development and redevelopment actions
• Physical habitat restoration
• Policy changes
• Public Education

• Overarching - Applies broadly to all areas (e.g., a broad policy change)
• Regional (e.g., several watersheds or large portions of Puget Sound)
• Local (e.g., one watershed or tributary, or Puget Sound bay)
• Site or population-specific

The results of such an evaluation process would include tabular matrices that list the known protection and restoration strategies and rate them under each of the preceding categories. In this way, managers will be better informed about which strategies are thought to be most effective, their breadth or reliability of application, and protection and restoration research needs.

The proposed assessment approach for each strategy will consist of a set of criteria which, taken together, provide the basis for an assessment of the particular strategy under consideration. To do this, a summary assessment would be conducted and each strategy rated for its status according to the following rating criteria (expressed positively so that all metrics will have the same sign or direction):

1. Perceived relative effectiveness
2. Level of scientific basis (alternative: research needs)
3. Certainty of success (alternative: risk)
4. Confidence in outcomes (alternative: uncertainty)
5. Low need for monitoring
6. Degree to which currently monitored
7. Low total cost
8. Benefits in relation to costs
9. Consistent with existing processes
10. Extent of existing application, i.e., level of participation, commitment, ownership, compliance, etc. and
11. Application to multiple threats
12. Capacity – technical and financial resources, etc.
13. Informs monitoring, learning, and adaptive management

Some of these criteria overlap with others; they should be refined to a handful of non-redundant, easily intuited criteria that are applicable to all strategies. The final set of criteria should allow us to objectively assess the performance and outcomes of the strategies to which they relate. And finally, they should be comprehensible to other, less technically oriented individuals and stakeholders.

The evaluation system requires scoring metrics and a process by which individual evaluators are able to review available information and indicate the extent to which each criterion is (or is likely to be) met. It would help if the criteria were framed as a series of questions that ensured that all aspects and dimensions of the strategy are considered.

We propose the following scoring metrics:

4 = the criterion is fully met

3 = the criterion is mostly met, but further improvements can be achieved

2 = the criterion has only partially been met, there is potential for further improvements

1 = the criterion has been barely met; but there is promise for the future

0 = the criterion has not been met; further improvements are unlikely

In designing the final evaluation, we propose the strategies would be listed in the matrices presented in tabular form as illustrated below, which combine all the features described above. In these tables, we rate each strategy in terms of its subjectively determined effectiveness.

The final evaluation of a given strategy combines consideration for the type and extent of the strategy with its rated performance as a scientifically substantiated strategy to gather with its perceived satisfaction of Action Agenda outcomes and priorities. We also recognize that the system posed here has certain drawbacks. For example, the rating score given to any one strategy for a certain category is highly dependent on the setting where the rated strategy would be applied and ecological and economic details of the particular application. Therefore it may be necessary to refine or revise the rating system to include additional considerations for how, where and by whom the application would be implemented. For now, however, this suggested process may be useful as a starting point for evaluation of protection and restoration strategies.

There would also be a need for periodically updating this information so it can be used to inform management decisions over time; i.e., maximizing the effectiveness of an integrated research, monitoring, and adaptive management program. The approach described above provides the Partnership with a vehicle for future evaluation and management of protection and restoration strategies into the future since it can be continually updated and revised.