In this chapter we described two method engineering cases. The cases were carried out as action research studies. The action research method offers possibilities for learning in three areas (Checkland 1991): the area of an application, the methodology applied, and the particular ideas promoted. In our studies the application deals with developing local methods for specific ISD environments. In Checkland’s (1991) terminology, the methodology denotes the general principles applied in inspecting an application area. Hence, for us this means method engineering along with related methods and tools such as metamodeling and metaCASE tools. The ideas promoted are the metamodeling language constructs and the a posteriori view of ME.

The following subsections discuss the evaluation part of action research. First, we describe in Section 6.4.1 general findings about local method development. Since our action research studies were not focused on all aspects of local method development (like costs or management principles) we shall only inspect the development of tool-supported methods. Second, we shall inspect differences between the proposed ideal ME principles and the cases (Section 6.4.2). Finally, we describe in Section 6.4.3 findings related to the use of incremental ME principles. Because we also participated in the ISD process, some findings could be presented about how to develop inter-organizational ISs in wholesale and improve the delivery systems of a cardboard mill. Our studies, however, were designed to operate at the ME level, not at the ISD level (cf. Section 3.3.1). Some general solutions for ISD, however, were already discussed as part of the cases.

The studies show that organizations develop their own methods. In the wholesale case, the ME effort was targeted to support a specific BPR project. The mill case included some features characteristic of developing a more universally applicable method: the consultants wanted to develop a method which would be independent of object systems, and appropriate for analyzing a variety of workflows related to logistic ISs. Because the method developed was intended to be used in all projects the consulting company engaged in, the second case followed an organization based ME.

Local methods were developed because of the limitations found in the existing methods used, inadequate tool support, and the lack of knowledge about other methods. In the wholesale case, the need to distinguish between organizations involved in the delivery chain and to characterize inter-organizational processes led to the establishment of the ME project. In the mill case, the need to automate analyses of workflows necessitated the development of specific tool functionality and as a by-product allowed the development of a propriety modeling method. Hence, modeling capabilities were addressed primarily in the wholesale case, and automated analysis capabilities (i.e. tool support) were emphasized in the mill case. It should be noted that in both cases the existence of a specific ISD project clearly influenced how the method development and evaluation effort was carried out. For example, if the wholesale company had not been in the middle of a major business process re-engineering effort, a local method would not have been developed.

The methods developed in the two cases had similarities: they addressed the modeling of business processes, described material objects or flows, defined organizational units, and characterized these modeling elements with some similar properties (e.g. volume, capacity). The main differences between the methods was the granularity of analysis. The wholesale company tried to understand its order entry and purchasing processes better in relation to its business environment, to support the move to a two level hierarchy. The mill case focused on individual workers and aimed to analyze the structures of tasks. More detailed differences between the methods can be identified by comparing the metamodels.

Both organizations found the developed method useful: interviews showed that the method developed was considered to work better than those used earlier. For example, in the mill case the consultants estimated that with the methods and tools used earlier, they could perform only half of the modeling and analysis tasks supported by the engineered method and tool. In general, models based on the developed method were considered to be easier to read and understand, to support communication better, and to allowed the combination and analysis of views of multiple stakeholders, or even of multiple organizations. Moreover, in the mill the connection of the models to the quality system was important.

Satisfaction with the developed method does not mean that no problems existed. In fact, the method refinements clearly showed that the methods had to be improved. The main difficulties included different interpretations of conceptual structures and analysis. Most of the effort in method development related to agreeing and confirming an understanding about the method and tool functionality. In the mill case, the consultants also considered the objectives for the engineered method to be too ambitious. Meeting these objectives in turn required significant resources and time.

Satisfaction with the tool was surprisingly different among user types. Method engineers considered the metaCASE functionality limited (i.e. all metamodeling constraints could not be supported). As a result, a lot of time was consumed while trying to find roundabout ways to build method-tool companionship. People in the organizations using the tool, however, were highly satisfied with the tool, although they requested several new features which did not directly address the method-tool companionship. These included importing available process maps into the tool (e.g. from the documentation of a quality system), providing links to external documentation tools, and improving method-independent reports.

The method engineering process was quite similar in both cases and all ME tasks were carried out. One reason for the similarities was the tool adaptation which required detailed method specifications. The similarities in the ME process were also a consequence of planning the action research, since the ME process needed to include the a posteriori ME tasks postulated.
During ME, a priori method selection (cf. Section 3.2.1) was made among relatively few methods, and included methods which were known or had been used earlier. However, in the second case selection was supported by a relatively large review of methods (cf. Tolvanen and Lyytinen 1994) and tools (cf. Lindström and Raitio 1992). During ME neither of the cases applied contingency frameworks, because such frameworks were not available. Those reviewed were considered to be too broad since they did not help to distinguish between modeling techniques based on identified characteristics. Similar observations were also made in Section 3.2.3 while analyzing other ME cases.

The main differences in the ME processes were the metamodeling languages used, the emphasis on different ME tasks, and the types of stakeholders involved. First, in the wholesale case the only metamodeling language applied was that used by the metaCASE tool, whereas in the mill case the early phases of method development had been carried out with another metamodeling language (i.e. the ER-based logistics data model). However, here a metamodel of the activity model method already included in the metaCASE tool was taken as a starting point during tool adaptation (i.e. reuse of an existing metamodel).

Second, the use of resources and duration of tasks differed greatly. In the mill case, the method engineering took more time and resources. One obvious reason was the objective of the ME project to develop a general purpose method. In other words, the method was expected to be applicable for solving logistic problems in other areas too. Moreover, the mill case stressed tool adaptation because it required implementation of the analysis functionality. About 1/3 of the resources were spent on tool adaptation, whereas in the wholesale case the tool adaptation was the least resource-consuming task.

Third, the participation of method users differed in the cases: the IT personnel of the wholesaler participated actively in the method construction and evaluation, whereas the personnel of the mill did not directly participate in the ME project. One reason for the difference was the two-party setting between the consulting company and the mill.

As the objectives of action research indicated, our interest was to demonstrate the viability of incremental ME principles. First we analyze whether the situational methods were possible to describe with meta-data models and the proposed metamodeling constructs. Second, we analyze whether the a posteriori view was appropriate as a mechanism of method evaluation and refinement.

In both cases, methods were modeled with metamodeling languages embedded in a metaCASE tool. This was needed to provide modeling tools for the ISD projects. In the mill case the metamodeling also included ER-based modeling, but only for outlining the concepts and relationships used in the method. Moreover, the ER model was used for metamodeling before the selection of the metaCASE tool.

Not all method knowledge, however, could be fully supported, because of the limitations in the metamodeling language. These limitations concerned modeling property types with unique, mandatory, and data type constraints. Other limitations related to defining that an object must participate in at least one connection, must have a specific number of instances, and can not participate in connections which are cyclic. Most limitations were related to integrating modeling techniques: constraints related to complex objects (i.e. exclusive component objects and aggregated relationships) and polymorphism structures (i.e. sharing of several property types at a time, and dependency on other instances) could not be defined.

The proposed metamodeling constructs allowed methods to be specified more completely than was adapted into the tools. In this sense, they can be considered sufficient for engineering the methods in the cases. Only one limitation in capturing method knowledge was found. In both cases a derived property value had to be specified. An example of the derived data type is calculating the processing time of a task from its subprocesses. Because this type of dynamic calculation is difficult to capture into a static data model, this requirement suggested the use of metamodeling languages other than those based on semantic data models (see also Section 4.5.3).

It must be emphasized that not all proposed metamodeling constructs were fully applied since not all possible rules of methods were needed in the cases. For example, an explosion structure was used (refined method in Figure 6-4) but cardinality of explosions were not used. Similarly not all scopes and checking modes related to the metamodeling constructs were applied. One reason for the limited use of scopes may be because of the relatively simple structure of the methods in the cases, i.e. the fact that they included only a few modeling techniques. Thus, the viability of every metamodeling construct was not demonstrated via the metamodels. During ME, however, awareness of method knowledge which is not specified into metamodels is valuable because it allows one to understand alternative method configurations and metamodeling choices.

The limited use of metamodeling constructs also suggests that specific metamodeling constructs are needed while modeling specific methods. For example, the requirement for modeling derived data types was not detected during the modeling of text-book methods (Chapter 4) which were mostly IS analysis and design methods. In contrast, our case studies required business modeling methods which often incorporated numerical values.

Although our aim was not to evaluate the metaCASE tool used, their limitations and capabilities influenced the metamodels. First, because the metaCASE tool did not support matrix-based representations tool adaptations were not made for all parts of the method. Second, checking reports allowed us to overcome limitations of the metamodeling languages through passive checking. Passively checked metamodeling constructs related to mandatory properties, unconnected object types, multiplicity of roles, and multiplicity of types.

The case studies followed the principles of incremental ME: in addition to constructing methods we also evaluated their applicability. This evaluation led to several refinements to the methods initially constructed. This finding supports our re-evaluation of method use (cf. Section 2.5.4): it seems to be difficult, if not impossible, to construct a situation-dependent method by following solely a priori tasks of ME.

The requirements for method refinements were obtained by following the a posteriori steps of ME: collecting experience, analyzing method use, and improving methods. The reporting of the ME cases followed the three evaluation mechanisms of incremental ME: 1) type-instance matching, 2) modeling capabilities, and 3) problem solving capabilities.

Type-instance matching suggested a large number of method refinements. These dealt with removing less frequently used property types, dividing object types and classifying relationship types, and creating linkages between types of the method which can refer to the same model data (instance values). The analysis of constraints leads to modifying methods in more detail. The changes introduced influenced all constraints except those dealing with uniqueness, cardinality, and inclusion. In other words, these constraints were defined adequately in the a priori constructed methods. It must, however, be noted that not all constraints could be fully evaluated since they were not allowed in metamodels. Although this part of the evaluation was carried out by method engineers, all the refinements were validated with the method users.

The modeling capability evaluation seeks to abstract relevant aspects of object systems and keep the resulting models consistent. The cases contained several situations in which methods were considered inadequate to model the object systems or parts of them. Individual differences were not analyzed, because only a few users actually modeled with the method: other method stakeholders were involved in reviewing the models and conducting analyses. This part of the evaluation proposed new types and related constraints, and even new modeling techniques. These extensions were partly the same as those found during type-instance matching.

The checking analysis revealed possibilities to improve the consistency of models via an integrated and more strictly defined metamodel. Better support for consistency was achieved by defining complex objects and polymorphism structures. These refinements decreased the need for manual maintenance and improved the consistency of models. The evaluation also revealed the need for derived property type values, and the use of a single tool. These changes, however, could not be carried out because such method specifications could not be captured in the metamodels.

The evaluation principles focused on analyzing the role of methods in solving the business problems. This part of the evaluation was divided into finding candidate solutions through form conversion, and supporting model validation by producing documents. During form conversion, new features relevant for the generation and analysis of design solutions were suggested, together with new analysis algorithms. In the second case, larger extensions to the analysis needs were also made, i.e. the addition of Activity Based Counting. Support for exporting design data and analysis results into external tools were considered adequate.

The integration and validation of models from different parts of the object systems required a new modeling technique. Other refinements suggested dealt with notations and reports: notations were modified by graphically showing information about model data instead of using model related textual reports. The contents and layout of reports were also changed.

The required method refinements were made into the corresponding method specifications (i.e. metamodels), or were achieved using the tool (i.e. checking reports). Not all the required changes, however, could be made because either the metamodeling language or the tool did not support them. In this sense, not all suggested method improvements could be taken into use. It should be noted that none of the required changes could be predicted. Moreover, because the refinements were found to improve the method, the a posteriori approach to ME is clearly viable.

Finally, while evaluating the results of action research it must be noted that the use of incremental ME was limited to one cycle (less than a year). For example, contingencies in the ISD object system did not change during that time. Therefore, new method refinements would be needed if the evaluation were to be carried out again. At the same time, the contribution of individual evaluation mechanisms would mostly likely be different.