Empirical studies of local method development are rare (Wynekoop and Russo 1993, Tolvanen et al. 1996). This makes the selection of a research method for our case more difficult: we can not directly build upon other work and confirm (or challenge) its findings. Though several studies have investigated method use in practice (see the studies discussed in Section 2.4 and surveys by Wynekoop and Conger 1991, Tolvanen et al. 1996, Sauer and Lau 1997, Wynekoop and Russo 1997) they operate at a general level of method knowledge. Surveys and field studies do not address detailed method knowledge; rather they show that adaptations occurred (see Section 2.4). Case studies have been carried out on method introduction and use, resistance to change (Wynekoop et al. 1992), social defense (Wastell 1996), and stakeholders’ interests (Sauer and Lau 1997), but none of them addresses the situational fit of the method use. As a consequence, most of the ME approaches reviewed in Chapter 3 are unproven for local method development efforts. Those approaches which include demonstrative cases do not go into details (e.g. Punter and Lemmen 1996) and address only a priori ME, i.e. mostly the construction phase.

Because ME is a relatively new research field, complementary research efforts and a variety of research methods are needed. To achieve the necessary pluralism we need more empirical studies (Tolvanen et al. 1996). Too often ME approaches, metamodeling languages and metaCASE tools are developed without an empirical grounding. Among empirical research approaches we believe that action research is appropriate to examine ME. Several researchers give support to this research approach (cf. Galliers and Land 1987, Galliers 1992, Wood-Harper 1985, Checkland 1981, Grant et al. 1992) in the context of studying ISD methods. Reasons for applying the action research method in our studies are manifold: it resembles incremental ME, it is iterative, it allows us to go into details, it is situation driven, and it offers possibilities for longitudinal observation.

Before we describe how the action research was carried out in this thesis (Section 6.1.2), and its similarities with incremental ME (Section 6.1.3) we briefly describe the action research method.

Action research can be understood as a variant of a case study and a field experiment (Galliers 1992). Analogously to a case study, action research uses evaluations of particular subjects, such as an organization, a group of people, or a system at a point of time. It attempts to capture the “reality” in greater detail and typically no control of the phenomena is exercised. Unlike a case study, in action research a researcher participates and acts in the area of study and simultaneously evaluates the results of this participation. This dual role means that the objectives of the research are twofold: on the one hand, the action researcher aims to improve the situation in the organization. Thus, action research resembles any organizational development or consulting effort. On the other hand, the action researcher aims to contribute to scientific knowledge by creating generalizable concepts and theories of the problem setting and its behavior. The generalization is necessary for future settings, and for researchers to build better theories.

The close interaction between theory and practice in action research means that during the research process, the roles of a research subject and a researcher can be reversed (Galliers 1992). As a result, the process of the action research separates the phases where action is taken, and where its results are evaluated (Checkland 1991, Jönsson 1991, Baskerville and Wood-Harper 1996). The dual role necessitates that action researchers be aware that their presence will affect the situation. Unlike case studies, the action research method permits intervention of the researchers into the events. In fact, the possibility to plan interventions and record them for evaluation purposes forms the essential mechanism of action research. The intervention can vary from direct intervention as an equal coworker, to indirect intervention through a catalyst role. An example of direct intervention would be participation in the method selection, and an example of indirect intervention would be playing an expert role in tool adaptation. However, in both modes the changes to be made must be planned and the effects of the actions recorded. This part of action research resembles a highly unstructured field-experiment. The process of the action research method is described in the next section in which its application in this study is explained.

The possibility to test and refine principles, tools, techniques, and methods, as well as to address real-world problems, makes the action research method very appropriate for organizational development (van Eynde and Bledsoe 1990) and for IS research (Baskerville and Wood-Harper 1996). The advantages of the action research method compared with other approaches come from the possibility to obtain a deeper, first-hand understanding of the situation. The action research method allows collection of information which would be difficult to obtain by outsiders, and permits use of longitudinal research designs (Checkland 1981, 1991, Baskerville and Wood-Harper 1996).

Action research also has limitations. The approach offers few possibilities for statistical generalization, and no possibility to exercise control over experimental conditions. Because action research is largely interpretive, its results can also be interpreted differently by individual researchers. The dual role of the researcher also raises some ethical problems: the goals of practice and research can be conflicting. For example, organizations often expect quick results whereas the researcher may expect slower and more gradual progress. Conflicts can also be faced by the individual researcher having to act in both roles simultaneously. The funding structure behind action research raises a dilemma when the researcher is financed by the organization examined. Although the funding indicates some commitment of the organization to the study and access to data as a “worker” of the organization, the researcher must seek to satisfy the organization’s objectives as well. For example, it is not usually possible to study failures by consciously planning them. As in all research, action research must be planned to obtain scientific knowledge, and to overcome or minimize the limitations of the research approach followed. In the following section we describe the action research method followed in this thesis.

Several models for action research can be found (cf. Baskerville and Wood-Harper 1996, Checkland 1991). They all consist of steps like planning actions, taking actions, and evaluating their results. In addition to these, entry and exit points to an organization must be planned (Buchanan et al. 1988). In studying incremental method engineering, the use of action research can be described according to the process model illustrated in Figure 6-1.

FIGURE 6-1 Action research process model followed.

1) Entering means getting access to real-world ME cases and establishing the action research. The criteria for selecting cases can be derived from our problem formulation. The site must be in the middle of a ME problem, deciding how to engineer a method for a particular IS development need?. In our study, the cases included two organizations which needed methods to carry out specific system development efforts. The first case was related to a business process re-engineering effort involving an inter-organizational IS development in a trade organization. The second case deals with developing sales and outbound logistics in a cardboard mill. Both of the ISD environments were suitable for our study because they lacked methods and detailed method selection frameworks. Both cases had several external stakeholders, were dynamic and consequently had a high uncertainty. Both cases also provided the possibility for longitudinal observation: access to the organizations was possible also after method construction.

Access to cases was obtained within a larger research project in which both companies participated. The funding was not based on the work carried out for the organizations but instead through the organizations’ participation in the research project. Participation was arranged as development projects. In these projects, ME efforts were organized as subprojects. In both cases, the ME projects had specific goals and a separate project plan which described its resources and schedule. In the latter case, the ME project also had a separate budget.

2) Action planning involves decisions about the objectives and questions of the study, and shows how the study bears on these objectives. The research objective of our studies was to demonstrate the viability of incremental ME. We examined whether situational methods could be specified using the proposed metamodeling constructs, and whether the a posteriori ME principles could be used to refine methods. The former was studied by analyzing whether all aspects of modeling techniques could be described with a semantic meta-data model. The latter was inspected by analyzing the outcomes of the ME efforts and the use of methods. If the a posteriori evaluation mechanisms neither revealed possible method refinements nor supported learning of method use, they could be considered inadequate. Further studies could then be carried out to analyze which circumstances favor incremental ME and which favor a more “radical” ME approach (cf. Section 5.1.3).

During action planning one also determines data collection mechanisms. To understand the constructed methods and possible method changes in detail, the data collected included metamodels, adapted tools, method manuals and other method descriptions, such as domain models and algorithms for model-based analysis. Method construction rationale was collected from documents describing the requirements for methods, from memos and minutes of the ME projects’ meetings, and by participating in the ME process. To understand the actual use of methods, the models developed, the analyses created, and the project documentation were collected. In particular, access to design data stored in tool repositories was important because it allowed to inspect how the method was actually used with the tool. The project reports describing the deliverables of the ISD projects formed anther source of information about method use.

Data collection also covered requirements to change methods and method versions. These were captured in documents relating to ME, and were outcomes of using the mechanisms of a posteriori ME. Finally, method engineers were interviewed twice during the study: first while the method improvements were being sought with the mechanisms of a posteriori ME, and second at the end of the project. The latter interviews were applied to verify earlier observations and to establish satisfaction (or dissatisfaction) with the method and tool developed. Interviews were not recorded but notes were taken and the resulting descriptions were checked by the method engineers interviewed.

3) Action taking carries out incremental ME as described in Sections 3 and 5, i.e. applying both a priori and a posteriori principles. Both ME phases were carried out by collaborating with stakeholders. They identified ISD problems and provided experience to assess method applicability. The method users were partly the same people who engineered the methods. In the wholesale case, the researcher also participated in the ISD efforts, but in the mill case, participation was limited to tool adaptation and guiding the evaluation mechanisms. Hence, in the latter case the researcher played an expert role. In this sense the latter case was closer to a case study.

4) Evaluating the effects of actions analyzes whether the actions have been taken as suggested and how they have affected problem solving. The results of ME actions were described in a baseline documentation which was checked for correctness by method engineers. Using this document, ME actions were analyzed based on metamodels, supporting tools and method manuals. The evaluation included an analysis of whether the metamodeling constructs were applied, and whether their use was considered successful. In the a posteriori phase, we analyzed whether changes followed incremental ME principles, and whether the changes improved methods.

5) Exit stops the action research cycle. Cycles of action research were simultaneous with the cycles of incremental ME. Each time a method was evaluated the results were recorded and analyzed. This allowed us to carry out a longitudinal study rather than inspect only a snap-shot of the methods (as in Section 2.5 while analyzing the descriptions of ME cases). Because of the time and resource constraints, the actions were limited to one cycle.

Although the cycles were simultaneous they occurred at different levels (Checkland 1991). The ME cycle deals with learning about methods. The action research cycle deals with learning about ME in general and about incremental principles. Both of these cycles should be documented as part of the research (Jönsson 1991). To clarify the different levels of actions we shall briefly compare the action research method and incremental ME.

The description of the action research method shows that incremental ME resembles it in many ways. These similarities are summarized below:

Both are iterative and focus on long-term changes. In principle they form never-ending processes in which learning is used in the consequent cycle.

Both ISD and ME methods are studied in real organizations and in actual use. This focus allows the study and improvement of methods and related technologies. Because incremental ME principles are being promoted only action research and field experiments can be applied to study their viability.

Both are situation-dependent, which can also be considered a weakness if statistical generalization is an objective of the research. On the ME level, the need for situational dependency was already discussed in Section 2.5. On the action research level, situational dependency is reduced through research cycles and by carrying out studies in different organizations (in this thesis two organizations).

Despite these similarities a key difference must be recognized: while the incremental principles operate at the ME level, the action research operates one level higher. These levels are summarized in Table 6-1 based on the domain in which they are applied. In this chapter we operate at the research level since our interest is to study the incremental ME principles. The research subject is method engineering and the mechanisms of incremental approach.

Because of this focus on the research level, in the following sections ME efforts are described from the research point of view. Studying ME principles, however, necessitates that we also describe how methods were constructed (the ME level) and how they were applied (the ISD level). This means that the cases are reported at three levels as follows: Sections 6.2 and 6.3 describe ME cases which followed incremental ME principles (i.e. the action taking part of action research). Although the reporting focuses on how methods were specified and evaluated with the metamodeling constructs, the ISD level must also be recognized. This allows us to explain how the constructed methods were applied and the rationale for their use. The outcomes of action research (i.e. the evaluation part of action research) are described in Section 6.4, in which lessons learned from ME and from the incremental principles are discussed.

TABLE 6-1 Levels of research, method engineering, and information system development.