Before extending ME principles further we need to argue for the necessity of an incremental approach and describe why and how the current ME principles are inadequate. This is important because there is a paucity of studies focusing on principles to support method evolution through ME principles (Tolvanen et al. 1996).

In the following subsections we first give basic motivation for the incremental approach and define it by taking an a posteriori view of the ME process. Second, different scenarios for refining methods are identified. Identification is based on analyzing the origins of ISD related experiences and distinguishing targets for making method refinements. The section concludes by discussing differences between incremental and more radical ME approaches. This allows us to describe local method development situations where the proposed principles are most suitable.

The motivation for incremental ME comes from our re-evaluation of method use (Section 2.5) and from the limitations of current ME principles (Section 3.2). In short, we claim that the situational applicability of methods is difficult to achieve solely through a priori ME principles. Major reasons are that 1) the required information for method construction is not sufficiently available, 2) the criteria that can direct method construction are difficult to identify beforehand, and 3) the ISD environment evolves.

In relation to availability, there are not many detailed metamodels available, nor readily applicable frameworks of ME criteria which are “filled” with known situational characteristics and related to metamodels. Most of the metamodels - which come with metaCASE tools, repositories, or are described in reference books (e.g. Olle et al. 1991) - focus on a limited number of methods and/or on only specific types of methods (e.g. object-oriented methods). Moreover, the metamodels described in books are not usually specified unambiguously and are at a relatively coarse granularity, at least when compared with the detailed metamodels required to model operational techniques. As a result, the pool of methods specified with metamodels for comparison and selection is small. Moreover, the frameworks of ME criteria, like the contingency frameworks applied to ME (i.e. Punter and Lemmen 1996, Harmsen 1997) focus on only a few aspects of methods applicability, and only on general method knowledge (see Section 3.2). The combination of metamodels and ME criteria into a larger baseline is difficult: different metamodeling languages focus on different types of method knowledge at different levels of detail and are not usually related to detailed method knowledge. Even assuming that a large body of metamodels and related ME criteria were available, the maintenance of this knowledge would be a huge or even unrealistic task. This fact also partly explains why contingency frameworks operate with method knowledge at a general level.

As a result, organizations are forced to test methods and try to make them more applicable by “learning” and deviating from them while they are used. After all, organizations can not stop developing their own variants although theoretically this can be assumed (e.g. Wynekoop and Russo 1993). Similarly, the proposed incremental principles take an a posteriori view of ME by seeking to understand the applicability of methods through an organization’s experiences. The a posteriori view also forms a key distinction to current ME principles reviewed in Section 3.2. It must be emphasized that typically the experience gathering and method refinements are carried out haphazardly and on a trial-and-error basis without any systematic principles (Smolander et al. 1990, Hughes and Reviron 1996).

We claim that it is difficult, if not impossible, to identify beforehand all relevant criteria for method construction. For example, the ME criteria of van Slooten and Hodes (1996) - resistance of end-users, aspects of the system to be analyzed, and management commitment - can hardly be known completely beforehand. Furthermore, as argued in Section 3.2 their relationship to method knowledge is not clear. In fact, van Slooten and Hodes apply the contingency framework in an a posteriori manner to analyze whether the criteria proposed in their framework have affected past projects and thus could be relevant to ME. How they can be applied to construct methods is not discussed. In contrast to the current ME view, cases of local method development (Jaaksi 1997, Tollow 1996) show that the characteristics and problems to be solved with methods were not known beforehand because of uncertainty about the problems.

As a result, we claim that the requirement for complete prior knowledge is both idealistic and unrealistic. Consequently, in situations of uncertainty and limited information, the incremental principles focus on improving method applicability through promoting small changes to methods while an organization obtains experiences and learns both about the method and about the IS domain. Although this option is partly dictated by practical needs, it also allows the creation of new knowledge based on experience, regarding both the method and the ME criteria. This is important, because current ME approaches rely on the existing body of information about both methods and ME criteria. Therefore, ME must be viewed as a learning process in which experience of successful (or unsuccessful) ISD efforts needs to be incorporated into future ME efforts: every use situation of methods should evaluate and analyze methods with a view to improving them. In fact, keeping the situational dependency of method use in mind, the most reliable information about method applicability can be obtained from an organization’s own experiences. This experience-based learning is generally an incremental process (Miner and Mezias 1996), and a main argument in favor of incremental ME.

A method use environment is hardly stable because situations can change even during a short ISD effort. These changes also affect the applicability of methods, leaving two options for method engineers: either continue the use of the method in its current state, or modify it to support the new situation. The former option is chosen at the cost of applicability and the latter at the cost of making a new version of the method and transforming models which have already been made. This topic is discussed in Section 5.1.3. Changes in ISD situations are common, and can be seen also in the documented ME cases (e.g. Cronholm and Goldkuhl 1994, Nissen et al. 1996, Jaaksi 1997). These show that once methods have been adapted to tools, requirements for maintaining and modifying the methods for new situations appear immediately. In fact, some of the requirements occur already during tool adaptation, or after a pilot use. As a result of this evolution, methods must be refined continuously.

At the level of the ME criteria, changes in the situation have been identified as changes in contingencies, shifts in problems of ISD, or changes in stakeholders’ requirements and values (Kumar and Welke 1992, Joosten and Schipper 1996). On the contingency side, one of the main reasons for introducing an ME approach is the inflexibility of contingency based method selection (Kumar and Welke 1992): in the worst case, a change in one contingency could lead to a selection of a totally different method. Similarly, a typical shift in problems to be solved (Checkland 1981) necessitates changes in methods: only in the case of a tightly defined and enclosed system can a method be presumed to be applicable every time. Finally, neither stakeholders nor their requirements are stable. People participate at different times and new people can raise different requirements and have different values which need to be reflected in methods (Nuseibah et al. 1996). Similarly, stakeholders’ assumptions change and they can not know all the relevant criteria (Joosten and Schipper 1996).

To summarize, an incremental approach extends, rather than substitutes, the current principles of ME by focusing on experiences of method use, i.e. on an a posteriori instead of an a priori view. By evaluating the applicability of methods in a given situation it aims to manage and refine methods. The accumulated experience can lead organizations to extend, modify or purge any part of the method knowledge, such as concepts, constraints, or notations. These refinements are gradual and small in nature, hence the name incremental for the proposed approach. Gradual means that method refinements are applied to the method currently used in a given situation, instead of selecting a radically new method. Small changes are a consequence of the gradual changes: applicability is achieved by modifying selected parts of the existing method knowledge. Before describing the incremental principles in more detail we first take a closer look at method evolution.

In the following subsections we analyze incremental ME according to two dimensions: the source of experiences and the target of method refinement. Identifying the sources of experiences allows us to find mechanisms to collect experiences and make them available for method engineers. Experiences can lead to method modifications in different phases of the ME process. This is described through the target of method refinements. In the following we shall also address relationships between different types of method knowledge by describing how requirements imposed on a method’s conceptual structure affect notations and supporting tools, or vice versa.

A requirement to modify a method arises when the method does not meet the situational requirements. These requirements can be collected while adapting a method to a tool, while introducing a method into an organization, or while using the method[23]. Each of these alternative sources can lead to iterations in the ME process and to a method modification. In each view the applicability of a method is determined based on different criteria:

1) Tool related feed-back occurs when a customizable modeling tool has limitations to support the constructed method (Cronholm and Goldkuhl 1994) or it offers possibilities which have not been considered earlier (Tolvanen and Lyytinen 1993). For example, most of the ISD methods used today still follow a “pen-and-paper”-mentality: they do not take full advantage of computer-based modeling environments. Therefore, the applicability of the method is determined here through a method-tool companionship.

2) Introduction of methods. Method refinements can also originate from the introduction or “pilot” use of methods (cf. Nissen et al. 1996), when a larger group of stakeholders can analyze the constructed method and tool support. Here a method is typically assessed in terms of its supporting materials, like tutorials, manuals, example solutions and reference models, as well as features of the method supporting tool. The applicability is determined mostly according to the pedagogical aspects, like how easy it is to learn and introduce the method into an organization. Some ME approaches focus on constructing methods so that they work as a learning device for teaching ISD methods (cf. Mathiassen et al. 1996).

3) Experience-based feedback occurs when developers face situations in which they feel that the constructed method is, or is not, applicable. If the method is considered inapplicable, they may rely on their experience more than on the use of the method. Hence, the applicability of the method is viewed in the light of current circumstances. This type of feedback is important because it is founded on actual method use. Several researchers (cf. Wood-Harper 1985, Galliers and Land 1987, Galliers 1992, Checkland 1981, Grant et al. 1992) have also emphasized the importance of the problem situation in which the method is used as a basis for evaluation.

In this thesis our main interest is on experiences related to method use, although the other sources mentioned are also possible starting points for iterations. Modifications which arise from the functionality of the tool are mostly dealing with technical issues and not related to the applicability of modeling techniques. In other words, our primary focus is not on the evaluation or improvement of modeling tools and their support for method-tool companionship. However, we claim that this type of empirical approach is required for evaluating tools (Tolvanen et al. 1996). Similarly, we do not consider here the effect of teaching approaches, other method-related materials, or the effects of piloting approaches, although we acknowledge their importance.

Collected experience can lead to method modifications at different phases of method development. This dimension describes whether the iteration leads to re-select a method or its parts (i.e. start the ME process from the beginning), to construct the method differently, modify the method only to achieve better tool companionship, change the way in which the method is introduced, or to interpret the method differently. Each target is described in the following.

1) Refine the method while using it. This possibility means making different interpretations and giving different meanings to the method knowledge. This type of refinement takes place in learning-by-doing when an individual learns by developing an IS. Moreover, it must be noticed that this type of refinement can often occur without any language or documentation. Hence, the refinements can also be tacit (Nonaka 1994, Hughes and Reviron 1996). For method refinements this means that experiences about methods are not shared and thus not explicitly used to modify “intended” method knowledge. As a result, method refinements performed only while individual persons are using the method are difficult to study and systematize, because of the tacit nature of method-related experiences.

An individual person’s refinements can be externalized (Nonaka 1994) by making them explicit and thus available to method engineers and other stakeholders. Experiences can also lead to organizational learning if they are collected and shared in some way with other participants. The remaining four phases of method refinement presuppose mechanisms to collect experiences and to make explicit changes to method knowledge.

2) Changes in the introduction or “piloting” phase deal with method refinements which change the way in which a method is taught; modify method-related materials, such as example solutions, tutorials, manuals; or demand more easily learned versions of the method. This last approach is commonly used both in text-book methods (e.g. Booch 1991) and in local variants (Jaaksi 1997) by developing “light” versions of the method. These are simpler and easier to learn, yet applicable for small or “first” projects.

3) Refinement of the method in a tool aims to make the use of the method easier with the tool. Examples of such modifications are changing the order in which the design information is added to the tool, or changing reports for consistency checking. Issues related to the tool only, such as the layout of dialogs and the order of reports and techniques shown in dialogs or menus fall into this category. Because these modifications change the behavior of the tool, they are explicit and well-structured changes, but deal with the surface structures (Wand 1996) of method knowledge.

4) Re-constructing a method represents more profound changes to the method knowledge. These include modifications to the initially constructed method knowledge: existing method components in the metamodel are removed or modified. A re-construction of the method also requires modifications to the underlying rationale applied for selecting and constructing the method in the first place. Ideally, each modification should be evaluated based on earlier knowledge of the method’s applicability: why a certain methods was not applicable, and how the modification can improve its applicability.

5) Selecting a new method or its parts deals with the most profound refinements to methods. Here new method components, like types, constraints, and modeling techniques, or totally new methods are selected. These changes are also typical when the re-construction requires new components or when unforeseen contingency factors arise, or there is a significant change in existing factors (van Slooten and Hodes 1996).

Together the two dimensions, the source and the target, form a space for possible method refinements. These are illustrated in the cells of Table 5-1. The horizontal axis shows the sources of experience and the vertical axis shows the targets of method refinement. The arrows show all possible choices when method refinements can take place: from the starting point of an iteration to the point of making the refinement. These scenarios are important because they allow us to restrict our view to experiences based on method use, and their influence on method refinements.

According to the possible scenarios, experiences about methods can be collected before, during or after the use of the method. Our interest in method refinements is in those related to experiences gathered from method use which can be externalized, i.e. represented, analyzed and refined with metamodels. These situations are grayed in the table. Thus, the first four phases of the ME process, which can also raise requirements for method refinement, are not considered in this thesis. They expect evaluations to be carried out before a method is used and are already partly covered in a priori ME approaches. Hence, we believe that the applicability of methods can be known only when the method is used.

The selected scenarios also reflect the depth of method refinements because all refinements made to metamodels expect modifications to be made to later phases of ME, i.e. to tools and their introduction phase. These are, however, also supported in most of the ME frameworks (see Section 3.2).

TABLE 5-1 Scenarios for incremental method refinements.

Method refinements can not always be carried out by modifying only one type of method knowledge. Instead, during the construction phase the modifications are interrelated: changes in one part of method knowledge cause changes in other parts of the method.

Based on our focus on modeling techniques, we shall analyze only two fundamental types of method knowledge subject to modifications, namely the conceptual structure and the notation. These types and their relationships are also identified by Kronlöf (1993) and Jarke et al. (1998). Thus, other types of the method knowledge shown within the shell model (Figure 2-2) are excluded. However, to emphasize the companionship between a method and a tool, we also analyze tool-related method refinements. Therefore, method refinements can deal with the following interrelations: 1) conceptual structure and notations, 2) conceptual structure and modeling tool, and 3) notation and modeling tool.

1) Conceptual structure and notations. The most drastic modifications in a method occur when a large portion or the whole conceptual structure is changed, as when shifting from structured methods to object-oriented methods. Similarly, domains which are less mature, such as business modeling and requirements engineering, have less stable concepts and thus are more likely to evolve.

Because the underlying conceptual structures are typically the foundation of the method, changes in the conceptual structure affect other types (Jarke et al. 1998): notations which represent these concepts, processes which operate on these conceptual structures, and computer-aided tools which capture, store, analyze and retrieve the models representing those conceptual structures. For example, adding a new type to the conceptual structure requires changing the notations by adding a new notation for that type. Accordingly, the completeness of representations (Batani et al. 1992, Venable 1993), i.e. the availability of a notational construct for each concept, is a well-known criterion for dealing with the relationship between the conceptual structure and the notation. In contrast, changes in notations do not necessarily affect the conceptual structure (cf. Ryan et al. 1996, Kronlöf 1993). With respect to method refinements, we identify here a causal relationship among interrelated modifications: all changes to the conceptual structure should be made before changes in notations. Changes in methods, however, often arise from notations because they are the most visible part of the method.

2) Conceptual structure and modeling tool. Method modifications can also occur because a tool can not provide the required modeling functionality, such as an abstraction mechanism, a checking, or a form conversion. Here the method may need to have additional constraints or properties to enable consistency checking, reporting or code generation. For example, most of the object-oriented design methods do not recognize whether an inheritance is virtual although this information is required for generating header files for C++. Although these concepts are added to the tool, they are also defined and maintained in the metamodel.

3) Notation and modeling tool deals with the surface structure (Wand 1996) of the method knowledge: method refinements are made by modifying symbols and notations based on the graphical capabilities of the tool.

Although most of the metaCASE tools available provide method adaptation possibilities (Marttiin et al. 1993, 1996) they focus on modifications which are carried out when the tool is introduced. Accordingly, later method modifications are difficult, if not impossible (e.g. Cronholm and Goldkuhl 1994, Nissen et al. 1996). It must be noted that by method modifications we also refer here to situations in which the models made so far are updated along with the modified method (e.g. to support reuse of designs).

Not all method development efforts are necessarily gradual or require small modifications to methods. In general, the literature on the development of business processes and on organizational learning distinguishes between radical and incremental approaches. For example, business process re-engineering (BPR, Hammer and Champy 1993) advocates a radical approach in terms of the rapidity and magnitude of a change, whereas total quality management (TQM, Flood 1993, Oakland 1993) relies on continuous small changes. Similarly, there is a wide-spread consensus on the distinction between incremental and radical models of learning (Miner and Mezias 1996).

Generally speaking, the type of change required and the type of learning are related: carrying out a radical change necessitates that the organization is capable of radical learning (i.e. to implement and introduce a large change quickly). Conversely, continuous small changes to existing processes expect incremental learning. Both approaches can produce benefits for an organization, and both types have advantages and disadvantages. In this sense they provide alternative strategies depending on how often the change is made and how large the change is. These alternatives are also valid in ME. By “radical” ME we mean a priori ME approaches in which methods are constructed solely in the beginning of each ISD project. This type of change is also the one most studied in ME research (see Section 3.2). It can be considered radical because each ISD project and each ME case is handled separately. A method is expected to be introduced once and no reflective learning during the method use is incorporated into methods. As described above, incremental ME is based on smaller and more gradual changes. At the extreme end of the scale, method refinement can be continuous and concurrent with the change requests from the method use environment.

Both modes of ME can be useful, depending on the situation, as both modes have their advantages and disadvantages: not all changes to methods can be radical, but on the other hand, small gradual changes may hinder development efforts which require more substantial changes. This also means that not all local method development efforts can be carried out according to incremental ME principles. Therefore, in the following we describe characteristics of ISD organizations or projects where incremental principles are most suitable. These characteristics are summarized in Table 5-2.

TABLE 5-2 Radical versus incremental modes of method engineering

As our motivation for the incremental approach showed, ISD environments exist where high levels of uncertainty and unavailability of method knowledge are typical and applicable for incremental principles. Areas of ISD where there are few methods available are, for example, the development of inter-organizational ISs (cf. Tolvanen and Lyytinen 1994), hypermedia systems (Isakowitz et al. 1995), and networked business processes.

Second, the longer an ISD project takes the more an organization will garner experience and the more likely are method modifications. Moreover, longer projects are also often larger and technically more complex, necessitating approaches to combine methods. Similarly, in long-term ISD efforts the technologies used may change and these changes need to be considered.

Third, as emphasized in our re-evaluation of method use (Section 2.5.3), successful method improvements are tied to an organization’s own experiences and to the level of maturity. Therefore, ME efforts relate to the maturity of ISD (Humprey 1988): organizations must have methods in use and an ISD process defined before any systematic experience gathering can be carried out. Also, method refinement efforts expect that methods are specified - otherwise their improvement is difficult (Jarke et al. 1994, Odell 1996). This means that an organization using incremental ME principles must be at least at the defined level according to the SEI maturity levels. In fact, the higher levels of maturity can be partly achieved by using incremental ME principles, in which methods are managed and optimized for the current situation. With respect to maturity, the organization’s current method situation reflects the mode of ME. Incremental ME is not necessarily an optimal strategy for initiating radical changes in ISD, e.g. adopting methods to be used for the first time, or moving from structured methods to object-oriented methods.

Fourth, incremental ME principles are more applicable for organizations which can invest in method knowledge. Quite often this is possible only when the method knowledge can be focused on specific areas. These types of situations are typical in ISD organizations which focus on longitudinal projects and on developing a limited number or type of applications. In contrast, consulting houses which provide services to other organizations find their choice of methods largely determined by customers’ requirements. As a result, method requirements can change from one customer to another and accumulated knowledge can not be utilized as effectively. Hence, in these cases the method selection and use can usefully be radical.

Finally and perhaps most importantly, one reason for following either of the ME approaches comes from their projected costs and benefits: how to change the method without discarding expensive investments in technology and methods. With respect to the technology investments, metaCASE tools are seen as offering one solution (Seppänen et al. 1996), as they decrease the costs and resources needed to manage method knowledge, and also provide a platform for cost-effectively building new CASE tools for a changed method, or different versions of methods. This also explains our interest in analyzing method use in modeling tools: metaCASE tools which support situation-specific evolution of methods are already available (Kelly 1997), but ME principles are not. With respect to method investments and the process of finding, analyzing and refining methods, it is the task of this thesis to decrease the costs related to ME. In other works, the incremental ME principles provide mechanisms to identify method refinement possibilities, manage method evolution, and automate part of the method refinement process. These decrease the costs of method improvements and thus of the benefits obtainable from engineering methods appropriate to the situation.

Most of the ME frameworks are based on an a priori view of ME in which no method refinements are expected during or after method use (cf. Section 3.3). Therefore, no principles or systematic guidelines for ME during and after method use have been proposed. To overcome this narrow view we analyzed the possibilities for iterations in the ME process. These possibilities were called method refinement scenarios. Based on this analysis we focused on specific scenarios which originate from method use experiences and lead to modifications in method knowledge or method-tool companionship. Hence, our interest is concerned with the deep structure of method knowledge: modifications of the conceptual structure, and its relation to notations.

Experience gathering and refinement should take place in modeling tools. This means that modeling experiences from tool-based method use are collected and analyzed. Although the analysis of “pure” tool support deals mostly with minor changes in the surface structure of a method (Wand 1996), our focus on method use with modeling tools is important for a number of reasons. First, method use in CASE forms a foundation for examining the underlying conceptual structure and notations. It makes ISD more transparent and the products accessible. Second, through the use of metaCASE technology we can allow method engineers to inspect the usage of methods. Third, by necessitating the use of a formal metamodel we expect that method modifications can be made explicit and formal. Finally, the resulting method refinements are also put into use through the tool adaptation. This means that method modifications can be shared quickly and can lead to learning in a whole ISD organization.

An incremental approach can be distinguished from other ME principles by identifying when and how method knowledge is constructed. In the radical mode, ME is viewed largely as an a priori method construction process, whereas in the incremental mode experiences of method use are collected and analyzed for the purpose of method refinements. Our focus is on this latter view. We claim that the applicability of the method can not be achieved based on a priori construction, but instead need to be investigated while using the method. This allows us to evaluate not only the applicability of a priori constructed methods, but also the relevance of the criteria that drive method construction.

We also identified ME situations which are most suitable for the proposed ME principles. These situations are characterized by high uncertainty and unavailability of method knowledge, and by a changing ISD environment. Hence, ISD projects which lack method knowledge or criteria for method construction benefit from incremental ME principles. Similarly, organizations which carry out long-term ISD efforts, often related to specific products, can take advantage of the incremental approach. Finally, incremental ME forms a part of various types of ISD improvements (Humprey 1988, Odell 1996). Therefore, organizations searching for long-term process improvement need incremental ME principles to improve their ISD processes.

[23] The method modifications can also occur while selecting or constructing the method, but we do not consider them because they are discussed in available ME approaches (cf. Punter and Lemmen 1996, van Slooten and Hodes 1996).